A SYN flood works because the server allocates state for each half-open connection. Send enough SYNs from spoofed sources, never answer, and the backlog fills.
SYN cookies fix this by making the server stateless during the handshake. Instead of storing anything, it encodes the information in the initial sequence number of its SYN-ACK:
- a slowly increasing counter (a timestamp, so cookies expire),
- an index for the MSS the client asked for (a few bits),
- a keyed hash of the 4-tuple (addresses and ports) plus a secret.
When a real client replies with ACK, its acknowledgement number is cookie + 1. The server recomputes the hash, checks it, and only now builds the connection. Spoofed sources never see the SYN-ACK, so they never produce a valid ACK.
The price: the server cannot remember options like window scaling from the original SYN (unless TCP timestamps are used to smuggle them), so cookies are only used when the backlog is already overflowing. On Linux that is net.ipv4.tcp_syncookies = 1.
Clever, cheap, and slightly cheeky: the packet that comes back is the memory.