Before any "smash the stack" challenge, make sure you can draw the frame of a function. Take the classic practice binary, built for the exercise:
void greet(void) {
char name[16];
gets(name); /* the deliberate bug in the challenge */
printf("hi %s\n", name);
}
On x86-64 the prologue is boring and the picture is what matters:
push rbp
mov rbp, rsp
sub rsp, 16
lea rax, [rbp-16] ; &name
...
higher addresses
+--------------------+
| return address | <- rbp+8
| saved rbp | <- rbp
| name[16] | <- rbp-16
+--------------------+
lower addresses
gets writes upward, towards higher addresses, with no idea how big name is. Past 16 bytes you are inside the saved rbp; past 24 you are inside the return address, which ret will happily pop and jump to. That is the whole idea. No spells.
The homework for your own lab VM: in gdb, break after the prologue, run x/4gx $rsp and info frame, and match what you see to the diagram. Then recompile with -fstack-protector-strong and look at the extra canary load and compare at the end of the function.