Email is a federated system: any server can send to any other, with no central owner, over protocols designed long before spam and fraud. Running a mail or messaging server means moving messages reliably and proving that yours are genuine, which is where most of the difficulty lives.
Key concepts
- Message flow: a mail user agent submits to a server, which relays by SMTP to the recipient's server, which stores it for retrieval by IMAP.
- DNS records: MX says where mail for a domain goes; SPF, DKIM and DMARC let receivers judge whether a sender is legitimate.
- Queues and retries: temporary failures are retried for days, and a bounce reports a permanent one.
- Transport security: TLS between servers and clients, and authentication before relaying.
- Reputation and filtering: receivers score senders and filter spam, so a misconfigured server is quickly distrusted.
Vocabulary
- MTA: mail transfer agent, the server that relays messages by SMTP.
- MDA and IMAP server: the parts that deliver to a mailbox and let clients read it.
- Open relay: a server that forwards mail for anyone, which must never be left open.
- Bounce: a delivery failure notice sent back to the sender.
- Federation: independent servers interoperating by a shared protocol, as email and many chat systems do.
What to study, in order
- Follow one message by hand: speak SMTP to your own lab server with a terminal client and read the headers.
- Install a mail server in a lab and send between two domains you control.
- Add MX, SPF, DKIM and DMARC records and check them from the receiving side.
- Configure TLS, authentication and a spam filter, and read the logs of each decision.
- Look at a federated chat server such as XMPP or Matrix and compare its model with email's.