Skip to the content ForoHackers
Log inJoin

Message queues and streaming: start here

Message queues and streaming
#2939 · · Quote · Report

Message queues and streams let programs talk without waiting for each other: a producer writes a message, a broker holds it, and consumers read it when ready. This decouples services in time and speed, absorbs bursts, and lets work be retried, at the price of new questions about ordering and duplicates.

Key concepts

  • Queue and log: a queue hands each message to one consumer and removes it; a log keeps messages in order for many readers to replay.
  • Publish and subscribe: producers send to a topic, and every interested subscriber receives a copy.
  • Delivery guarantees: at most once, at least once and exactly once, and why the last is really at least once plus idempotent handling.
  • Ordering and partitions: order holds within a partition, so keys decide which messages stay in sequence.
  • Backpressure: slow consumers need a way to tell fast producers to ease off.
  • Failure handling: acknowledgements, retries with backoff and dead-letter queues for poison messages.

Vocabulary

  • Broker: the server that stores and routes messages.
  • Consumer group: consumers sharing the work of one topic.
  • Offset: a reader's position in a log.
  • Dead-letter queue: a place for messages that keep failing.
  • Event sourcing: storing a system's state as the sequence of events that produced it.

What to study, in order

  1. Concurrency basics and the producer-consumer problem in one process.
  2. Run an open-source broker locally and send messages between two small programs.
  3. Make a consumer crash mid-message and observe redelivery; then make it idempotent.
  4. Partition a topic by key and verify ordering and parallelism.
  5. Build a small stream pipeline that counts events over time windows.

Related tags

#distributed-systems #backend #architecture #concurrency #data-engineering

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply