Saltar al contenido ForoHackers

Message queues and streaming: start here

Colas de mensajes y streaming
#433 · Ophuscado · · Denunciar

Message queues and streams let programs talk without waiting for each other: a producer writes a message, a broker holds it, and consumers read it when ready. This decouples services in time and speed, absorbs bursts, and lets work be retried, at the price of new questions about ordering and duplicates.

Key concepts

  • Queue and log: a queue hands each message to one consumer and removes it; a log keeps messages in order for many readers to replay.
  • Publish and subscribe: producers send to a topic, and every interested subscriber receives a copy.
  • Delivery guarantees: at most once, at least once and exactly once, and why the last is really at least once plus idempotent handling.
  • Ordering and partitions: order holds within a partition, so keys decide which messages stay in sequence.
  • Backpressure: slow consumers need a way to tell fast producers to ease off.
  • Failure handling: acknowledgements, retries with backoff and dead-letter queues for poison messages.

Vocabulary

  • Broker: the server that stores and routes messages.
  • Consumer group: consumers sharing the work of one topic.
  • Offset: a reader's position in a log.
  • Dead-letter queue: a place for messages that keep failing.
  • Event sourcing: storing a system's state as the sequence of events that produced it.

What to study, in order

  1. Concurrency basics and the producer-consumer problem in one process.
  2. Run an open-source broker locally and send messages between two small programs.
  3. Make a consumer crash mid-message and observe redelivery; then make it idempotent.
  4. Partition a topic by key and verify ordering and parallelism.
  5. Build a small stream pipeline that counts events over time windows.

Related tags

#distributed-systems #backend #architecture #concurrency #data-engineering

Inicia sesión o únete para responder.