Saltar al contenido ForoHackers

Message queues and streaming: start here

Colas de mensajes y streaming
#2939 · · Citar · Denunciar

Message queues and streams let programs talk without waiting for each other: a producer writes a message, a broker holds it, and consumers read it when ready. This decouples services in time and speed, absorbs bursts, and lets work be retried, at the price of new questions about ordering and duplicates.

Key concepts

  • Queue and log: a queue hands each message to one consumer and removes it; a log keeps messages in order for many readers to replay.
  • Publish and subscribe: producers send to a topic, and every interested subscriber receives a copy.
  • Delivery guarantees: at most once, at least once and exactly once, and why the last is really at least once plus idempotent handling.
  • Ordering and partitions: order holds within a partition, so keys decide which messages stay in sequence.
  • Backpressure: slow consumers need a way to tell fast producers to ease off.
  • Failure handling: acknowledgements, retries with backoff and dead-letter queues for poison messages.

Vocabulary

  • Broker: the server that stores and routes messages.
  • Consumer group: consumers sharing the work of one topic.
  • Offset: a reader's position in a log.
  • Dead-letter queue: a place for messages that keep failing.
  • Event sourcing: storing a system's state as the sequence of events that produced it.

What to study, in order

  1. Concurrency basics and the producer-consumer problem in one process.
  2. Run an open-source broker locally and send messages between two small programs.
  3. Make a consumer crash mid-message and observe redelivery; then make it idempotent.
  4. Partition a topic by key and verify ordering and parallelism.
  5. Build a small stream pipeline that counts events over time windows.

Related tags

#distributed-systems #backend #architecture #concurrency #data-engineering

No has iniciado sesión, así que esto se publica como anónimo: no se guarda con ello ninguna cuenta, nombre ni dirección, y no puede llevar enlaces ni imágenes. Inicia sesión o únete para publicar con tu nombre. Los registros de nuestro servidor web guardan la dirección y la hora de cada petición hasta 14 días; para que la tuya no quede en ellos, usa Tor. Para escribir al equipo de forma confidencial, usa el buzón. Firmar como fuente habitual (una clave guardada solo en este navegador)

Enviar tarda unos segundos aquí: tu navegador resuelve antes una prueba de trabajo contra el spam.

Responder