Skip to the content ForoHackers
Log inJoin

"Exactly-once delivery" is at-least-once plus a dedup you must design

DatabasesDistributed systemsMessage queues and streaming
#1048 · mutex_mike · · Report

A message broker can lose an acknowledgement in transit. Consumer processes a message, crashes before acking, broker redelivers. Or the consumer acks but the ack is lost, same result. You can only choose between:

  • At-most-once: do not retry. Messages can be lost.
  • At-least-once: retry until acked. Messages can arrive twice.

Over an unreliable network, delivering exactly once is not something the transport can promise. What you can have is exactly-once processing: at-least-once delivery combined with a consumer whose effect is idempotent.

The robust way to do it is to record "I have handled message X" in the same transaction as the effect:

BEGIN;
INSERT INTO processed (msg_id) VALUES ($1) ON CONFLICT DO NOTHING;
-- if zero rows were inserted, we saw it before: COMMIT and just ack
UPDATE tallies SET total = total + $2 WHERE id = $3;
COMMIT;
-- only now ack the message

Timeline of a crash:

consumer:  BEGIN ... UPDATE ... COMMIT ---x(crash before ack)
broker:    ........................ redeliver -------->
consumer:  BEGIN INSERT conflicts -> already done -> COMMIT -> ack

Details people miss:

  1. The dedup record needs a retention policy longer than the broker's maximum redelivery window.
  2. The id must come from the producer (or be stable per business event), not generated on each attempt.
  3. Side effects outside the database (an email, an HTTP call) are not covered by the transaction. They need their own idempotency key at the receiver.
#1560 · two_phase_commitment · · Report

This is the happy cousin of two-phase commit: instead of coordinating two systems atomically, you make the second one safe to repeat. The commitment is one-sided, and it works.

#1561 · race_condition_rita · · Report

Point 2 deserves a bold: an id generated per attempt makes the dedup useless, and it looks fine in every test that has no failures. The test to write injects a duplicate delivery and checks the total changed once.

#1562 · legacy_cobol_whisperer · · Report

On the mainframe we called it a restart key. Same idea, older name: every input record carries a unique key and the update step refuses a key it has seen. Every decade rediscovers a good idea and gives it a conference talk.

#1563 · cron_job_carl · · Report

And monitor the dedup hit rate. A sudden jump means redeliveries are up, which means something upstream is sick. The duplicate counter is a free health signal.

#1564 · mutex_mike · · Report

Good one, Carl. Duplicates per hour is one of the first graphs I draw on any queue.

Log in or join to reply.