Skip to the content ForoHackers
Log inJoin

Restartable batch jobs: how to survive a failure at 3 a.m.

DatabasesInformation systemsMaintenance and legacy code
#970 · legacy_cobol_whisperer · · Report

Mainframe batch has a lot to teach modern pipelines. The central idea: a job will fail halfway, so design for the restart, not for the success.

Pattern I keep reusing:

  1. A restart table with job name, run id, and the last committed key. The job reads it on start.
  2. Commit in chunks (say every 1000 rows), writing the checkpoint in the same transaction as the work. Either both persist or neither.
  3. Idempotent writes: processing the same input twice must give the same result. Use upserts or a unique key on (run_id, record_id).
  4. Output goes to a temporary file and is renamed only on success, so downstream never reads a half-written file.
  5. No "reset and rerun from the top" as default. If a 6 hour job restarts from zero every time, the on-call person will eventually disable the restart logic out of frustration.

Written like that it sounds obvious. In practice most failures I have seen came from step 2: the checkpoint and the work in different transactions, so a crash between them either skipped or duplicated a chunk.

Calm batch is boring batch. Boring batch is why payroll runs on time.

#1286 · r3l1c_0f_bbs · · Report

Same philosophy as FidoNet packets, actually: if the call drops, you resume the transfer and the receiver can tell what it already has. The old systems were designed around bad connections, and it shows.

#1287 · null_terminated_hopes · · Report

"A crash between the checkpoint and the work" is the most frightening sentence in this thread. It is also the one that happens at 3 a.m. on a Sunday.

#1288 · legacy_cobol_whisperer · · Report

It is, and that is why I always ask in design reviews: "show me what happens if the machine loses power between these two lines." People go quiet, then draw a better diagram.

Log in or join to reply.