Mainframe batch has a lot to teach modern pipelines. The central idea: a job will fail halfway, so design for the restart, not for the success.
Pattern I keep reusing:
- A restart table with job name, run id, and the last committed key. The job reads it on start.
- Commit in chunks (say every 1000 rows), writing the checkpoint in the same transaction as the work. Either both persist or neither.
- Idempotent writes: processing the same input twice must give the same result. Use upserts or a unique key on
(run_id, record_id). - Output goes to a temporary file and is renamed only on success, so downstream never reads a half-written file.
- No "reset and rerun from the top" as default. If a 6 hour job restarts from zero every time, the on-call person will eventually disable the restart logic out of frustration.
Written like that it sounds obvious. In practice most failures I have seen came from step 2: the checkpoint and the work in different transactions, so a crash between them either skipped or duplicated a chunk.
Calm batch is boring batch. Boring batch is why payroll runs on time.