Skip to the content ForoHackers
Log inJoin

What 'test your restores' actually means: a checklist

Backups and recoveryReliability engineeringSite reliability
#1203 · cron_job_carl · · Report

Everybody says "test your restores". Almost nobody says what that means. Here is a ladder. Each step is a stronger claim than the one before it.

  1. The job ran. Weakest claim. It says nothing about the data.
  2. The backup has the expected size and file count. Compare to the last run. A sudden drop is an alarm.
  3. The tool's own integrity check passes (restic check, borg check, or the equivalent: it verifies the repository structure and, if you ask, the data).
  4. A sample restore matches. Restore to a scratch directory and compare checksums of a random handful of files against the originals.
  5. A full restore, timed. Now you know how long recovery takes, which is the number the business actually cares about.
  6. A restore on a clean machine using only what is documented, including the encryption key and passphrase. This catches the classic: the key was on the machine that just died.

Most setups sit at step 1 or 2 and call it a day. Put step 4 in a cron job and step 6 in the calendar, once a quarter.

A backup is what you can restore. Everything else is optimism with extra disk space.

#2118 · raid0_believer · · Report

I restored a file once, and it worked. Does that put me at step 5?

#2119 · cron_job_carl · · Report

One file, once, from a copy you made that week: that is step 4 on a very good day. Step 5 is everything, and you write down how long it took. Also, I notice you did not say which RAID level the data lives on.

#2120 · tired_sre_at_3am · · Report

Step 6 is the one that matters at 3 a.m. I have been in a room where the backup was fine, the restore tool was fine, and the key was in a password manager that lived on the server being restored. Blameless post-mortem, sad coffee.

#2121 · yak_shaving_daily · · Report

My step 6 turned into a three-day project: a throwaway VM that boots from nothing, pulls the docs, restores, and runs a smoke test. It now runs from a timer every month and sends me a one-line verdict. Best three days I have ever lost.

#2122 · floppy_disk_farmer · · Report

Adding step 7 for the vintage crowd: can you still read the medium? Having the tape or the disk is not the same as having a drive that works.

Log in or join to reply.