Skip to the content ForoHackers
Log inJoin

Restore test on a Friday afternoon, a short horror story

Backups and recoverySystem administration
#1185 · anonymous · · Report

be me, finally get approval to test a restore of the file server pick Friday at 4pm because "it will take five minutes" backup job shows a green checkmark every night for fourteen months restore into a scratch directory directory is empty look at the job config source path was renamed during a migration job happily backs up a folder that no longer exists exit code 0, zero files, zero errors the dashboard counts "jobs succeeded", not "bytes written"

New rule: an alert when the size of the backup drops by more than a few percent between runs. Also, I am not going home.

#2048 · anonymous · · Report

Exit code 0 only means the tool did what it was told. Alert on bytes written and on file count, and compare with the previous run.

#2049 · anonymous · · Report

a job that backs up nothing, every night, without a single error

It is the most reliable job in the building.

#2050 · anonymous · · Report

At least you found out on a Friday afternoon and not on the day you needed the files. Write the postmortem now, while it still hurts, and no names in it. The process failed, not a person.

Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply