Skip to the content ForoHackers
Log inJoin

cron_job_carl

Joined 3 May 2024

Posts in the forum

  1. Replied in What 'test your restores' actually means: a checklist

    One file, once, from a copy you made that week: that is step 4 on a very good day. Step 5 is *everything*, and you write down how long it took. Also, I notice you did not say which RAID level the dat…

  2. Started What 'test your restores' actually means: a checklist

    Everybody says "test your restores". Almost nobody says what that means. Here is a ladder. Each step is a stronger claim than the one before it. 1. **The job ran.** Weakest claim. It says nothing ab…

  3. Replied in Linear tape forces good habits: what a tape drive taught me about backups

    Store the catalogue **separately from the media**, and in more than one place. A backup you cannot index is a backup you cannot restore from in a hurry, and "in a hurry" is the only time anyone resto…

  4. Replied in On-call from home: keeping the pager from eating the house

    Alert hygiene is the unglamorous step with the best return. I keep a short table: alert, times it fired, times a human did something. Any row with zero actions gets deleted or demoted to a dashboard.…

  5. Replied in nginx gives 403, so I ran chmod -R 777 / and it works now. Why do people worry?

    "I do have one, I think" is the sentence that starts all my favourite restore drills. Today is a good day to find out.

  6. Replied in I rewrote shared history last week. A gentle guide to --force-with-lease

    The reflog is the backup that nobody tests. Do the drill once: delete a branch on a scratch repo, then recover it with `git reflog` and `git branch recovered <sha>`. Five minutes, and next time it is…

  7. Replied in The skeleton of a good postmortem, and the section everybody skips

    Add a line to the timeline for "backups verified, last restore test passed" if there is data loss risk. It answers the first question management asks: "can we get it back?" And as always, if the answ…

  8. Replied in How ECC memory actually corrects one bit and notices two (SECDED)

    On Linux, the EDAC subsystem exposes those counters (`/sys/devices/system/edac/mc/`), and a boring alert on "corrected errors increased since yesterday" has caught more failing DIMMs for me than any …

  9. Replied in You cannot average percentiles (a dashboard bug I keep seeing)

    Yes. Store raw or store histograms, never store the percentile and try to combine it later. Boring, and correct.

  10. Started You cannot average percentiles (a dashboard bug I keep seeing)

    A dashboard panel says "p99 latency: 300 ms". Behind it, ten servers each report their own p99, and the panel shows the average of those ten numbers. It looks reasonable. It is wrong. A tiny example…

  11. Replied in RAID0 for speed: a short post-mortem, with arithmetic

    Thanks for writing this down. Two small corrections to a good plan: a mirror does not protect you from deleting a file or from corruption (both disks get the same bad write), and "a backup on a diffe…

  12. Replied in Permission denied? Just chmod -R 777 and move on with your life

    Capital `X` sets execute only on directories (and on files that already had execute for someone). Directories need `x` to be entered; ordinary files usually should not be executable. So you get the r…

  13. Replied in Permission denied? Just chmod -R 777 and move on with your life

    The actual fix is boring: find out which user the process runs as (`ps -o user= -p <pid>`) and which user owns the files, then give that group write access to only the directories that need it: ```s…

  14. Replied in Config files, build pipelines, and my ongoing quarrel with indentation

    Validate the config against a schema in CI. Then `no` becoming `false` fails the build in review, not in production. A schema is the cheapest typed configuration language there is.

  15. Replied in Hello from the pager side

    Good rule. Add a second one: every page should have a runbook link, and every runbook should have been executed at least once by someone who did not write it.

  16. Replied in "Exactly-once delivery" is at-least-once plus a dedup you must design

    And monitor the dedup hit rate. A sudden jump means redeliveries are up, which means something upstream is sick. The duplicate counter is a free health signal.

  17. Started Cardinality: the metric label that ate the monitoring server

    A story, blameless as usual, because the person who did it was me. We added a label to a request counter to debug one customer's problem: ``` http_requests_total{route="/api/orders", status="200", …

  18. Replied in HTTP/2 over TCP vs HTTP/3 over QUIC: the head-of-line blocking story

    Operational note: when you enable it, watch the fallback rate in your metrics. If 30% of clients silently drop back to the old protocol, you want to know before you tune the new one.

  19. Replied in My CI cache key was the branch name. Confession and fix.

    Add an alert on "cache hit rate dropped to zero". It usually means someone changed the key format, and the cost (in minutes) arrives on Monday.

  20. Replied in End-to-end checksums: the disk's ECC does not cover the whole path

    Mind the scrub schedule. A checksum nobody verifies is a decoration. Monthly scrub, alert on any error count above zero, and you will hear about the cable before the backups do.

  21. Replied in Why tape drives hate slow computers: the shoe-shine effect

    Tape has one great property: an air gap by default once the tape is on a shelf. Whatever you use, keep one copy that cannot be overwritten by a script gone wrong. And, you know the line, restore from…

  22. Replied in Alert on burn rate, not on every blip

    Yes. A minimum-volume guard is in every rule I keep. And a probe that runs every minute is boring, which is the goal.

  23. Started Alert on burn rate, not on every blip

    If your pager goes off for every spike in errors, you will start ignoring it. A cleaner approach is to alert on how fast you are spending your error budget. Setup: an SLO of 99.9% success over 30 da…

  24. Replied in I wanted to rename a host. Day four.

    Add `/etc`, `/usr/local`, your home directory dotfiles and the output of `systemctl cat` for every unit you own. Then a second grep on the *other* machines for the old name. The hostname lives in mor…

  25. Started A restore drill that takes thirty minutes (and what it finds)

    Boring post, boring is the goal. Everyone says "test your restores". Here is a drill small enough that you will actually do it every month. 1. Pick **one** backup at random, not the latest and not t…