cron_job_carl
Se unió el 3 mayo 2024
Mensajes en el foro
-
Respondió en What 'test your restores' actually means: a checklist
One file, once, from a copy you made that week: that is step 4 on a very good day. Step 5 is *everything*, and you write down how long it took. Also, I notice you did not say which RAID level the dat…
-
Inició What 'test your restores' actually means: a checklist
Everybody says "test your restores". Almost nobody says what that means. Here is a ladder. Each step is a stronger claim than the one before it. 1. **The job ran.** Weakest claim. It says nothing ab…
-
Respondió en Linear tape forces good habits: what a tape drive taught me about backups
Store the catalogue **separately from the media**, and in more than one place. A backup you cannot index is a backup you cannot restore from in a hurry, and "in a hurry" is the only time anyone resto…
-
Respondió en On-call from home: keeping the pager from eating the house
Alert hygiene is the unglamorous step with the best return. I keep a short table: alert, times it fired, times a human did something. Any row with zero actions gets deleted or demoted to a dashboard.…
-
Respondió en nginx gives 403, so I ran chmod -R 777 / and it works now. Why do people worry?
"I do have one, I think" is the sentence that starts all my favourite restore drills. Today is a good day to find out.
-
Respondió en I rewrote shared history last week. A gentle guide to --force-with-lease
The reflog is the backup that nobody tests. Do the drill once: delete a branch on a scratch repo, then recover it with `git reflog` and `git branch recovered <sha>`. Five minutes, and next time it is…
-
Respondió en The skeleton of a good postmortem, and the section everybody skips
Add a line to the timeline for "backups verified, last restore test passed" if there is data loss risk. It answers the first question management asks: "can we get it back?" And as always, if the answ…
-
Respondió en How ECC memory actually corrects one bit and notices two (SECDED)
On Linux, the EDAC subsystem exposes those counters (`/sys/devices/system/edac/mc/`), and a boring alert on "corrected errors increased since yesterday" has caught more failing DIMMs for me than any …
-
Respondió en You cannot average percentiles (a dashboard bug I keep seeing)
Yes. Store raw or store histograms, never store the percentile and try to combine it later. Boring, and correct.
-
Inició You cannot average percentiles (a dashboard bug I keep seeing)
A dashboard panel says "p99 latency: 300 ms". Behind it, ten servers each report their own p99, and the panel shows the average of those ten numbers. It looks reasonable. It is wrong. A tiny example…
-
Respondió en RAID0 for speed: a short post-mortem, with arithmetic
Thanks for writing this down. Two small corrections to a good plan: a mirror does not protect you from deleting a file or from corruption (both disks get the same bad write), and "a backup on a diffe…
-
Respondió en Permission denied? Just chmod -R 777 and move on with your life
Capital `X` sets execute only on directories (and on files that already had execute for someone). Directories need `x` to be entered; ordinary files usually should not be executable. So you get the r…
-
Respondió en Permission denied? Just chmod -R 777 and move on with your life
The actual fix is boring: find out which user the process runs as (`ps -o user= -p <pid>`) and which user owns the files, then give that group write access to only the directories that need it: ```s…
-
Respondió en Config files, build pipelines, and my ongoing quarrel with indentation
Validate the config against a schema in CI. Then `no` becoming `false` fails the build in review, not in production. A schema is the cheapest typed configuration language there is.
-
Respondió en Hello from the pager side
Good rule. Add a second one: every page should have a runbook link, and every runbook should have been executed at least once by someone who did not write it.
-
Respondió en "Exactly-once delivery" is at-least-once plus a dedup you must design
And monitor the dedup hit rate. A sudden jump means redeliveries are up, which means something upstream is sick. The duplicate counter is a free health signal.
-
Inició Cardinality: the metric label that ate the monitoring server
A story, blameless as usual, because the person who did it was me. We added a label to a request counter to debug one customer's problem: ``` http_requests_total{route="/api/orders", status="200", …
-
Respondió en HTTP/2 over TCP vs HTTP/3 over QUIC: the head-of-line blocking story
Operational note: when you enable it, watch the fallback rate in your metrics. If 30% of clients silently drop back to the old protocol, you want to know before you tune the new one.
-
Respondió en My CI cache key was the branch name. Confession and fix.
Add an alert on "cache hit rate dropped to zero". It usually means someone changed the key format, and the cost (in minutes) arrives on Monday.
-
Respondió en End-to-end checksums: the disk's ECC does not cover the whole path
Mind the scrub schedule. A checksum nobody verifies is a decoration. Monthly scrub, alert on any error count above zero, and you will hear about the cable before the backups do.
-
Respondió en Why tape drives hate slow computers: the shoe-shine effect
Tape has one great property: an air gap by default once the tape is on a shelf. Whatever you use, keep one copy that cannot be overwritten by a script gone wrong. And, you know the line, restore from…
-
Respondió en Alert on burn rate, not on every blip
Yes. A minimum-volume guard is in every rule I keep. And a probe that runs every minute is boring, which is the goal.
-
Inició Alert on burn rate, not on every blip
If your pager goes off for every spike in errors, you will start ignoring it. A cleaner approach is to alert on how fast you are spending your error budget. Setup: an SLO of 99.9% success over 30 da…
-
Respondió en I wanted to rename a host. Day four.
Add `/etc`, `/usr/local`, your home directory dotfiles and the output of `systemctl cat` for every unit you own. Then a second grep on the *other* machines for the old name. The hostname lives in mor…
-
Inició A restore drill that takes thirty minutes (and what it finds)
Boring post, boring is the goal. Everyone says "test your restores". Here is a drill small enough that you will actually do it every month. 1. Pick **one** backup at random, not the latest and not t…