Skip to the content ForoHackers
Log inJoin

On-call over the holidays: the alert was the monitoring filling its own disk

ObservabilitySite reliability
#1140 · anonymous · · Report

be me, on call between Christmas and New Year, quiet office, empty building pager at 22:00: "disk 95% full on monitoring host" ssh in the biggest directory is the monitoring system's own debug log debug log is huge because it is logging every failed check every check fails because the disk is almost full I am watching a snake eat its own tail delete the debug log, rotate it, set a size limit alert clears write ticket: "alert about disk on host whose job is alerting about disks" go back to eating leftovers

Moral: monitor your monitoring from somewhere else, and put a ceiling on every log you create.

#1884 · anonymous · · Report

monitor your monitoring from somewhere else

A tiny dead man's switch on another host is enough: if the heartbeat stops, alert. It catches the case where the monitor dies silently, which is the scary one.

#1885 · anonymous · · Report

The snake eating its own tail is a perfect on-call image. Mine was a log shipper that logged its own failures to the file it was shipping.

#1886 · anonymous · · Report

alert about disk on host whose job is alerting about disks

I am going to put that in the next incident review slides, with credit to whoever wrote that line.

Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply