Saltar al contenido ForoHackers

On-call over the holidays: the alert was the monitoring filling its own disk

ObservabilidadFiabilidad de sitios (SRE)
#1140 · anónimo · · Denunciar

be me, on call between Christmas and New Year, quiet office, empty building pager at 22:00: "disk 95% full on monitoring host" ssh in the biggest directory is the monitoring system's own debug log debug log is huge because it is logging every failed check every check fails because the disk is almost full I am watching a snake eat its own tail delete the debug log, rotate it, set a size limit alert clears write ticket: "alert about disk on host whose job is alerting about disks" go back to eating leftovers

Moral: monitor your monitoring from somewhere else, and put a ceiling on every log you create.

#1884 · anónimo · · Denunciar

monitor your monitoring from somewhere else

A tiny dead man's switch on another host is enough: if the heartbeat stops, alert. It catches the case where the monitor dies silently, which is the scary one.

#1885 · anónimo · · Denunciar

The snake eating its own tail is a perfect on-call image. Mine was a log shipper that logged its own failures to the file it was shipping.

#1886 · anónimo · · Denunciar

alert about disk on host whose job is alerting about disks

I am going to put that in the next incident review slides, with credit to whoever wrote that line.

Firmar como fuente habitual (una clave guardada solo en este navegador)

Enviar tarda unos segundos aquí: tu navegador resuelve antes una prueba de trabajo contra el spam.

Responder