be me, on call between Christmas and New Year, quiet office, empty building pager at 22:00: "disk 95% full on monitoring host" ssh in the biggest directory is the monitoring system's own debug log debug log is huge because it is logging every failed check every check fails because the disk is almost full I am watching a snake eat its own tail delete the debug log, rotate it, set a size limit alert clears write ticket: "alert about disk on host whose job is alerting about disks" go back to eating leftovers
Moral: monitor your monitoring from somewhere else, and put a ceiling on every log you create.