Skip to the content ForoHackers
Log inJoin

Everything is scheduled at midnight on the first of the month

System administration
#1079 · anonymous · · Report

be me, new sysadmin cron on the main server backups: 00:00 on the 1st log rotation: 00:00 on the 1st report generation: 00:00 on the 1st database vacuum: 00:00 on the 1st invoice batch: 00:00 on the 1st

Every first of the month at one minute past midnight the machine sounds like a jet engine and the monitoring turns the colour of a fire truck. Everyone "knows" it. Nobody has moved a single job in nine years.

I want to stagger them. Is there a polite way to ask permission to make the server boring?

#1664 · anonymous · · Report

Do not ask permission to stagger, ask permission to measure. Collect load for one month, show the spike, then propose moving the jobs by ten minutes each in order of dependency. Nobody argues with a graph.

#1665 · anonymous · · Report

Also add a random delay to anything that must run on the first. With systemd timers that is RandomizedDelaySec= and with plain cron a sleep $((RANDOM % 600)) at the top of a bash script ($RANDOM is a bash feature, not plain sh). Make sure the invoice batch does not start before the report it depends on, though.

#1666 · anonymous · · Report

Also check where the output of each job goes. Five jobs mailing their output to the same inbox at 00:01 is its own incident. Send them to a log and alert only on failure.

Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply