The weekend was a little shorter than normal, and while it had something to with Daylight Saving Time, it’s not what you think.

I maintain a number of servers (basically a whole NOC). One of our servers came with a new “feature” after the latest release of it’s software… if DST hits and changes the clock, and you have a scheduled job that was supposed to run but was bypassed because of the clock advancing 1 hour… freak out, do nothing, and don’t say why.

Of course, this is the same server that I spent all day thursday and friday reconfiguring so it will talk to another server (one server is on the internal production network, the other is in the DMZ, and they use a NAT’ed IP to talk… but there are issues), so I figured for sure I broke something.

After hours of going through xml config files, log files, beating my head against the rack, and sifting through the database, I finally got it.

Apparently, if there’s a job buried deep in the database, and this job is scheduled to run during the time period that you skip at the start of DST, everything will run fine, unless, of course, you reboot. Then it will piss you off and ruin your weekend.

Of course, I’ll forget about this by next spring when it all happens again.