Incidents

The site and every service on the server were down for about 16 hours. The most likely cause: the system drive stopped responding after two and a half days of an incompatible graphics driver writing errors to the log non-stop. The server did not restart by itself, and I only found out about the outage 8 hours later.

Timeline (Moscow time)

1. 25.10 05:02The server boots. From the first second the graphics driver tries to initialise the card, fails and retries — roughly once a second. Every failure is written to the system log.
2. 25–27.10185,000 errors in 57 hours; the log grows to almost 1 GB on the system drive. From the outside everything works.
3. 26.10 23:24Last check at which the site was definitely up.
4. 27.10 14:00:22Last log entry — cut off mid-stream, with no clean shutdown. The server stops responding.
5. 27.10 ~22:15The outage is noticed during a routine check. External checks from four countries: “No route to host”.
6. 28.10 06:15The server is rebooted by hand. Every service, the site included, comes back automatically.
7. 28.10Investigation from the logs and fixes — see below.

Cause

The server had a graphics driver that does not support the old graphics card installed in it. The service watching the driver kept restarting it in a loop, and every loop produced new log entries. Two and a half days of non-stop writes is a heavy load for the system drive; most likely it stopped responding, and the log was cut off because there was nowhere left to write.

What remains unclear

The drive theory explains the cut-off log and why the watchdog did not fire: the kernel kept feeding it. But external checks got “No route to host”, which usually means the server did not respond even at the network level — that should not happen if only the drive hung. There is no direct evidence and there cannot be: the log lived on the same drive. So the cause is called most likely, not proven.

Why the outage lasted so long

• The hardware watchdog was on (5 minutes) but did not fire: apparently the kernel kept signalling “alive”.
• Uptime monitoring ran on the same server — and hung with it. No alerts were sent.
• The driver errors started at boot, but nobody read the log: the noise was invisible until it became an outage.

What went well

• After the reboot every service came back by itself — nothing had to be started by hand.
• No data was lost: the data disk is healthy, and the site keeps no state that could have been corrupted.
• The logs survived up to the last second before the failure — enough to reconstruct what happened.

What was fixed

• [x] The incompatible graphics driver was removed: no more errors in the log.
• [x] The log is capped at 100 MB; the 933 MB that had piled up were deleted.
• [x] Hardware watchdog set to 30 seconds instead of 5 minutes.
• [x] An external uptime check of the site — from a third-party service, not from the same server: an outage now triggers an alert.

Lessons

• Monitoring that lives on the same server cannot see that server go down.
• A watchdog does not catch every hang — only the ones where the kernel stops.
• An endless stream of errors in the log is not cosmetic — it is a reliability risk.