[svn] - [hg] - [git] - [bugzilla] - [ssl] - [mail]
News archive: (submit news)
For lack of a better explanation, I think icculus.org got hit by a power surge today. The system rebooted itself, and went into some sort of nasty loop. Zeph called me to report that nothing was getting through, and I checked it out. After trying unsuccessfully to ssh to the server from home, I drove to the Cyberspace Matrix and checked out the machine. From home, I could finger some accounts, and get static http requests...but anything that took disk or CPU time just sort of hanged indefinitely. When I got to the CSM, I couldn't log in as root on the console, there was a kernel panic in the ext3 driver scrawled across the monitor, the system load was sitting around 175.3, and the forkbomb defuser was going nuts putting out fires from user 99 ("nobody").
However, it doesn't appear that the server has been compromised. I also find it hard to believe that a system under normal load would decide to go wonky after 120 or so days of uptime short of hardware failure. Further research (once I got the system back up on Slackware 8.1's rescue disk...it has ext3 support, so I could play back the journal; hallelujah!) showed that there wasn't any file system corruption at all, let alone the sort that would cause a kernel panic in the ext3 filesystem driver.
Add to the facts that the server next to icculus.org was powered down mysteriously, and it seems that our section of the server room got an unexpected jolt.
This is actually a good thing, overall. I have been waiting to do some updates to the server that require reboots, and was riding out the uptime thing before doing them...hardware changes and kernel upgrades and such.
The bottom line is that everything appears to be fine. If anything looks suspicious or is giving you trouble, please let me know and we'll get it sorted. You shouldn't miss any incoming email, but some might be trickling in over the next few hours (in practice, it almost always floods in immediately, though).
For now, it's way past my bedtime.
[ email newsmaster | get news rdf ]