View From the Bushes

Well, today illustrated why Monday has such a bad reputation. I was woken up around 5 am by a call from my work telling me that one of the webservers was down. I restarted it from home, but I probably should have just gone into work since I didn't really manage to fall back asleep. Then shortly after I did get into work, something else broke. Our intranet server acts as the gateway for about a dozen or so other programs and one of them was hanging. This has been happening with some frequency for the last week or so, and every time it's been a different program, so rather than fighting the symptoms I determined to stamp out the disease. As usual, it was my hubris which ended up destroying me.

Looking at the server, it seemed to be having some problems communication with the Domain Name Server. Talking to some of the hardware guys, they mentioned that they had seen this problem before and the fix was to delete it from the domain, reboot it into local mode and then re-add it to the domain. Since I had to reboot the server anyway to fix the symptomatic problem, this seemed a good time to do it. So we removed it from the domain, rebooted it in 'local-mode' and went to log on, only to discover that we didn't know the administrator password. I hadn't set up the server and as a matter of fact, we couldn't find anyone who remembered setting it up. We tried every single password anyone could think of and nothing worked.

Fortunately, even though the server didn't think it was part of the domain, everything else did, which meant that mostly things were still working as they should. In fact, the problem which had started the whole thing had even been fixed. Over the course of the next few hours, we were able to mostly fix things so that externally the intranet looked pretty good, but of course internally it was still very, very sick. We tried selectively restoring files and directory from back up, we tried a couple of password crackers (neither of which could see the SCSI raid). We brainstormed like madmen, but it seemed like there was nothing to be done but start from scratch. Eventually, we did find a password cracker that 'works' -- the only problem is that it's estimating it's going to take 21 days in order to brute-force the password... I'll keep you posted on what happens.

Carpe Diem Quam Minimum Credula Postero
Ross