View From the Bushes

For all those who tried to access the website yesterday and found it unavailable, I apologize. Before I get into the story, comments have been disabled; it will become clear why, but I wanted to make sure to get that out of the way up front, since I know that very few people make it all the way to the end of my blogs. In any case, the story starts Tuesday night...

I noticed that DireKobold.com was running really slowly, but it was still running. I was also getting SQL errors in my e-mail. Now I had seen this before when some spammer really went crazy filling my blog with spam comments, but it had always been temporary. Also, I checked the comment table and it wasn't that big. Plus, I wasn't 100% sure that it was the blog, because the SQL errors were actually being generated by the forums, which are old and possibly insecure. So I was concerned, but I didn't immediately know what to do, so I went to bed without taking any action.

The next morning I woke up and DireKobold was down: no web, no e-mail, no ssh, though I could still ping it. When I tried to ssh to it, it would tell me connection refused, and somehow that, along with the fact that it pinged, somehow led me to believe that it had been hacked through the forum. So I called around and managed to scrounge up one of the many shiftless, but tech-savvy that seem to cluster around my friend Josh, like moths to the flame, to go take a look at it. When he tried bringing it up, it got to the part where it tried to find the hard drive and hung. And one of the warning LED's was lit on the mother board. This did not bode well.

So he unhooked it and took it over to Josh's house. When he tried bringing it up there it came up without a hitch. This was good news, but it didn't rule out a major hard drive issue, and my initial inclination was to tell him to immediately buy a new HD. Instead, I figured I'd live dangerously and I told him to just make a back-up and then put it back. I managed to make it to the datacenter as he put it back. As soon as the server was back in the rack we started checking it, and it was immediately apparent that it was once again incredibly slow. So we checked the process list and there were hundreds of apache threads. We looked at the connections and in the five minutes it had been back up, hundreds of connections had been made. The last thing we did was look at the logs, and it was apparent that the comment script for my blog was being hit hundreds of times a minute.

Once we finally realized what the problem was I turned off the webserver, and then a little later, after I disabled the comment script, I turned the web server back on, and it's been smooth sailing ever since. So as I mentioned, comments are disabled. Which leaves me with several options: the first is to turn comments on and cross my fingers, the second is to continue to blog here without comments, the final option is to move my blog somewhere else like blogger, or blogspot or livejournal, etc. At the moment I'm leaning towards the latter, but I'd like to know what you think... except you can't tell me because there is no commenting...

Filled with murderous rage for spammers
Ross