Owning and operating your own company is really stressful. When I was handed complete ownership of Host-A.net I didn’t realize how much stress it was going to be. Now I worry about mirrors going down, hard drive failure and much more. It has been just over a month since the reigns have been handed over and we have had 3 major problems. One being a major hard drive failure on the main server, how we managed to fix that was to redirect all the traffic to one of the mirrors, unfortunately for this system users couldn’t upload any files.
The second major problem was when I switched the servers from PHP4 to PHP5 (a long needed upgrade) I forgot to install some extra package, like GD. We rely on GD to make the CAPTCHA image for when users sign up so we didn’t have any signups for a while. Also I the php5-mysql package was not installed, this is pretty major because it means that the site can’t talk to the database and no one can get at their files.
Our third major problem, probably the biggest, was all my fault. I was working on a function for the admin side of things to be able to remove certain users and their files. Yes this may lead to some power abuse, but I will deal with that when the time comes. For the record I can delete any user that needs to be deleted right now and that this function will only make it easier. What happened was when I was testing it everything seemed to be fine, except that I noticed it taking a little too long. So I do a quick “ls -l” on the directory where all the users files are, not just the single user that I was trying to delete for this test case and I noticed that everything was disappearing quickly. I was able to “pkill” the task before it deleted all of the files, luckily it only deleted 50% of the files and we back up to a few mirrors nightly. I switched all the links going to the main server to the secondary server so the users could still get their files and started to rsync the files that were deleted back onto the main server. Let me tell you 60GBs is no small thing to transfer over the internet.
In conclusion, we haven’t expected all problems that could happen or even will. There probably will be more in the future, but we have a great team in place to be able to deal will all sorts of problems.
Filed under: Rant
