A little background, I do software support for a package that does Hotel reservations.
Last night at work I had another problem that could have been resolved quickly with a little work had the IT person at the property knew what the hell he was supposed to do.
You see, one of their hard drives crashed. It most likely crashed because they tried to hot swap the drive out and it wasn’t hot swappable. Dumb fucks. But that really isn’t our problem. We tell them to get the drive working and that we will help restore the data from backups.
Of coarse the guy in charge of doing the nightly backups, the dumb fuck I spoke with for hours last night, never ever bothered to check the integrety of the backups. So he goes to do a restore and finds that the last good backup they have is from October 10th. Yep, every backup they have done since October 10th(and this is supposed to be done daily) is screwed. So instead of restoring 2 files and doing about a half hour worth of data reentry(that they would have to do) I got to spend 6 hours talking to numeorus developers to finally get the problem fixed.
To top all this, the dumb fuck kept telling me that “we have to get this fixed” because, if it didn’t get fixed, he’d get fired.
First of all, he wasn’t doing anything to solve the problem, just bitching about getting fired. Second, if I had any say they would can his ass instantly.
Dumb motherfucker. On the bright side I’m gonna talk to my boss (who I had to page numerous times last night due to this situation) and see if this is billable. I sure as hell hope it is.
This is the THIRD time in the past two weeks where I had to deal with situations where the ITs had no valid backup. In one case the backup tapes they were using were 6 years old. That property lost months worth of data and countless manhours for data reentry because they were too cheap to buy new backup tapes, and these things ain’t that exspensive.
You’d be surprised what you can bring back by mounting a NTFS drive with a Ubuntu Live CD. I recovered a NTFS drive that way the other day… if it was in a windows system as a slave, it’d crash the sucker on boot, and it couldn’t even be ghosted.
We had been begging one of our companies (we’re the parent) to spend 5k to fix their back up solution. But since the agreement says they get to decide when to spend money and how, they ignored us. We think the $28,000 they just spent to recover the data off their server may have been a wake up call. But we still spent way to many hours over Thanksgiving to feel smug about it.
In defense of the poor SOB who screwed up the backups, he may not have been given adequate support by management. I’ve been in that position before. It takes money and time to setup a proper system for backing up a computer. I’ve spent way too many hours dealing with cheap tape drives and poorly designed backup software. You really need a spare system to test restoration procedures. I’ve been burned by backup sets that verified OK and were found to be corrupt when needed. I’ve also had to deal with systems that had a high probability of crashing in the middle of backups.
I undertand that some people are put in situations that they have no control but:
#1. This property has brand spankin new hardware. There is obviously something wrong with the tape backup but, had the dumb motherfucker done a check on the backup tapes when he made them (which, BTW, is part of his daily procedures), they probably would have known something was wrong a while ago and probably got a new backup system in place. The tapes he had were not even readable by the backup system.*
#1.5. I forgot to meantion this in the OP but the guy wasn’t even doing the backups every day, not that it would matter.
#2. We are not a data recovery company. My job as a tech is supposed to be this: A) determine that they need to restore B) confirm with a developer that restoring is the only choice C) after the property restores, check the intergrety of the databases and make sure everything is where it is supposed to go. At no point am I, or my company, responsible for the actual restore or the data on the backup tapes.
#3. The guy pulled the hard drive out while the machine was running. The drive is not hot swappable. You don’t yank drives out of servers while they are running. It’s just a BAD IDEA.
#4. I’ve dealt with this guy before. He is a fucktard.
I deal with this issue about once a week. Most of the time it is straight forward. It takes some time and some data reentry but, usually it is no big deal.
Slee
*My guess about the backups is that the guy probably takes out the tape and then takes it to the metal safe where he uses a big old magnet to secure it to the back wall of the safe. That is just my guess though. He really is a dumbass.