Today I royally fucked up at work. Commiserate with me

Miscellaneous and Personal Stuff I Must Share
1

I manage a small IT infrastructure team. We support, among other users, a development team that’s responsible for the application and database code that’s our primary product offering.

Due to a minor screwup earlier today, it was requested that we perform a full restore of their development server to last night’s backup. This server contains working copies of all of the stuff that’s at the core of the business, plus the version control system that keeps track of of changes to those working files.

Normally I’d foist a job like this off on to my staff, but I have a couple of new people on the team and decided that I’d better run through the procedure with them looking over my shoulder.

So I logged in to the storage system, and pointed to spot where we keep the nightly snapshots of the development system.

“See these snapshot files? With the names like dev20131201, 02, 03? We want dev20131204. I’m gonna select it and hit restore.”

And I did. The system flashed a warning message on the screen:

“All changes made after this snapshot copy was created will be lost.”

“No biggie,” I said. “Those are the changes we want to blow away.” I clicked OK.

Except those were not the only changes the system was referring to. It was also talking about destroying all of the subsequent snapshots since the one I had selected to restore from.

Of course, since I picked last night’s snapshot, there weren’t any subsequent snapshots to worry about, right?

Wrong. I hadn’t picked last night, 20131204. In my distraction while explaining the process to the team, I picked 20131104 - last month’s.

The system happily reset itself to a month ago, destroying any work that had been performed since. Eight developers. One month’s worth of effort. And all the backups of it.

Now, we might be able to recover some of this. People have local working copies of the stuff that’s in active development, and we have a good record of changes that have been deployed to production in the interim.

There’s also the fact that this system was set up exactly as my peer, the Development manager, specified. We asked him about offsite backups, etc, etc, and he didn’t think we needed any (because nobody anticipated anybody would be so stupid as to blow away a month’s worth of work.)

And he has a bit of a habit of being a bit run and gun with his change management process - his team doesn’t log their change requests in any centralized control system, instead relying on CVS to be self documenting.

And I’ve been the strongest advocate in the company against putting all of our eggs in one basket this way, thanks to previous screwups (mostly not my team’s fault) that caused actual customer-impacting problems, for which I’ve been met with nothing but resistance from the same peer who wants everything wrapped up in one nice, neat, monolithic, fragile package.

But still, I fucked up a month’s worth of productivity for eight people thanks to two seconds of inattention. So we’re going to spend probably the next week painfully retracing our steps for the past month before we can even start recreating any of the work. Thanks to a dumb cowboy mistake.

I just wrote my apology speech, accepting full responsibility - none of that Mistakes Were Made shit. I called the CEO and delivered the short version over the phone (he seems both pissed and bemused, which is good) and I’m going to deliver the full version to both teams tomorrow before we get down to business.

But first, I’m gonna drink myself to sleep and take comfort in your fuckups. Please share.

2

Oh boy, I’ve never fucked up that bad. Jeez.
Just kidding! This didn’t happen at work, but it did happen:
I once left a party after getting backout drunk on whiskey. Driving myself home, I blew off a stop sign, and t-boned a car that was going probably 40mph. Spun him around and knocked him into the other side ditch. Luckily, somehow, he wasn’t hurt, and neither was I. Both cars totalled. I sure dodged a bullet that day. ( That day LONG ago. )
You didn’t fuck up so bad.

3

I once missed seeing half of the tests that the office photocopied for us, so that when we distributed the tests to the 300 person class, who were taking their final exam at 7 pm when the office was already closed, half of them didn’t get copies.

It took 45 minutes of getting the document, sending one person to Kinko’s to photocopy and trying to make sure no one was talking (even the people who didn’t have the test, because half of the room already did) before it got sorted out.

And then the legal threats from PARENTS came in the next day, the prof met with the department head, etc. etc.

This was my first semester of grad school. At least it never got any worse than that.

4

I commiserate. Back when I was still a kid, some time in the 90s, I, as an outside developer, was sent to a company that was doing clinical trials to copy their Access database over, do something with it, then discard it. I come in the morning, copy it over to another disk, futz around with it, then delete the copy. Then I go back to the original computer and to my horror find that instead of copying it I moved it, and there is nothing left.

With trepidation I turn to the guy sitting next to me and ask him if they have any backups. “What backups?” is the answer. I swear I felt my heart stop and sink into my feet.

Thankfully, the database was used to send out stuff to their clients, and there were records of what was sent out. So I spent the next 16-18 hours straight writing something that would read in those records, reformat them to the original database specs and create a new database from it. And testing it. And retesting it…

In the morning the client didn’t know what happened. Don’t think they know to this day. My boss knew though. Oh boy he knew… But there were no repercussions since I “fixed” it.

5

Raise your hand if you’ve ever forgotten the “WHERE” clause in an UPDATE statement in SQL. And then died.

6

This isn’t a fuck up like yours, but I was plenty embarrassed at the time! I was about 20 and working my way through college waitressing at a pizza joint. I had the banquet room with a birthday party of I’d say about 15 30-something guys. They drank a LOT of beer. Our restaurant had this deal where the birthday person would get a small free pizza. When I brought him his, the guys started saying, “Give him a kiss! Come on!” Completely oblivious to what I was saying, I replied, “No, but I’ll blow him one”.

I really DID mean I’d blow him a kiss! The room erupted. I was mortified, but held my head up, took all the jokes and got a great tip. :smiley:

7

low whistle That will fuck some shit up.

8

Yikes. That’s why I normally punt to the staff of this kind of thing.

Good point. I nearly drunkenly drove myself and two friends into a two-story excavation for a soon-to-be-built parking garage. I don’t do stuff like that any more.

9

I now type the WHERE clause first, then go back and prepend the update. Learned that lesson the hard way, yeah.

10

This isn’t a work one, but rather a school one, and I still feel a bit guilty to this day (even though the person involved has long forgiven me for it, and I might not technically be culpable, but more a circumstance of bad luck/timing).

Anyhow, I didn’t have a computer in college, so I would check my email on my roommate’s computer. And I had permission to do so, even if she was away. Anyhow, so, one day, I go to check my email. I see she has a Word file open with her final paper being typed up. I know she had been working on it for the last few nights, and she’s something like 15 or 20 pages deep into this. And this was before the days of autosave, so you can see where this is going…

Anyhow, I log into my email account, check my email, and I get the fucking BSOD. What pisses me off is that I had a fleeting thought that said “hey, maybe you should just save that file just in case she hasn’t already.” She returns about a half hour later, as I’m off to work. I explain what happened, she explains she hadn’t saved it yet, and I just shuffle out the door because, well, what more can I say? I was completely mortified. She had to type the paper up from scratch. Ugh. It must have taken ever last bit of restraint for her not to murder me then and there.

Things were very quiet between us through the end of the year (there were only one or two weeks left in our lease, anyway.) I kind of avoided her for the rest of my time, but it seems all has since been forgiven, as she Facebook friended me and we’ve been on good terms.

Now, I suppose in retrospect, I think, who the fuck doesn’t back up there work when they’re 15 pages into a term paper, but, truth be told, I don’t remember being good about that kind of stuff back then, either.

11

back rabbit: You checked the Recycle Bin? Are those VMWare snapshots?

You should try to recover the deleted files:

Not having a second copy is insane. You are also vulnerable to disk crashes.

12

nods sadly

Today in our proprietary database I updated a bunch of new addresses. It went swimmingly according to the upload function (a bear in this database). But I learned something… updating mail addresses wipes clean all phone and email records for the new address record. Thank god it keeps the old address record as an alternate… Cue an hour of fixing that, and lesson learned.

13

<Raises hand>

Eeeep. Oh, the pain!

14

We’re talking volume-level snapshots on a NetApp. Undelete doesn’t exist.

15

Are you sure?

http://www.krollontrack.co.uk/data-recovery/netapp-recovery/

16

How to avoid the UPDATE screw up:


**BEGIN TRAN**

UPDATE mytable SET price = 0

SELECT * FROM mytable

Oh shit! :eek:


**ROLLBACK TRAN**
17

That’s what I do now, too. I’m still always afraid I’ll run something by highlighting it and forget to highlight the WHERE bit. shudder

Good one!

18

I was testing an auto-mail function for our software, and accidentally fed data to it, not from the correct test account but from my personal email.

Anyone who’d sent me mail in the past year and a half got email from our system. One for every mail they’d sent me.

My husband got a LOT of spam that day.

After the dust had settled, my colleagues bought me a can of Spam as a memento of the occasion.

19

Why doesn’t it ignore the snapshot directory and not delete anything from it? I’ve never been in that line of work but it seems like you could never have too many snapshots.

20

It’s my favourite preventative. OTOH, make sure you don’t get distracted by something else and forget the COMMIT, otherwise you’ll spend the afternoon debugging the Work Order module before figuring out that the WorkOrder table is locked for your update. :smack:

I’ve been fortunate enough to just have a string of small disasters, like the time I “temporarily” left an open share on a spare PC and started spamming a few hundred employees with Blaster scans… or managing to lock the keys to the client’s server closet in the server closet… or the emergency DB restore where MSSQL 2000 doesn’t quite do what you expect and basically booby-traps you to overwrite the first DB in the list…
I swear I’m a professional! :o