What is the single largest screw up you've ever witnessed at your job?

In My Humble Opinion
21

A colleague of mine sent out an e-mail to 700-ish students with an attachment – an Excel spreadsheet with the names of every faculty and staff member. The purpose was to facilitate submitting nominations for an award. What the colleague didn’t realize was that sheet 2 of the file contained birthdates and SSNs of every faculty and staff member. Realizing her error, she then sent a follow-up e-mail to students saying, “Don’t open that attachment,” thereby ensuring that every student will open the attachment. Identity theft panic ensues and big bucks are spent on credit monitoring for all of us for a year.

Hmmm. I wonder if anyone did have any problems as a result of that data breech. I know I didn’t.

22

This was back in the late 1980s, when Intel 286 chip PC’s were pretty much what you had. The company I worked for had a software package marketed to the newspaper industry. Potential customers were informed that they could purchase the hardware from my company or, for a hefty per-machine fee, could have their own equipment “assessed” for compatibility.

To my knowledge only one company ever elected to pay for this assessment; on about 8 PC’s. You can probably guess where this is going. Whoever was assigned to verify the customer equipment evidently didn’t. When I arrived on site for the system install not a single PC worked correctly. Within a few days I was recalled and my understanding is that the customer received their assessment fee back and my company provided the hardware gratis. I do not know if anyone was ever punished for that failure, but the financial hit was probably around $30,000.

23

I work for the US Government. I am currently awaiting notification if I am one of the lucky folks whose personal information was looted (apparently by the Chinese).

Thing is, we get classes (mandated by the US Government) every year telling us how important it is that we protect of Personal information…

24

A Big Healthcare Provider (acronymically known as ABHP) issued a new iMac to somebody in the art department. Due to an excess of team spirit, this enterprising person decided he wanted to fly the company banner in every way possible so he went into the computer setup and renamed his computer to ABHP.

It went out onto the network and introduced itself as ABHP, and very quickly met up with ABHP, the Primary Domain Controller. Like David versus mighty Goliath, our plucky little iMac insisted that it was, in fact, ABHP. A turf war ensued and the entire corporate network went down for a couple hours.

Luckily, this wasn’t the first time this had happened, so they were able to set things right pretty quickly.

25

Not me, but a guy I knew when I lived in Seoul:

His company was negotiating a Very Important Real Estate Deal, worth multi-millions of dollars, with a firm in Hong Kong. He was supposed to fax them an offer of some sort… however instead of that, he somehow managed to fax them his company’s ENTIRE negotiations strategy.

Oops.

26

I witnessed the following. A mainframe implementation at a very large financial institution has gone wrong, resulting in many thousands of incorrect transactions being written to one of the larger databases. Ordinarily, while not ideal, this isn’t too much of a problem, just roll the database back prior to the update, fix the code, and reapply the transactions. It’s also Saturday morning, so there’s no significant user downtime.

So the senior guy on site, one of the guru-level greybeards, goes off to the operators console and starts the rollback. One tiny problem, it is early in January, and while he gets the day and time of the rollback position correct, he’s forgotten that it is now 1996, and not 1995, so unbeknown to us the database is now attempting a rollback of over a year.

Anyway, we are sitting about having a coffee waiting for the task to complete. It has got to the stage where people are muttering “Hmmm, this is taking a lot longer than I thought it would” when the aforementioned greybeard emits a strangulated moan, like a wounded animal, and jumps up and sprints to operator’s room, swearing profusely.

We managed to get the db back to the proper state at about 6 am on the Monday morning. The big boss, being an old mainframe hand himself, took it surprisingly well, and he even authorised the overtime payments for all of us who were involved, except one.

27

I was doing prep work with another tech for an upgrade on a 5ESS switch(40K+ telephone customers). We were following the step-by-step instructions…and managed to skip a page in the manual. We powered down half of the admin module out of sequence and took the office off the air for about 20 minutes. Luckily it was 11:30 at night and about 95% of our customers were businesses that weren’t open.

28

When I was working at the university I did this one, although it wasn’t exactly *my *mistake: http://boards.straightdope.com/sdmb/showpost.php?p=13730452&postcount=9

In USAF and the airlines I’ve heard about countless screw-ups, some fatal, but haven’t personally witnessed anything too severe. This is about the worst I’ve seen with my own eyes:

As you’ve probably noticed, at most hub airports there’s a continuous stream of catering trucks & baggage tugs towing 2-5 carts driving back and forth just behind all the parked jets. There’s usually even painted stripes forming what we call the “truck road” there to keep everybody to a predictable and organized path.

So one day we push back across the truck road as always. The ground crew disconnects the tug & tow bar from us per normal. And without looking, the tug driver drops it into reverse, stomps on it, and runs his 80,000 pound (40 ton) tug into the side of a fast-moving train of 4 baggage carts passing behind him in the truck road.

He hits the thing about in the middle, knocking 3 of 4 carts over and scattering hundreds of suitcases. The tug largely crushes the 3rd cart and a bunch of bags as well. The fourth cart ends up on its back 50 feet away. Fortunately the bag tug didn’t get hit or overturned, so nobody got hurt. Scared the heck out of the bag tug driver though. We had a ringside seat for this one.

A few weeks later a different tug driver at the same station ran over a man and killed him instantly. Two good long time workers & friends. A moment’s inattention and bang. Fortunately I wasn’t there to watch that one.

29

Another one, but I’ve been trying to distill it down. Story told before.

One weekend, my factory had a batch that wouldn’t start; it ended up on the floor, coagulated into a thigh-high, sticky sponge. This was on Saturday; cleanup was finished by Monday lunchtime. Tuesday, the official investigation begins. The interim factory manager was worried because we’d broken protocol in that at the point where it said to call the person On Call so he’d confirm the order to drop it, we saw it was someone who wouldn’t confirm and, uhm, suffered a horrible telephone malfunction. She wasn’t worried that we’d broken protocol (she knew as well as we did that the guy was flakey), she was worried about how to explain it.

And then she got this mail that another factory had been making the same product with what turned out to be a faulty batch of raw material, but it had been on Monday and with everybody there. So they’d added trigger component… again… again… called the process engineer for confirmation… and he’d denied it. “No, no, add more trigger!” And again. And again. And still the reaction wouldn’t start. By this time, most of management is saying “keep adding”, the warehouse manager and a senior lab tech get themselves written up for protesting that it’s a completely stupid thing to do. Eventually the reaction starts and runs almost instantaneously, leading to the same uncuttable sponge that we had on the floor - but in their case, it’s inside the reactor. Half a dozen people fired, over two weeks of cleanup.

When you’re migrating databases, an important thing to keep in mind is that cleaning the data before you load it takes a lot less work than doing it after. If one of the consultants proposes to do it later, please fire his ass with complete prejudice: he’s an imbecile or a leech. I’ve been in charge of the “later” cleanup several times: stuff that can be done beforehand in a couple of weeks (to 97% completion or better) ends up taking six to nine months. “We’ll clean it up later” is a separate project in its own right.

30

I once deleted the phone records for a certain state for the month. I wrote a very simple and untested script for prod that had a wee bit of a bug in it.

Fortunately it was mitigated by

  1. The fact that the month up to that point consisted of 2 hours in the middle of the night.
  2. The only guy who discovered it owed me some big time favors and no one else ever found out.

We did a monthly full backup at midnight on the last day of every month. As standard practice we ran all clean-up scripts immediately after the back-up, in case bad things happened. Well one month, on ones state’s database, my clean up script cleaned up every damn thing in the database. The Sysadmin who had the job of getting up at 2: AM and verifying the backup kicked off the normal status scripts and noticed that one database was .00001% of the normal size. As I was DBA on duty, he gave me a call to investigate.After figuring out the issue, The best way to minimize impact was to restore the database back to its backed up state immediately, losing all new records from midnight till 3:AM. We can up with some,as I think about it now, highly implausible, story that the database never entered a write-transactional state after the back-up, and was finally given a hard reboot at 3:AM. Amazingly after the initial bullshit incident report was submitted, nobody ever questioned or even mentioned it again.

31

I work in pharma, and have witnessed a mistake which ended up costing the company 6+ million.

Not a good day for that guy or his department.

32

Our Nation Wide cable/internet provider strung a very, very important fiber optic line across the county recycling area. At about 12 feet high.

The trucks that take the roll offs for recycling get much higher than that when pulling the roll offs on and off. It seems that one of those trucks broke the line. Don’t know for sure.

This took all internet and cell phone communication down for the entire county for 8 hours. You want to use a credit card, use your cell phone or computer? Good luck. Here’s a stamp.

I can see the fiber cable from my office window. They did not put a new poll in place to help prevent this in the future.

They did, however, throw and attach two orange safety vests on it so people will see it. It’s been that way for months. :rolleyes:

33

Not nearly as bad as some of the others here, but:

Many years ago, I worked at a conference hotel (I was just a lowly conference porter at the time). One morning, we started to get what would turn out to be about 80 or so delegates arriving for a conference that we knew nothing about. It turns out that the sales agent that took and confirmed the booking forgot to enter it into the booking system. Or charge for it. Or, you know, tell anyone about it.
Fortunately, we had the conference room available, so it was ‘all hands on deck’ while we set up the room in record time and the restaurant staff put together teas and coffees for the delegates while they waited. Apart from a few vague complaints about the wait, I don’t think anyone from the conference noticed the cock up.

More recently, I was setting up a fairly simple backup job for our company server, using robocopy. It was only once I’d run the first test that I realised I’d mixed up the source and destination - meaning that rather than backing up the server to the backup disk, I backed up the backup disk to the server. As it had the /purge switch, it neatly (and quickly) erased everything from the server that wasn’t on the backup disk - which was pretty much everything. :eek:
Luckily, this was only a secondary backup (believe it or not, I’m quite paranoid about backups), so I was able to restore from the main backup and nothing was actually lost.

34

I was building something for NASA that would go into the shuttle mission simulators at Johnson Space Center. A critical component required large, very pure sheets of glass for use as spherical mirror/beamsplitters. None of the manufacturers stocked glass I could use for this but a large German glass manufacturer agreed to make them custom. (It’s amazing what people will do for you when you tell them that it’s going to go into training astronauts.) But as the crate containing the first four pieces was being loaded into the shipping container, it fell off the forklift, breaking all but one piece. I had the unpleasant task of calling NASA to let them know that delivery of my systems would be delayed by a couple of months. (Fortunately, I was never responsible for delaying a shuttle mission.)

35

Someone at my current place of work accidentally sent £40,000 (approx $60,000) worth of perfectly good stock to be destroyed (it was supposed to be a different product and a much smaller quantity).
There were some tense words spoken, but no long term repercussions. Weird.

36

When I first started working in Eastern Airlines reservations, I booked someone to fly from BOS-PDX, instead of BOS-PWM. I realized it later that night when I got home. I couldn’t remember the name of the customer so I could pull it up, and call them. Hopefully, they realized it at the airport before flying out. :eek:

Neither I, nor the customer even realized it when I recapped the information after reserving it. I can’t figure out why, since there was several hours difference in flying time. Even the connection in ORD didn’t alert us. :smack:

37

I used to work in retail software - travel sector.

We had a client, a traveler’s club, who engaged us to create an OEM version of our software for them to send to their members.

To develop the software, they sent us a whole lot of content that was a year old to play with.

Later, when their newer content was ready and our software was ready, they sent us new content.

But our database guy just delivered the year-old content to the development team again. We in dev didn’t catch that no data had changed (we didn’t have a direct way, other than had it occurred to us to bump the old data set against the new to make sure there were differences), and QA didn’t catch that no data had changed (they only had a year-old dead tree version of the content from the client, and didn’t make any waves about the fact that they never happened upon any changes in the data). The product shipped like that.

The client wasn’t happy.

38

A colleague got a needle stick injury while working with a particularly nasty virus. She ended up in quarantine for a few weeks but was fine in the end.

I’ve seen more than one dive boat captain turn a customer’s tank valve to closed position just as the customer was entering the water.

39

Years ago I was supervising design on a video game that was being developed by a third-party developer. We were paying, they were doing the work, and it was my job to make sure that what they were making was fun.

They had the basic mechanics in place and they were supposed to be building out the missions, but for some reason none of the missions seemed to ever make it to the point where they were playable. They’d get a batch halfway done – enemies placed, some basic scripting – and then stop and move on to the next batch.

This was a problem, of course, because there was no way to evaluate how the game would play with the missions all half-done. At first we weren’t too worried – sometimes dependencies in game development mean work has to be put on hold – but by the time they had half the game in this incomplete state we really alarmed. The fact that the team was really secretive and touchy about their creative freedom didn’t help matters.

Eventually we forced them into a production audit on threat of cancellation. A senior producer and I went in and spent two days combing through their assets and interviewing key members of the team to find out why the game wasn’t making progress.

The revelation came halfway through the first day. We were talking to one of the mission designers. “Why aren’t the missions getting done?” we asked him. “It’s the programmers’ fault,” he told us. “We can’t script missions until they write the AI.”

Okay, let’s talk to the AI programmer then. “Why aren’t the missions getting done?” we asked him. “It’s the designers fault,” he told us. “We can’t program the AI until they give us a design for it.”

It turned out that literally no one on the team had been tasked with designing enemy behavior. They’d been building a game for more than a year without anyone actually working on gameplay.

We cancelled the project.

Here’s another one. Years ago I designed a first person shooter about hostage rescue. During development we amassed a bunch of reference art for real-world hostage and terrorist situations. We didn’t use any of these real world scenarios in the game – that would have been creepy – but we used it as visual and design reference for the fictional scenarios we created.

When it came time to release the game we gave the marketing company a huge dump of art from the project – screenshots, 3-D models, and all of our reference art. Except no one explained to the marketing company where these images of real-world events had come from.

The marketing company created a bunch of sell sheets for the game and we started sending them out to the media. Then one of the junior designers walked into my office holding a sell sheet. “Does this building look familiar?” he said. The image was of a gun-wielding bad guy on the roof of a building. There were sniper cross-hairs superimposed over him. But there was something distinctive about the building’s roof line.

It turned out it was the Lorraine Motel where Martin Luther King Jr. was assassinated. They’d pulled an image from our reference art collection showing James Earl Ray’s perspective from his sniping point, and Photoshopped in a person getting sniped on the roof. Fortunately only a few hundred sell sheets had been distributed. The original plan had been to use the same image in an ad in PC Gamer magazine … .

40

Of course I wasn’t there and don’t know the type of helicopter but I flew Hueys in the Army for five years. I seriously doubt the hat bent the rotor blade.