What is the single largest screw up you've ever witnessed at your job?

In My Humble Opinion
81

Yep, I’ve seen the same thing happen twice. The first was up on the North Slope. As I was getting out of the chopper with my hands full the updraft ripped off my ballcap and the main rotor cut it in two. We had to shut down the copter and inspect the blades. Apparently the leading edge is somewhat resilient but the rest isn’t, built with different parameters in mind. Fortunately no damage except to my pride, being the program chief.

Then about a month later we’re in Eagle where the Yukon comes in from Canada, I’m riding with the pilot in a real tight forested area trying to land in a space only feet larger than our copter and we nick a little pine cone coming down. Immediately I can feel a difference in the smoothness of the rotor airflow, it’s a little buffeted. We flew back to camp, did an inspection and we’re going to need an entire new main rotor. Eagle is so far off the beaten path, I think we had to have it flown by another company chopper all the way from Homer but I do remember it was days, probably 3 or 4, before it got there and a master mechanic and help had to come along with it to do the switch. There’s understandably some real precision tinkering that must be done to get it just right. It was easily 7 or 8 days before it was good to go again.

I remember Jim, the pilot, was so broke up about the money he’d cost his boss. He was off for a few days, went up to some mountaintop and tore up his pilot’s license. He made up for it by just going back to work and being a model pilot but yeah, I was kinds worried he was going to quit there for awhile. He was a Nam vet, real Zen kind of guy and the mistake really shook him up.

82

While the screw ups that have occurred at my present job are numerous, there’s only one significant one that I was a party to. After an upgrade we had some spare network equipment around. I got approval to appropriate one of the older switches to use for testing equipment in my area. Got everything hooked up and passed the info off to the network engineer to do the configuration. Somehow instead of logging into my switch to do a factory reset on it he managed to be connected to the core switch for the entire facility. Oops. Everything was down for about 30 minutes while he got the backup restored. Fortunately we hadn’t moved to the VOIP phone system yet or there would have been no phones either.

If not for the timing and use of forklifts instead of clamp trucks I’d accuse you of being my neighbor.

I was working at a paper plant and the warehouse had stacks of bulk rolled paper. One of the drivers managed to hit a water line causing tons of paper to be ruined and then doing it again a few weeks later.

83

I nominate her for the Marley23 award!

84

Now there was a guy that was hired the same week as me, kind of a nerdy professorial type that was a geophysicist specializing in AVO and processing and we went through, make that started, a training program together. Problem was there also was this absolutely stunning 28 year old, super fit, busty blonde, half his age that began with us too. The professor took one look at her, and she did dress to kill, and he lost his everlovin’ mind! He had this strange nasaly voice, used to rub his hands together when he talked and I had to tell him to cool it a couple of times when his comments became inappropriate. Apparently he kept talking when I was away because just a few days into this new job he was encouraged to find suitable employment elsewhere.

Ah professor, we hardly knew ye.

85

Back in the day when PC’s were becoming fairly common office equipment I worked at a medium sized bank. I was just getting pretty good at DOS and had been introduced to wildcard characters when my boss asked me to clean up a suddirectory on a bank hard drive (call it the oops directory). I noticed that all of the files that were to be deleted began with that letter “O”. Being eager to use the wildcard feature and same lots of time deleting the files, instead of “del c:\oops\o*." (delete all files starting with “O” in the directory “oops”) I typed on the command line "Del c:\o.*” (delete all files starting with “O” on the entire drive. What I didn’t know was that the all of the bank’s bank overdraft files for the last 5 years were stored on that drive. “Overdraft” starts with the letter “O”, as did all of the overdraft files on the computer. I erased them all!

I didn’t get fired primarily because my bosss had talked his boss into storing the files on a PC instead of the mainframe on the promise that the PC files would be backed up daily. My boss never got around to setting up the backup system. If he fired me, he would have to tell his boss why and would likely have joined me in the unemployment line. Instead we called in some favors from a friend in IT who introduced us to his copy of Norton Utilities. We managed to recover about 85% of the files.

86

I used to do computer programming and support for a county 911 agency and have some fun stories to share. For context, you need to know it was a large, consolidated county agency which means we served all the cities and towns in the county as well as the rural areas. The police dispatchers for the largest city had to keep track of 30 - 40 cop cars on average, so they were just a little beyond the point where they could work on paper (meaning, the computer was pretty critical).

  1. Shockingly, the county supervisors (bigwigs) never considered 911 to be an important service (as compared to say welfare and family services, which were in a huge, palatial new facility) so we were in the basement of the downtown administration building. For whatever boneheaded reason, they deemed the dispatcher’s room not important enough to be attached to the backup power generator. The computer room which was located on the other side of the building from the dispatchers had backup power to run for a couple hours. More than once while I was there those darlin’ little machines got to run happily along on full power while their users were sitting in pitch black, scrambling for flashlights and cell phones to call the cops and tell them why none of the dispatchers were responding to the radios anymore.

  2. Another time city workers were doing some road repairs in front of the admin building. Someone in a backhoe hit a gas line, the product of which (apparently natural gas is heavier than air, according to the firemen who responded) flowed down the ramp to the small parking area under our building and next to - you guess it - the 911 center. It’s a very strange experience to evacuate a 911 center. They had trouble getting all the dispatchers out, dedicated bunch of people that they were.

  3. One of my coworkers, a new guy I was training, had the dubious honor of massively corrupting the 911 database while it was running. He was Not Pleased, but I found the whole experience fascinating and marvelous (I enjoy the forensics of such things). The system was a two-server cluster configuration where one server was production and the other one was a backup/failover system. We were preparing for Y2K and I had taught him the command to set the time on both servers simultaneously. He typed it but did NOT hit enter. Instead he hit an abort command key sequence (control Z) that for unknown reasons actually submitted the command instead of aborting it as in the case of every other command. This wasn’t actually his mistake, which is why I never blamed him, it was a really stupid screwup by the OS/System designers. (Later, when I described the event to OS/system experts on the internet, none of them had been aware you could do that, so it’s not even like anybody should have known better. It was not documented anywhere.)

So this 911 system had a robust but home-grown database with designed-in redundancies and checkpoints and every transaction was timestamped to the hundredths of a second. It was a great system; it took quite a lot to make it fall down. He realized right away that his aborted command did not in fact abort, so he yelled at me and I reset the time to the correct. But in the span of 1 minute and some seconds, the database had generated 6 monthly files, checkpointed them all, reset the year and did the end of year jobs, and then died screaming and smoking when we set the date back to “today, right now”. It took us a few hours to restore it from backup tapes, during which I amused myself with the aforementioned forensics and irritated my poor trainee with my glee at what a breathtaking mess it was. Some of the incidents that had been running when this happened had really bizarre timelines.

Oh, and this wasn’t the result of anybody’s screwup, but once we also had a flood in our computer room. All the critical machines were still running, I think we just noticed it because some minor bit of equipment failed and when we opened the floor to troubleshoot we met with an unexpected lake. After we got everything shut down and dried out, we were opening up all the floor tiles to dry it out fully and I came across a high-voltage cable with something plugged into it. Still running power through it. It had grown a beard from the electrical/chemical reactions of being immersed. I’d never seen anything like that before.

87

If it makes you feel better, in the aftermath of Hurricane Katrina, the New Orleans Police Department realized that it might not have been such a good idea to store the physical evidence for thousands of criminal cases in the basement. In a city that is already below sea level :smack:

Made for a fine mess in the courts, since they’re not supposed to send people to prison without evidence (that isn’t contaminated by flood water, sewage, industrial chemicals, and God knows what else).

88

Okay, that clears it up.

89

Years ago a lender had added a 1 in front of a $4 million wire for a settlement that we were closing. I imagine that someone got fired for that $10 million mistake.

90

Not the biggest screw-up ever, but maybe the most embarrassing. I work at a retailer and from time to time the Merchandising department will have sample products left over that they will sell to employees for charity, or just give away, depending on value. A couple of years ago they had a bunch of ladies’ jeans available, and send a corporate-wide email announcing that they had limited numbers and limited sizes available for cheap on a first-come, first-serve basis.

One lady, working in a different building, wanted to know if they had her size on hand before she made the 15-minute walk to Merchandising. So she replied to the email, AND EVERYONE ELSE, asking if they had any in her (generous) size. Which she provided. And let’s just say it was not a size she probably intended to broadcast to all of her co-workers.

91

Not in the millions area but we were flying a BIG job with IR Color aerial film. ( 500 feet by 9.5 inches. ) About $7000. Late 70’s and they did not get cheaper over the years.
Took about two days to pull one roll.

We carried several rolls so we could stay close & get more done each day. He/we in those days used a dark bag and switched rolls by feel.
Cameraman had accidentally included a roll of regular color film in our rolls of additional film… Back then, about $3000. So off we go, as the weather was good for a few days. About a week later we are back home & he sends all the film out for developing. Still had not noticed the switch. To be fair, the labels look almost the same. Got to read them careful.

Film comes back with a note from asking why we did not tell them a roll of color was part of the job. We told them it was all IR Color .

And the weather went down for 2 weeks. Customer, boss, pilot, ( me ) where unhappy. Pilot it the buck stopping place. I had not checked everything. Bad oops…

As an instructor, I let a student crash his own plane because he would not do/learn, what I was trying to teach him. I was fixing to have another instructor try to teach him. The last time I flew as an instructor on short final of it and, he was making a mess and violently shaking the stick, while yelling at me with his head turned back glaring at me. ( His airplane, little 7 AC Champ be bought because that was what he wanted. ) That is small 56 HP tandem seat fabric plane with a tail wheel similar to a Piper Cup for you non airplane enthusiast types.

We were going too slow while too high, way left of center line near the bob wire airport boundary. I was hoping he would do a better job than fighting over the stick at this point. Fighting over the controls at this point was a real bad idea.

I was reasonably sure I could keep us from dying but the airplane with him in control was gonna get broke. His airplane.

Crossed my arms, did not say anything, and let him crash it. Didn’t do too much damage considering that it flopped over that fence and broke a strut, has two per side… People ran out and helped get it on all 3 wheels and shove it into an empty hanger. FAA was not liked much on that airport back then. He had no insurance, no fiance involved, he was just going to get a local to fix it.

That proved I was not an primary instructor and I did not enjoy it anyway so I never flew as in instructor. I was no good at it. My older Sister was great at it.

Me, I knew then that I was should continue as an instructor.

Helped a few people over the years in twins, instrument flying and tail wheel control.

Now old
Not dead
Not killed anyone with an airplane
Wish I had nuff $$$ to fly PVT now…

92

No, in some cases, you actually do need a big red button to shut everything down!

For example, I was working in the main data center for a nationwide chain of banks. This was back in mainframe computer days, and the center occupied nearly a whole block-square floor. And right by the door was a big red switch on the wall clearly marked “Emergency Shutoff”. It was designed to be hit by the computer operators, as they were running out of the building in a fire emergency. Because there was an automatic fire suppression system that pumped Halon gas into the center, replacing the oxygen that a fire needed to burn – or that humans needed to breath! And there was only a small percent difference between the amount needed to suffocate a fire and what would suffocate humans. Thus the location of this switch just beside the exit door.

And, Yes, it too got hit once. But not by accident: an auditor was there, checking out the data center procedures, and wondered if this emergency shutoff worked – so he pushed it!

And it worked.
But 1980’s mainframe computers don’t like an abrupt shutoff of all their power, including their cooling systems. They don’t turn back on quite as easily – took about 24 hours to get them back working again.

Next time I was there, the big red switch was still there, but now behind a pane of glass, with a hammer chained next to it.

93

A secretary at an insurance company had typed out the contract for a fire/business interruption policy on the headquarters building of a national bank chain. And the deductible of one million dollars she had typed as one thousand (probably a much more common deductible amount). And it was quite clear, both in digits (“$1,000”) and words (“one thousand dollars”). And none of the big-shot lawyers or executives who reviewed that before signing, from both the bank & the insurance company, noticed.

Nobody noticed, until a year or so later, when the bank headquarters was destroyed in a Thanksgiving Day fire. Then everybody noticed the text of that contract.

I understand the bank & the insurer eventually settled out of court by ‘splitting the difference’, So it was a $500,000 mistake. And the secretary was not fired – she claimed the notes she had been given were not clear on the amount, and that it was reviewed by all those people. (Plus the bank would have given her a job immediately if the insurance company had fired her.)

94

or a sunhat

This thread is making me cringe all over.

95

Temple Grandin has said many times that a person with autism wouldn’t have placed the Fukushima power plant right next to the ocean, either.

96

One I saw many years ago at my dad’s work: they had a lab on the third floor of their building which needed some plumbing work done. The plumber finished up the job late one evening and when everyone arrived the next morning it was raining on the floors below the lab!

To get around a support pillar, rather than cutting the copper pipping and using elbow joins the plumber had just bent the metal pipe to go around the obstruction. He must have thought that if the bends were less than a right-angle there would be enough room for water to get through the pipe.

Of course the pipe burst and spent the night pumping water into the floor/ceiling space.

97

These stories are bizarre to read as I just watched a documentary on Netflix about two Huey chopper pilots in the Vietnam War who had to rescue a bunch of ARVN soldiers trapped in a bamboo grove by an overwhelming number of NVA troops. There was no clear spot, so the choppers simply dropped straight down, chopping a hole in the bamboo with their rotors. They made several flights in and out to get all of the troops out.

And then a lowly hat came along…

98

Speaking of mainframes - this was 1978, S370-168.

The term “IPL” (Initial Program Load) morphed into “Boot” when micros came along. In the case of the S370, there was (not huge) red button on the face labelled “IPL”.
3rd shift was noted for screwing up (the lead op’s favorite expression was “It’s not my job”), but this time they out-did themselves by playing Frisbee with a write-protect ring for a tape (I still have one of these).
Yep, re-booted the thing just as people were arriving at work and trying to log on.

The other great screw up from that shop was personal. I seem to have been the only straight in the shop who knew that two of the fellows were gay and another was bi.
The bi worked closely with a woman who had just been dumped by her middle-aged asshole boy friend.
They started dating.
One of the gays was the guy who got called when things went seriously wrong.
She called him - and her boyfriend answered the phone.

Did I mention she was seriously homophobic?

Gawd, I loved it…

Apparently, she could not find another job, because she and her now ex continued to share a tiny office - she had to walk past him to get to her desk.
Then I announced that I was moving to San Francisco.
Poor child - I do feel sorry for her. Hee hee.

99

I’ve deleted important databases before, with no backup available.

My favorite screwup that I did was when a client asked a really stupid question to a number of people in my company and I, feeling incredibly smart, decided to Reply All (deleting out the clients email address, of course) and explain to my co-workers just how stupid the question was.

Of course, I didn’t actually delete the clients email address. :smack:

100

Ah, there are so many. Might as well confess MY worst one. I did software support for a company processing scientific data. I was tasked to integrate a new type of data filter that was supposed to mix several bits of input data into a single output. It in fact did that, but because of a single mistyped array index, what it actually output was not the mixed data, but one of the inputs. So rather than enhancing the data, it simply threw half of it away. The difference was not visibly obvious in the output. Worse yet, because of the importance of treating every bit of data the same way, once I discovered it… I couldn’t fix it because it would introduce a “glitch” between the old and new data. It remained unfixed until the next major revision of the program.

Oooops.

Oh, and usetdobe, this was on… you guessed it… a 370/168