Biggest fuck up you ever made at work?

Miscellaneous and Personal Stuff I Must Share
121

A guy that I work with was cleaning the bearings on a laser stage on a stepper, a multimillion dollar machine used to make computer chips. He noticed that a part of a mirror had broken off, so he did the appropriate thing and freaked out–the mirrors cost around $200,000; they’re made out of large chunks of quartz and polished to a flatness of a few ten millionths of an inch. Setting them up is one of the major pains in the ass for our part of the industry. He called our rep who said “Did you find another job for yourself when you were in there?” Money was tight, so we tried for a day to get the tool to work with the broken mirror. After that, we gave up and bought the part. It took several weeks to replace the part and requalify the tool. Oh yeah, my coworker had a reputation of being rough on tools, and and all of this was done under the watch of a new department head that was proud of his reputation as a hardass.

The guy swore that he didn’t do anything, and I’m pretty sure that he was right. I had bumped the mirror in the same place two years before, and I think that it took that long for the crack to propagate in the quartz (The tool had been acting worse over the last six months in a manner that backs me up.) I’m not sure which was worse, to have made a $200K screwup, or to be the flake that, when something broke when I was nowhere around and the “cowboy” of the department was doing the work, took responsibility because they bumped something years before.

122

Botched a Novell Netware upgrade which resulted in every administrator and faculty account password in the college being set to null. Then the exciting panic of deciding whether I ask them all to make a new password as soon as possible, or generating random new passwords for them and having to explain why they had changed, all while hoping nobody stumbles across the situation before the accounts are protected again…

123

Really? I will have to go see if I can dig up any recent screen shots of what it looks like now.
I still remember the banner (with ROSCOE in large letters) coming up on the screen. ASCII art at its finest!

124

A long time ago while I was working for an asbestos removal company at a closed power plant in Carlsbad NM I cut an energized cable in a tray that was labeled incorrectly. This caused an overload somewhere that resulted in power being cut to all of Carlsbad, White city, Loving, Artesia and so on all the way to El Paso. I have no idea how much money it cost to bring the plant back on line or to all the businesses that lost power for 3-4 hours. I do however know that workman’s comp paid over 25 grand for the hospital bill to fix my irregular heartbeat and treat me for the burns and concussion. My buddy who was standing outside of the manhole enjoying the sunny afternoon said the flash was so bright he saw the light come out of the manhole like a flash on a camera.

125

My first job out of the USAF was as security manager for a new resort community.

They had just dug the ground for a new lake, and I decided to ride around in the lake bed with our jeep.

I knew I’d be fine because the vehicle had 4 wheel drive.

Yeah, * if* at least one of those 4 wheels can get purchase, and certainly not if all four are sunk in silt.

Cost me a week’s pay.

The other was when I worked for Sherwin-Williams. I was supposed to mix up twenty gallons of Tile-Clad floor enamel for a new shopping mall and gave them Metal Latex instead. The painters didn’t notice and put on the Metal Latex anyway.

Insurance covered that one.

But I did get the color right! :smiley:

Quasi

126

System admin dum-dums…

  1. A user quits, their files are copied to another user & ownership change to that remaining user. To change the ownership, you “chown” the files.

cd /home/username ; chown <username> 777 -R * .*

so - the first “" wildcard matches all regular files, the ".” is supposed to match “hidden” files which always start with a “.” character.

Well, “.*” has a wildcard that matches “…” (dot-dot). “…” is unix shorthand for “the directory above me”. The parent directory of /home/username is /home and /home contains all user directories. The “-R” says “recurse”. The result is that the command automatically progresses up to the parent level then recurses down through every user on the system - locking most of them out of their files.

  1. “kill -9 -1”

A shorthand command for “kill all the processes I own”. Safe enough if you want to kill all processes for a particular user - first you “become” the user, then you “kill -9 -1” and all their process die.

To become the user, first you become the “root” user, the master machine account, then you become the user. You can become the user because you’re root and root isn’t forbidden any action.

Problem, of course, is when you “kill -9 -1” before that second step. The command kills all the processes that run on that system. The server dropped like a stone and took the entire phone network monitoring center for Kansas offline.

  1. The last one wasn’t me directly but it was the people I supervised. My subordinates monitored the phone network equipment for all of the Bell system in Kansas. One day we lost power in our primary Wichita office and we dropped to battery power.

Now a battery for an equipment bank this size is a round “cell” that’s about three feet high and a foot and a half across. It’s plates of lead bathed in sulfuric acid that delivers 1.5 volts with enough amperage to weld with. We’ve got banks of these wired in series to got to the ~50v that’s necessary to run a phone system.

When you drop to battery power, they of course start to discharge. At a certain voltage level, alarms go off and you’re supposed to start a generator to take over. My subs were the ones to get the alarm.

The missed it.

The network ran on the batteries so long the batteries ran to zero volts and then “flipped” - they kind-of slingshot past 0 and become slightly negative. They had become so hot that the normally clear casings had turned white and started to bulge because the plastic softened.

And, of course, most of Wichita Kansas had no phone service.

Every battery had to be replaced - not sure the dollar cost on it but I’m sure it’s thousands per battery.

127

So YOU were really the Wichita Lineman!

Well, somebody had to say it.

128

Yep! But I’m no longer on the line.

I sell pens and pencils now.

129

Just popping in to say how cool it is that y’all are talking about this! Will there be a judging at the end, or are we just gonna say to ourselves: "Well, hell! At least I didn’t do that! :slight_smile:

Quasi

130

I brought down the internet for an entire state for about a day…

I missed one tiny configuration parameter and boom.

They were surprisingly forgiving, and told me it was a mistake anyone could have made and about 6 levels of review had missed it, but… ooops.

131

A long, long time ago, I had a job as a computer operator in a bank. I managed to screw up and totally destroy a complete days worth of transactions.

132

Explosions are nothing. As a physics researcher, I nearly blew out an eardrum with an implosion. I was pumping down a vacuum chamber when an half inch thick piece of an acrylic window shattered while my ear was about four inches away from it. I couldn’t hear right for about a day and a half.

A week later it happened again.

133

Nah. Given the current economic conditions, some of us are going to be laid off and wonder afterwards if we reminded someone of why we should be on the layoff list.

134

sad smile Yeah, you’re right.

I apologize if I came off sounding flippant.:frowning:

Q

135

That reminds me of a story from the same company that published the “pubic investigation.” The managing editor of the business magazine was famous for letting her subordinates do all the work and never, ever checking the proofs. Sure, the pages were supposed to be perfect by the time she gave them a final look-see, but nonetheless it was her responsibility to review them one more time (and mistakes being as ineradicable as they are, the managing editors who did their jobs not infrequently did find errors).

Knowing that she signed off on everything without reading it, her team actually prepared TWO versions of one of the magazine’s sections - a correct one, and one that had a large headline proclaiming “The Managing Editor of This Magazine Is an Asshat.” They gave her the asshat version, and sure enough she signed off on it.

No, the wrong version didn’t get printed - the pranksters weren’t stupid enough to be so careless. But for years afterward, that editor was known behind her back as “Miss Asshat.”

136

Mine’s quite boring actually. My first job when I was 16 was at a department store and as I was bagging up someone’s purchase, I also threw in the industrial stapler from the counter. They were nice enough to bring it back and my manager just got a chuckle out of it.

137

This is one of the most interesting and entertaining posts I’ve read here. Some of your stories made me cringe.

For whatever reason I found the one below so funny, I laughed until I cried. A bucket of chili? How long did that take to clean?

138

Before I got married and moved back to Texas, I worked in a cutlery shop while I was in college. One day a customer was buying a knife or something that probably cost less than $20, and paid with a hundred-dollar bill. I counted out his $80-plus in change and handed it to him.

Along with his original hundred-dollar bill.

Whoever you are, Mr. Knife Buyer, I hope you enjoyed your ill-gotten money and your free knife, because I got fired over it. I’m not saying I didn’t deserve to be fired - that was a colossally stupid thing to do - but at least it was an accident. You kept money you know you didn’t have a right to, and you did it on purpose. What kind of dishonest asshole would do something like that?

139

The kind with a pocket full of money and a free knife.

140

I can’t believe you’d get fired over this… at least the first time. So the store lost a knife and $80… surely it costs more to hire someone new.