Make me feel better 
Well, I did, actually. Back in the day, when I was an Air Force computer operator, we had to run a personnel program that ran all evening long after all of the day shift people went home. This was back when you could only run one program at a time on a monster Burroughs 3500 machine, and it had these gigantic tape players with tape reels something like a foot wide. The day shift at Personnel would input data and it would all go on this tape all day long, and those of us in evening shift would take the tape and feed the data into the Personnel program and let it just process all night.
One time I hung the wrong tape. And it processed all night long. We got backed up requiring the midnight shift to have to process the correct tape, which they had no experience doing. Fortunately nobody treated me badly for it, it was just a “oh, that’s too bad, don’t do that again.”
The network admin at my old job pushed a change to everyone’s logon script that rerouted every print job for 35 departments to the president’s office. It was beautiful. He kept his job and kept fucking up. Not so beautiful.
The networking folks had some update to the email system to take live that involved some tweaks to desktop setting as well. They scheduled it for Saturday and sent an email to everyone just to make sure nobody was planning to come in and catch up. They described the changes as “transparent to the user” aside from the short downtime outside of normal business hours.
ISTR it was Tuesday before the last stragglers got their email access back. That was with the changes backed out. “Transparent to the user” became the running joke for a while.
One evening, I got a call from the ICU from a nurse who said a patient had received a morphine overdose, and was given Narcan, and there were some issues with the IV bag we had sent up. I felt sweat breaking out in places I didn’t know I could sweat, because of the possibility that I was responsible, and went upstairs.
She showed me the bag, and not only was it not the size on the label, that bag had been made by our best technician and approved by our best pharmacist. :eek: The patient had probably received 2 1/2 times the morphine they should have! I took the empty bag back to the pharmacy (those people were not there that evening) and followed procedure for documenting errors.
I could tell when the pharmacist found out about it. While he never said anything to me (we all knew each others’ handwriting), that day, he looked like someone had just informed him of a death in the family. Fortunately, the patient recovered fully, but this illustrated to all of us that anyone is capable of such a thing.
One of my fellow nurses was giving a shot, recapped the needle (BIG no-no!) and in the process, stuck herself. The patient was a recent immigrant from Nigeria, the country with the second highest rate of HIV. He refused HIV testing.
That probably ties with the other nurse who decided it would be a good idea to get high with her patient and his housemates, then got really paranoid and called our boss’s brother to come pick her up. But at least she didn’t need to get HIV tests every three months for a year. She just got fired.
This goes back a long time, to the days of large disk drives with demountable disk packs. This was at a well-known US university, with a large data center that had 8 of these disk drives (each roughly the size and shape of a small washing machine).
The night computer operator was supposed to mount a certain disk pack. He dutifully did, but it failed to read properly. So he mounted it in the next drive - where it also failed to read. He continued this 4 more times, until the thought came to him that perhaps the problem lay with the disk pack, not the drives.
Indeed, the disk pack had been damaged (probably was dropped) and as it spun up it destroyed the heads in each of the 6 drives he’d used. Each drive needed a major repair that was said to cost around $6500.
A gust of wind blew my sun hat into the rotating prop of the helicopter that I used to ferry me around out in the field. It was practically vaporized, which was pretty amazing and funny. An hour later, the pilot returns to the site to pick up me and my assistant. His face is grim. Apparently while waiting for us out on the levee, he discovered that one of the prop blades was bent. Who knew that a $10 Walmart hat could cause $20,000 of damages. And the helicopter was out of commission for two weeks, which really fucked up my field season. Doh!
Not a big screw-up in terms of causing major problems, but certainly big news in our company:
An employee sent an email to her husband, who was employed in a different part of the company. It was clearly NOT work-related, but contained slightly risque married-couple suggestions for the evening. Her husband’s name was Allen, and the nickname she had set in her email was Al.
But she typed an extra L, generating ALL. Which some fool in corporate Info Technology had:
- set as code for send to every email address, and
- had left unsecured, so that anyone in the company could send broadcast messages.
So 18,000+ employees, and over a thousand outside suppliers, got this email message.
The company probably took a hit from the work-hours spent discussing this, but no other harm.
This isn’t damaging to anybody but the eyes of people involved and perhaps the ears of IT personnel. My current client initially got my name wrong but hey, at least they corrected it once I explained. In the recent change of mail servers, anybody who’d ever gotten a name change got their original name back. It took about two weeks just to fix that particular and humongous pile of tickets.
At least the email addresses themselves don’t change: when they correct someone’s name they set the old mail to redirect, so both the wrong and corrected addresses work. But I spent a week being asked “what happened with your name?” by my very-confused colleagues (a week because I threw my boss at the mail people - he’s also their boss).
How about a big screw-up I performed personally?
Some government offices and businesses still rely on fax machines so we have one old fax machine tucked away in a back office. There was one time I had to send a fax someplace. The good thing was that I was sending a legitimate work-related document - it wasn’t like I was sending my buddy a Dilbert strip.
Because at one point I mistakenly pushed the red button on the fax machine. Which does transmit the document in the machine. But the red button was supposed to be used for escapes - it was preset to send out faxes to several dozen police agencies in a three state area.
Here’s a bad one, told to me by a friend who was involved in the aftermath:
A large scoreboard in a multi-purpose sports arena needed some minor service (a few bulbs had to be replaced). The normal way this was done was to use built-in electric motors to lower the scoreboard to the floor - but that system had been acting up and was deemed unreliable.
The backup scheme was one of those self-propelled electric scaffold devices that can raise the operator to any chosen height. But this apparently had not been used for a while and was stored in an area where a bunch of other stuff would have to be moved to get at it.
So the choice was simply to fetch a long extension ladder and use this to climb about 40’ to the scoreboard. The flaw in this scheme was the fact that the arena was being prepared for the scheduled evening event: a hockey game - meaning the feet of the ladder were set onto smooth ice. As the worker (the one who’d conceived and organized this scheme) neared the top, the feet started to slide - he fell to the ice, and died from his injuries.
Shortly after starting graduate school, I joined my advisor’s research group. He was very impressive, and I was very lucky to have the opportunity. On the first day working in one of his labs, I broke a $75,000 arbitrary waveform generator. The first day. Almost the first hour. I felt really bad, though he never really commented on it. It cost about $20,000 to fix.
That was the second worst mistake I ever made. I am not going to post the worst mistake I ever made on the web. Ever.
And then there were the hiring decisions made over the years, and not just at my health care jobs, either.
Promoting a certain department manager to assistant manager at the grocery store was a big mistake, too. Hoo boy, did it go to her head.
I worked for a payroll processing company and sometimes the computers have a glitch and do not print out the paychecks. Well one time this happened and the other operator and I both agreed to reprint the checks. Well it turns out that we didnt reset the parameters right and it double paid the people.
Now the thing is my boss’s job every morning was to doublecheck the records and check for things like this but this guy failed to do so. When it all came out he went out of his way to dump all the blame on me when down deep it really was an error anyone could have made but it became a problem because my boss didnt do his job and he needed to pass the blame to someone else. Me.
When I threatened to go over his head he backed off.
Remind me not to buy a helicopter
How about (Andersen Consulting strikes again!) a huge system written in one language which, by decree of somebody under the influence of AC) dictated could not use its native I/O (file and terminal), but had to link to a separate module to pass the data?
The languages are NOT compatible (COBOL and PL/I).
When the COBOL program invokes the PL/I, the PL/I completely re-maps the memory and registers, causing the COBOL to crash.
This was a huge project of about 50 people and was to perform all financial activities of a Fortune 500 company.
The only way for those programs to work was to hard-link them.
Which they decided was an acceptable solution.
About 200 COBOL application programs with these PL/I modules hard-linked into them.
I changed jobs - don’t know haw the huge Bechtel CFIS system disaster ended.
Well, there was the hotel front desk agent who managed to get himself arrested - at work - for dealing drugs…
Or the undercover narcotics agent who had a side job with an automobile repo company (not a problem, cleared with his chief,) who started using his county-provided vehicles to “hook up” those repossessed cars (big problem, not okayed by his now-former chief!)
The absolute worst, however, that I know of from a former workplace: the bar bouncer who got into an argument with a customer one Saturday night. The customer’s body was discovered (throat slashed) the next day, behind the building. :eek:
There was that night, as a graveyard shift computer operator (we called it “owl shift”) that I just plain plumb forgot to do the nightly backups. And of course, the next day the system crashed and lost everything and they had to do a full restore. Fortunately, they had another machine with all the same most-important files. So they started a full backup on that machine, and as it completed reel-by-reel, used that to do the restore on the crashed machine.
There was that weekend I upgraded our BSD 4.2 Unix system to BSD 4.3, which entailed an entirely new file system structure. So it was absolutely necessary to start with a full back-up, then a total disk wipe, install new system, and restore files from the back-up. Well, surprise, surprise! It turned out the 4.3 restore program couldn’t properly restore from 4.2 back-ups. Oops. (The 4.3 restore fucked up when restoring symbolic links from the 4.2 system, of which we had many, producing a resulting trashed file system.) We had words with our vendor over that one.
There was the supervisor I worked for, as a Unix sysadmin at a Silicon Valley tech company, who fucked up everything he touched (and lied about or blamed other people a lot). At least two departments in the company had declared either him or our whole sysadmin group to be persona non grata because of him. I was in charge of doing back-ups there, but I had a flunky high-school kid assistant who came in and did that. I finally quit in disgust. Supervisor promptly fired the flunky assistant.
The story got back to me about six months later: They had a full system crash. Assured by supervisor that the back-ups were current, they did a full disk wipe and re-install. Then, it turned out that no back-ups had been done in the six months since I left. Not that it mattered much, really. The whole company was in the process of crashing and burning already anyway.
When I was the payroll programmer for a bank I put everybody on disability pay. Fortunately somebody in the payroll department caught it before the checks were printed.
Then I screwed up the CEOs paycheck and he didn’t pay state taxes for a few months. Although I considered that to be partly his fault for voting himself such a big raise that his salary went outside the tax charts we were using.
I think the worst though, not me this time, was the bank employee who had over $3,000,000 worth of checks shredded. The bank had to pay dozens of employees overtime to pick through the shreds and tape the checks back together.