What were the biggest screw ups you've been aware of at work?

Miscellaneous and Personal Stuff I Must Share
1

Inspired by the recent, what happens when you’re Written Up thread.
https://r.tapatalk.com/shareLink?url=http%3A%2F%2Fboards.straightdope.com%2Fsdmb%2Fshowthread.php%3Ft%3D836942&share_tid=836942&share_fid=28240&share_type=t

Sometimes things don’t go as planned at work. Heck, let’s be honest, sometimes the shit hits the fan and you get splattered. :smiley: Regardless of who is at fault.

Remember to exclude any Company Names, or other identifying information. The people involved shall remain anonymous.
Around 1988, 89 or possibly 90. That’s a wide enough window. :smiley:

I was in my third year supporting Payroll as a Senior Programmer. My supervisor was a Systems Analyst.

He was on call. We had terminals and a modem at home. We could log in and monitor the Batch Jobs as they ran throughout the night. It’s less stressful on us then waiting for the dreaded page.

If the batch job aborted, the person on call fixed the problem, restored the files from the last backup, and had the Operator restart the job from the Console.

The Operator has full responsibility to start the batch jobs, mount tapes, run Saves (to tape), and manage the line printer. He puts the Greenbar reports into the appropriate Dept’s cubbyhole for pickup the next morning. Checks are burst and collated with a machine he used and put in a secure area for the Comptroller to pick up. He signs off on the starting and ending check numbers used in the run. EVERY blank check must be accounted for. He takes any damaged in the line printer, to the incinerator.

A seasoned, veteran Operator works alone and is a blessing for the programmers and analysts on Call. A newbie can screw up a lot of important work. Or pester us to death with pages throughout the night. We’d already worked a 9 hour day and Call is extra hours. We were required to respond to pages within 20 minutes (by phone).

One late night, my Supervisor observed (from home) payroll’s batch job is stuck waiting for a tape mount. 15 mins later, it’s still stuck waiting. So, my Supervisor calls the operations center. No answer. Calls again. No answer.

He gets dressed and drives to work. Rings the bell to be let into Operations. No response. Uses his key to let himself in.

He finds the Operator sitting, drunk as hell, crying. and trying to rewind the Check History Tape (by hand). He had dropped it and it had unwound & spilled out on the floor. He had walked on it.

WTF!

My Supervisor had been trained as an Operator by this guy 15 years earlier. He had to take over as the night shift Operator. Doing everything needed for ALL THE NIGHTS BATCH. It took him an hour just to rewind the tape and get it mounted

Thank goodness I wasn’t on Call that night. I had never trained or worked as an Operator. I didn’t have a key to the Operations Center. I came out of college as a programmer. It’s very, very lucky my Supervisor was on call. He had five years experience as the night shift Operator.

The Operator was an alcoholic and trying to hide it. He got written up. Then within two weeks he got a DUI and arrested coming to work. The Operations Center manager had to make the day shift Operator work a double shift. The drunk employee was fired the next day.

They found bottles stashed throughout the Operations Center. They kept finding his stash for weeks.

It’s a shame. He had over 30 years experience. Started in data processing in the military. That’s why he was our night time Operator and running the most critical batch jobs.

It was very close to being a disaster. We only had one business day to get the Direct Deposit tape to the bank.

If that payroll job hadn’t finished. Me and my Supervisor would have been urgently running it the next morning. I would have hand carried the tape to the bank. Otherwise, people on direct deposit would get their accounts credited a day late. That not good and we made DAMN SURE it never happened during our careers.

It got close. A couple times, stuff aborted, and it was after lunch before I could take the Direct Deposit tape to the bank.

The shit did splatter. My Supervisor got an oral warning for having a key to the Operations Center. It should have been turned in eight years earlier after he was promoted to Programmer.

2

A patient got 2 1/2 times the morphine they should have, and Narcan was successfully employed.

It turned out that the IV bag wasn’t even the right size - and this wasn’t caught by our best technician (who had made the IV), our best pharmacist (who checked it) and also by the ICU nurse who hung it.

I could tell when that pharmacist found out about it, because that day, he looked like he had a big black cloud over his head. He never said anything to me, even though he would have known that I had documented it, because this kind of thing can happen to anybody.

:frowning:

3

You win the thread. For now.

Glad the medication error got remedied in time.

A few hundred employees that possibly could have been paid a day late, seems unimportant compared to life & death.

4

It seems we had this thread a few months ago, but I can’t find it. Here are a couple somewhat further back in time:

http://boards.straightdope.com/sdmb/showthread.php?t=837125

5

Heh. All my examples are big company damaging. Please excuse the lack of details.

In one place we were discouraged from writing papers or going to conferences. Too expensive, takes time from work. So people never read papers either. One group walked right into a big problem which had been amply documented in papers, and which the “experts” in the group had never heard of. Delayed the project a year and cost the top guy his job.

Then I worked on a chip that had a new instruction set but which had to run legacy code. To save area, which was at a premium, they slowed down the running of this code so that it was slower than chips costing a tenth as much. Important when there was almost no software with the new code out there. So, not a big rush to adoption. There were plenty of other problems, but this cost easily a billion bucks.

Then when AT&T was trivesting, in order to maximize the value of the IPO for the hardware part they wanted to get people off the payroll. The offer was so sweet that 1/3 of my center left at once, including me. They caught the rising edge of the bubble, so it looked great for a while, but it soon crashed.

I’m not 100% sure about this, but I’ve heard that the Sony Hollywood people starved the IT department, not being as sexy as making movies and TV. Was great, until the Koreans stole and published their emails.

Screwups that worker bees make are trivial compared to these.

6

Very early in my career, a department had accidentally erased a very large amount of business-critical data - years’ worth of invoices, receipts, etc. So we were called in to restore it from the tape backup. They had been using only one tape for backup, and the deletion had occurred before the backup run, so the data was gone.

The other one was an accident. A file server was processing flight trials data in a shielded room. The room was shielded both for security reasons and because we were just off the end of a runway and the room was used to test radar and the like. It had a set of doors in an airlock arrangement and if both doors were open at the same time the power in the room would get cut to prevent an airplane coming down on us and prevent spooks from getting any info. This duly happened and they then realised that no one had put a UPS on the server. The server had, of course, crashed. All their data was gone. Calling out a data recovery specialist would have cost a LOT. So the whole department had to work over the weekend to get back on schedule. Fortunately, they had the original data, but it was still a lot of work.

7

Someone failed to note that a large project required Illinois State Prevailing Wage which, in our field, almost triples the labor costs. When the state asked for the certified payroll, the results bankrupted the 50ish year old, 100+ employee company.

8

Here’s another (4-pager) from 2015: What is the single largest screw up you've ever witnessed at your job? - In My Humble Opinion - Straight Dope Message Board

9

Late edit:

Hey PastTense: Your second link is to this same thread we’re posting in. :slight_smile:

10

I saw an airplane land wrong and tear the fuck out the landing gear DOORS (not the landing gear themselves). Apparently even land gear bay doors are expensive (about half a million dollars).

I also learned that being at the other end of a very short runway when a big assed plane lands and shit happens is probably a bad idea.

11

I worked nights in a residential care home and the nurse in charge was in a habit of writing up all his notes and doing the paperwork stuff just after our checks at midnight so he could go and have a few hours sleep in one of the empty rooms. Remember saying to him it wasn’t wise and he’d end up a cropper one day but he fobbed me off and said I was too by the book and didn’t know anything and so on.

One morning I found a lady dead in her room at 6am and it wasn’t expected at all which meant the police had to be called out and do all the stuff that happens after a sudden and unexpected death.

Her notes said *“Had medication, breakfast and two cups of tea. In a lovely bright mood this morning. No problems or areas to note” *and were signed off at 7.30am

12
  1. Somebody came back from a month’s vacation and, instead of pushing the latest test build of Windows 7 to the ten pilot desktops, he set something wrong in the push software and sent the auto-installing package out to every machine in the enterprise EXCEPT those ten.

Turns out Windows 2008 servers don’t server so well when they’re running Window 7.

  1. Somebody else (different company) was in the server build room, where they have a nearly-exact copy of the hardware setup of all the branch offices, so they can build whatever kind of server they need from a CD and then test the communications & such between the different machines. These machines were built using the same type of IP address patterns that were used out in the field, and the room had a disconnectable connection to the network so they could download patches and such. The guy noticed a network cable that wasn’t plugged in to the obvious port it should go into and, being a good Samaritan but a lousy engineer, plugged it in. Turns out the servers in that room had not the same TYPE of IP addresses, but the ACTUAL IP addresses of, say, primary domain controllers out in the enterprise. Ten seconds later, everybody’s pager goes off as the entire network takes a nap.

  2. A very large hospital chain had a huge data storage monstrosity which held all the work and personal files for about 8,000 employees. They were only supposed to use it for work files, but people found that if they uploaded their playlists to their personal folders, they just had to carry a set of earbuds around and listen to music at whatever station they happened to log into. A four-day weekend ensued, during which about 700 people downloaded a ton of new songs into their iPods. Tuesday morning, at 7:55 am, 700 people logged into their workstations, dropped their iPods into their charging station/data transfer thingy, and started to synch… As it turns out, the system was set to reindex whenever somebody added a file that way. For each file. This ended up eating up all the storage space in the root directory in no time flat and took the data storage down for the better part of a day. All those x-rays…

13

I worked for awhile as a part-time HTML jockey at a small IT services company. This was in the very early days of web hosting and their primary product was simply “We’ll buy & maintain a server for you in our office and help you set up a hokey website on it”. For various small local Mom & Pop retail businesses. Total amateur hour.

One fine day about a year into it a server crashed unexpectedly with a full hard drive. Which should never have happened based on the expected server use. IT discovered that A) somehow FTP had been left enabled on that box; B) with no password; and C) some unknown somebody somewhere on Earth had been using our box and bandwidth to store (or distribute) his porn. Many, many, many GB back when 100GB was a lot of storage and today’s unlimited free internet porn was still years in the future.

We were afraid to look at more than a couple of the images just in case we found something we couldn’t stand to see and also couldn’t stand to not report. We quickly physically destroyed the disk drive, bought a new one & installed the OS, then restored their website from backup. Then physically destroyed all the backups and started fresh. Last step was to make damn sure FTP was really disabled on every machine, not just most of them.

Oops. :eek:

As Sgt Shultz used to say: “I zee NuhTHIIIING!!!”

14

heh

15

my boss who was in a wheelchair and had rheumatoid arthritis years before all the wonder drugs came out for it had a used video game stand which ive written about before

Well there was 2 rare as hell nes games that people asked for and we had one… the price we were selling it for was 49.50 but boss who tagged it while she had a flare up at home had it for 4.95 …kid bought it for his dad who was a good customer himself

dad brought it back saying somethings wrong and offered to pay the rest she said no they argued and he gave her 20 for it anyways

16

Back in the days of big mainframes, we had an underground computer room about 1 block square, supporting hundreds of banks over a multi-state area, all their ATMs, running all kinds of financial processes (we did payroll for several hundred small-to-medium businesses, etc. Just alongside the secured airlock entrance to the computer floor was a big red push switch labeled “Emergency Power Shut Off”. Designed to be used by the computer operators as they ran from the room just before the fire suppression system replaced the oxygen in the room.

One day, an insurance company safety auditor was coming out of the room after an inspection,saw that , and wondered “does this actually work?” – so he pushed it!

Alarms went off all over the building, and everyone was evacuated. (Into a Minnesota winter.) Fire engines started arriving. Since the master vault was also located in a sub-basement of that building, the local police, sheriff, & FBI were alerted and started arriving. The regional Federal Reserve bank, 1 block down the street went into lock-down mode, just to be safe. Cutting off access from all other banks in the region to the Federal Reserve.

1980’s mainframes didn’t like such an abrupt cutoff of electricity. IBM had to fly in parts from half a continent away. Took over a day before systems came back up, and we could start the lengthy task of recovering from this.

When I was i the computer room a few days later, that switch was now behind a glass door, with a hammer chained next to it.

But our emergency plan did pass inspection!

17

Looking over all these threads I wonder how it is that almost all of the screw-ups are in IT? Don’t plumbers or electricians or … make mistakes? Or are they just not as colorful?

18

“To err is human; to really foul things up you need a computer.”

19

I don’t know what this guy was thinking.

Back in the late 90s, DRAM chips were in a worldwide shortage. A relatively new hire to our company decided to start stealing reels of them and selling them on the grey market.

I worked closely with material planners at the time and they knew something was going on after a couple of reels just vanished.

So, of course, security (whose job was basically to check your ID on the way into the building) got involved.

They secretly monitored the storage room and caught him red-handed walking out of the building with a reel.

Rumour had it that he was able to pay cash for a brand new Passat when he was finally caught.

Needless to say, his high-tech career came to a grinding halt at that point.

20

Lots of folks seem to think that because a business owns many of something, they can’t possibly actually keep track of that many. These are the same people who count like 1, 2, a few, lots.

The idea that a business would know exactly how many come in, how many are on the shelf/in the factory, and how many go out is just a total mystery to these people. Ditto that a business would be motivated to track items that are A) valuable and B) readily steal-able.

Idjits.