What were the biggest screw ups you've been aware of at work?

Miscellaneous and Personal Stuff I Must Share
61

At one job, we used a very high-end network interface card (NIC, the thing you plug the network cable into on the computer) in a new product. Everything worked fine in testing, and then we started selling the product to customers. Pretty much as soon as they were hooked up to real Internet traffic, customers started to complain that a small percentage of the traffic would be corrupted and discarded. It turns out that the NIC had this little issue where it would crap all over a message if it contained at particular 8-byte pattern in the wrong place. We never generated that pattern in testing, but the Internet managed to pretty quickly. Bastards originally tried to argue with us that the laws of probability said that the issue could only happen once ever hundred years or so. We had to inform them that in the real world, it happened every 7 hours or so (as it turns out, people don’t send literally random data over the Internet. Who knew?)

The producer of the NIC wound up having to recall everything that they’d shipped and make a tweak to a design of the chip to fix it. That would have been … expensive.

62

I have a few submarine sea stories as well.

There was one that involved another submarine (not mine) in my squadron that was also stationed in Groton, Connecticut. The submarine in question was heading out for a six-month deployment, which involved proceeding from the submarine base down the Thames River and out to sea.

There is a railroad bridge crossing the Thames River that was supposed to raise up when submarines passed under it. However, at some point the raising mechanism failed and it was several months before they got it fixed. In the meantime, it was determined that submarines could just make it under the railroad bridge with about 2-3 feet of clearance above the sail (formerly referred to as the “conning tower”) so long as all masts and antennas were lowered.

Note that surfaced submarines usually have several masts and antennas raised, for various purposes including communications, ventilation, etc. They also usually have one or both periscopes raised, because they supplement the lookouts up on the bridge at the top of the sail.

A few minutes before passing under the railroad bridge, the Officer of the Deck (OOD) gave the order to lower all masts and antennas. The OOD (as well as the CO and lookout) all looked behind to ensure that all masts and antennas were retracting. They then turned their attention back forward to the upcoming railroad bridge.

What nobody noticed was that one of the periscopes had not fullly retracted. (This was because the operator down in the control room had not fully rotated the periscope lowering actuator. As a result, the fairing that housed the scope had retracted, while the actual periscope had not.)

Nobody noticed that anything was amiss until about 15 seconds before going under the railroad bridge. That was the point that the lookout noticed the periscope sticking up. There was a frantic order given to “LOWER NO. 2 SCOPE!!” but it was too late. :eek: They went under the bridge, and crushed a multi-million dollar periscope, essentially destroying it. :smack:

The sub subsequently made it into Long Island Sound and promptly turned right back around to head back to port to have the periscope replaced. The sub left for deployment again the next day without incident.

Rumor had it that many wives, girlfriends, and significant others were quite surprised when sailors who were supposed to be away for six months unexpectedly returned home after the aborted initial departure. Some of these reunions were happy ones. Others reportedly were not…which led to break-ups and divorces. :frowning:

63

My parents company delivered phone books. To verify the delivery, we had a 32-seat call center.

Wrote out a proposal one weekend saying it would be worthwhile to explore revenue-generating opportunities with the call center. My parents said “No, we deliver phone books and that’s what we’re good at.”

Oh well!

64

My town has an Infamous Truck Eating Bridge (yes, that’s what it’s been dubbed) and you guessed it - every few weeks, a truck gets stuck underneath it.

:smack:

The local newspaper prints calendars every year with pictures of the crumpled trucks, as a fundraiser for a literacy program they sponsor.

65

I worked for a bit as a security guard at various places.

One contract was with what I will call an engineering research centre. We were hired basically to go do a daily final check round just after everyone was supposed to have left, just to make sure no-one had left the wrong stuff on or passed out in the bathrooms or anything, occasionally sit in when someone was pulling an all nighter, and make sure the local kids didn’t cause a nuisance in the car park.

It was designed as a pretty secure facility; almost every door but the bathroom required a passcard to get in and out, as there was some seriously expensive gear and secret research projects from some major companies in there. From the main hall, there were only two exits that could be opened without a passcard; the fire exit, which set off all kinds of alarms if opened, and the delivery doors that you could drive a truck though, which could only be operated from inside.

Got there one night to find the place deserted, and the huge doors wide open. Cue phone call to the site manager. Turns out some outside contractors had been there that day to install something, no-one had issued them a pass (don’t want the plumber wandering round, right?) and all the staff had just gone home and left the contractors to finish up. Only door they could open to get out was the delivery doors which couldn’t be closed from the outside. Then they’d just got into their van and drove off without telling anyone, leaving a door a truck could fit though wiiide open into the main workshop filled with something in the order of £50 million worth of equipment, plus secret research projects.

There weren’t even cameras on that section, which is why we’d been hired in the first place. Oh, and a public footpath went right past, so it’s not even as though it was some totally isolated spot.

So far as I know, they got away with it and no-one came in or nicked anything, but dammit, that was dumb.

66

Well, there was that time a guy sent a live M-60 round ping-twanging through the armory containing around 100 people.

67

When I started airlining mobile phones were large, expensive, and rare. Substantially nobody had them.

We’d leave for 2, 4, or 6 days and maybe call home once from a hotel, maybe not. The culture at the time was “out of town is out of mind.” A call to/from the hotel was usually the sign of a family emergency.

It often happened that the trip was rescheduled on the fly so you might return anywhere from 3 days early to 3 hours early to 3 days late.

A standard saying in the industry was “spend a dime; save a marriage. *Always *call from the airport if coming home early.” :smiley:

I always did. I don’t know if it paid off or not, but I never walked in on an adverse surprise.

68

One of the agents for which I worked had a kid. His wife was also his accountant (and not particularly good at it).

That particular agency always played it too closely with the subcontractors’ payments, but that month no payments took place; the internals’ salaries were more than a week late. When the two internals in our team mentioned it they were surprised that the rest of us (all subcontractors) got so angry; even more when we provided them with the exact references of the Labor Code that were being broken and contact information for several lawyers who would be happy to assist them in taking the agent to have his hide ripped off at Labor Court. We explained that in our case getting paid late(ish) was part of the cost of doing business with assholes; for us, claims on late payments weren’t worth it below a certain amount and a certain time. But for them, not being paid on time was instantaneous grounds for being able to sue the other party and to leave the company without so much as a fuckyaboss.

One of the internals was our team leader; she forwarded the highlighted legal documents to The Asshole. He paid everybody within 24 hours and hired an actual accounting firm.

69

My dad is a retired firefighter, and worked 24 hours on, 48 off. Many, many times, he had co-workers who went home sick, and if he thought he was sick when he left work, he felt REEEEAALLLY bad when he caught his wife in bed with his best friend, brother, neighbor’s husband, etc. (I never heard of any wives being caught with other women, but I’m sure it’s happened), passed out drunk with the kids demolishing the house, snorting cocaine off the coffee table, etc. :eek:

70

That reminds me of one I saw when I was working at a paper mill. Tractor trailer rigs would get backed up to the warehouse and dropped there to be loaded while the cab and driver go on to other tasks. Procedure is that a stand is placed at the front of the trailer once the tractor pulls out to prevent this. Well, someone didn’t put the stand there and that happened once a couple of rolls of paper were loaded. Fortunately the clamp truck had backed out before the trailer overbalanced.

71

I’ve stopped counting the number of times I’ve seen ground power units pulled away from an aircraft without being disconnected from said aircraft. Usually results in both a sheetmetal/composite repair and an electrical repair to the aircraft, and often a replacement of the GPU’s power cable.

About 9 years ago, I was working four 10hr shifts a week, with every Friday off; I’d work the full day shift, then work with the night shift for the remaining 2hrs. This allowed me to consistently fly with the Air Force Reserve, and allowed my employer to know well in advance when I would be doing my Reserve duty.
We had been working hard all week, working off a bunch of last-minute discrepancies, to get a customer’s A109 MkII finished and out the door. Late Thursday afternoon we discovered yet another problem while doing a landing gear ops check; when I left, the night shift was preparing to tear into one of the main landing gear for troubleshooting. When I returned to work on the following Monday, the sheetmetal guys had a section of the tailboom’s skin removed. I asked around, and found out that on Friday, my lead mechanic had lowered the aircraft off the jacks, but had forgotten to ensure sufficient clearance from all the ground support equipment they had been using. A corner of the ground hydraulic power cart had torn a gash in the skin of the tailboom as the aircraft was lowered. My employer had to eat the cost of the repair, which consumed the margin of the entire job… and then some.

A coworker once got out of sequence in a maintenance check of the engine fire extinguisher system of an S-76, and blew both engine fire bottles simultaneously in the hangar.
CF3Br stinks, and engine fire bottles are loud. Especially when you’re working 2ft from the bottles, inside that very aircraft.

A (different) coworker didn’t read all the notes, cautions, and warnings in a maintenance procedure, and broke the cage of one of the internal roller bearings that support the main gearbox output shaft of an EC135. The customer got a new main gearbox on my employer’s dime, and the coworker was “encouraged” to retire.

The C-17 that landed at a small GA airport, rather than MacDill AFB, a few years ago? That aircrew (and aircraft) was from the same Wing I was in. Damn lucky that Barney is such a capable aircraft, even when not configured for a short-field landing.
Trivia note: Current SecDef James Mattis was a passenger on the aircraft during this incident.
***** All photo links are examples, and are neither the customer nor the specific airframe involved. *****

72

Is it this one? https://www.youtube.com/watch?v=USu8vT_tfdw

73

In my previous life as a utility power engineer.

We were attempting to tie in a new transmission line to a substation. It was a double duty facility in that it operated as a transmission sub and also as a distribution sub.

We switched in a new transmission line and then disconnected it from its other source. However, we didn’t prep the relaying controls at the new source substation and it tripped out taking our substation and a few medium sized towns south of Kansas City with it. Yep, I turned off a few cities.

Another time a green engineer was working with me and we were isolating circuit breakers at a substation and then trip testing them. To do this you basically close some switches and temporarily switch two circuits onto one (1) circuit breaker. Then you can open and close the (2) isolated breaker to your hearts content.

After testing breaker 2 we reversed our switching order and put two circuits onto that breaker 2 and isolated breaker 1. Green engineer had been watching us test breaker 2 but then didn’t much pay attention to the follow up, assumed it was still isolated and casually reaches over and turns the manual switch. Yep he turned off a few cities. I got to own that one since he was my intern.

74

My first paying job out of college was working as a COBOL programmer on an IBM 360/40, punch card, DOS environment. It was pretty cool until the day we needed to run a very specific job.

Turns out my predecessor had lied on her application. She didn’t know COBOL. Her husband had taken one semester of COBOL. She looked great at work when she’d get a job assignment. She’d run through the requirements, flowchart it, etc., then take it home and her husband would write the code. She’d bring it back, punch it in and run it. If it didn’t work, lather, rinse, repeat. She finally got found out and let go, hence my arrival.

The job we had to run was her last job. It had never been run on live data. She had simply gotten it to compile with no errors, then loaded it into the disk library. Nobody caught this. When we ran it, it destroyed stuff right, left and sideways. It took us two weeks to recover the data and I wound up throwing her code away and rewriting the application from scratch.

75

No, it’s not.

50 miles away, there’s a college bar which has a double-decker bus, and THAT rammed into a bridge by a driver who I’m guessing was as impaired as the passengers. :smack: :eek: :mad: Had anyone been in the upstairs portion, they would have been killed instantly because it sheared off the top, and fortunately nobody was driving right behind them.

76

Um… How 'bout the mainframe operator who, while trying to impress the young and pretty new Duty Officer (Air Force computer facility, and she was the new baby lieutenant in charge of the computer facility for that shift), demonstrated how to use the Halon system abort button. Without looking at the wall (because he was that elite). And the thoughtless engineers had put the “manual discharge” button right next to the “abort” button. And they were identical buttons. Well labeled, but that doesn’t help if you’re not looking.

The young lieutenant was only slightly injured (bruises) when she was knocked down flat from the Halon discharge from the nozzle directly above her. At least she had someone good to blame (the soon-to-be-former machine room operator) for how every mainframe in her charge suffered an emergency power off crash.

77

In my previous career as a drilling engineer, I noticed that we had been intentionally drilling wells under the natural pressure of the reservoirs, which is normal in the western US. They are able to do this because the reservoirs are such low permeability that you have to hit a fracture (or create one) to get them to flow. The reason that this came to my notice was that we were planning a new development in a deeper horizon that drilled through 6 current reservoirs and we were planning on using the same casing program for the new development. Basically, the plan was to shore up the weak rock in the first 2,000 feet and then drill 7,000 more feet through the reservoirs before landing in the new horizon and setting another string of casing.

The question that came to my mind was that we had been intentionally hitting fractures in these other zones what was the odds of hitting one unintentionally. So I went to our geologist and asked the question and after they thought about it decided the answer was 3% per horizon and nothing I needed to worry about. So I went to my boss and told him this was a bad idea, we were going to drill 100 of these wells and we only had an 83% chance of making it through successfully. He told me I was worrying about nothing so I went to his boss and explained that over the course of the field we would absolutely hit a fracture at least once and more than likely multiple times and once we did we would probably have an underground blowout and be on the national news for killing a bunch of people. He called several meetings and I was told to quit worrying. I drilled the first two wells with the original casing plan with no problems and then left the company.

Apparently, on well 4 they hit a fracture that flowed and while no one was killed it cost them almost $30 million to get it back to where they could abandon the well. I was told that they hit another fracture cluster shortly afterward and the field was abandoned due to it being too expensive to drill so not only did their lack of understanding of probability cost them 30 mil but in the end, they lost the entire reservoir since it couldn’t be drilled. Of course, they went bankrupt a couple of years later too.

78

I worked in a large, British* bank, and TPTB decided to start promoting mortgages for purchases in Europe. We informed the branches overseas to prepare for more business, got the information packs together, advertised in the press, on TV, in the cinema, on the radio, and in branches, for customers to ring a number to have the packs sent out. The number was an answerphone, where the customers would leave their address details.

What we didn’t do was arrange for anyone to check the answerphone. The advertising campaign was not a huge success. Several months passed before we remembered someone in our office should be checking the bloody answerphone, at which point it was full. For how long people had been unable to leave their address on a full tape was unclear, but we duly sent out hundreds of requested packs.

*It isn’t British anymore

79

This reminds me of all the organizations that couldn’t get volunteers, in large part because the phone number was not answered, and the call not returned if anyone did leave a message, or e-mails not replied to either. :rolleyes: :confused:

80

The other morning I was walking in the living room pecking away on the smart phone when I trod upon the Hound’s tail with my bare feet. He immediately yelped and I discovered a tail being rapidly withdrawn from the space between your foot and your toes feels incredibly weird. Needless to say, he milked it for all it was worth for the next ten minutes, hurt looks, whining, and sucking up apologies like a sponge.