The process was found to be incorrect, so it was taken out the back and shot.
On a much smaller scale, an AT&T installer and I took a 5ESS switch off the air during a hardware update once. We were sitting next to each other at the Master Control Display, following the procedure step-by-step, doing exactly what we had done the night before, but somehow we managed to turn two pages in the manual, skipping several steps. Upon hitting the execute button, system indicators went red all over the display and major alarms started going off. We looked at each other and immediately realized what had happened.
It took about 15 minutes for the switch to fully recover. Luckily it was in the middle of the night and the traffic was extremely low so the customer impact was minimal. We got a stern talking-to by management the next day but there was no disciplinary action taken. Back in those days there was a saying that you were never fully qualified as a switch technician until you had taken an office off the air, so in a way I kind of joined the club.
I doubt the personnel involved in yesterday’s “incorrect process” will get off as lightly as I did.
Coincidentally, I was just listening to a podcast about mistakes (I think this one at Hidden Brain). Their finding/hypothesis was that severely punishing mistakes in corporate and medical settings leads to more mistakes.
When there are strong punishments people are afraid to report mistakes, so they get hidden and covered up. If people feel safe reporting mistakes, then they can be learned from and changes made to lower the risk of it happening again.
But, as can be guessed from thinking about this for a bit, an environment where it is safe to report mistakes may initially look worse, because there are many more recorded mistakes, than when people are discouraged from reporting.
Years ago I worked at a major private contract research institution. One of my colleagues was fond of saying, “What mistakes did you make today? If you didn’t make any mistakes, you didn’t learn anything.”
Woke up this morning to an email from AT&T promising to make this right, that they apologize and they are going to offer me an account credit.
I guess $5 is better than a stick in the eye, but it’s not even enough to go get myself a Whataburger for my troubles.
Hmm, my message from AT&T was, in its entirety:
It’s AT&T. We apologize for Thursday’s outage, which may have impacted you. As a valued customer, your connection matters and we are committed to doing better.
Makes me jealous of you!
Heh, I got the same confidence inspiring text.
How much do you pay per month for the service?
Divide that by 30*24=720 and that’s what you pay per hour of availability. Multiply that by however many hours they were out for you.
Good bet a $5 rebate means you made a profit on the deal.