# Today I royally fucked up at work. Commiserate with me **URL:** \ **Category:** Miscellaneous and Personal Stuff I Must Share\ **Created:** [December 6, 2013, 2:15am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631 "2013-12-06T02:15:50Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Suburban\_Plankton](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/suburban_plankton/32/219_2.png) [@Suburban\_Plankton](https://boards.straightdope.com/u/Suburban_Plankton)\ **Post date:** [December 6, 2013, 6:50am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/21 "2013-12-06T06:50:36Z")
I’m sure you know this already (and if you didn’t before, you certainly do now), there is \*\*\*never \*\*\*a substitute for off-site backups. So far, I’ve been lucky; the worst thing I’ve \*personally \*been responsible for is accidentally pulling the plug on our Exchange server…but there’s always tomorrow… ---
**Author:** ![Nava](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@Nava](https://boards.straightdope.com/u/Nava)\ **Post date:** [December 6, 2013, 6:56am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/22 "2013-12-06T06:56:23Z")
> [@ZipperJJ](#): > > Raise your hand if you’ve ever forgotten the “WHERE” clause in an UPDATE statement in SQL. And then died. No, since I am not dead, nor do I write SQL, but I do have forgotten to either write or **WRITE CLEARLY ENOUGH** the equivalent part in the functional specs. It’s a good thing I’m in a line of work where SOP is to test everything to death, then resurrect it and test some more. One with backups, not mine: Back in the Pleistocene, we had these VAX boxes at work, and this boss who kept changing the settings for the backup tapes. For some reason he’d have a different person run backups every time and give different settings every time. After a few months we glommed onto this and started writing the settings used on the tape labels. At one point Carlos needed to recover some files. He located the appropriate tape, stuck it in, wrote the read command appropriate for the settings listed on the tape. The VAX paid about as much attention as if we’d started singing lullabies. Well, strictly speaking, it gave some sort of message, but it didn’t read the tape. After a few tries, Carlos went to the boss’ office and came back looking like the sad smiley’s pink brother. Bossman had handed that particular tape to the newest person a couple of weeks back “to teach her how to run backups”: not only was the original backup overwritten, but the settings on that tape did not match its label any more. And since Bossman had been overwriting backup tapes for several weeks, who knew what was where any more. Several of us happened to have partial backups, as we didn’t trust bossman’s further than we could throw him. Our coworkers didn’t laugh at our “paranoia” again and bought zipdisks of their own (kept offsite, of course, and their reader in a drawer). ---
**Author:** ![msmith537](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@msmith537](https://boards.straightdope.com/u/msmith537)\ **Post date:** [December 6, 2013, 7:01am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/23 "2013-12-06T07:01:35Z")
My only fuckup is entrusting critical work tasks to idiots who don’t know how to make & restore backups, write SQL, or save their work.😃 ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [December 6, 2013, 8:14am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/24 "2013-12-06T08:14:42Z")
I realise the horse has bolted, so I’m not quite going to recommend shutting the stable door. But, actually, I am. For next time. As part of your apology letter, include a plan on how this will be prevented from happening again in future. It may be as simple as two people reading, repeating and confirming the meaning of the digits before and as the restore process is kicked off. -That’s what I always do when restoring a snapshot - I write the date version we’re looking for in words (“Test environment backup from 4th September, 2013”), then I also write the expected shortened name for the backup record (i.e. something like “Test\_20130904”) - I pass ONE of these descriptions to a colleague and ask him/her to construct the other, then we both check that we agreed on both values, and both talk each other through the restore process, confirming the meaning of each segment of the date identifier before committing. ---
**Author:** ![DMark](https://avatars.discourse-cdn.com/v4/letter/d/e68b1a/32.png) [@DMark](https://boards.straightdope.com/u/DMark)\ **Post date:** [December 6, 2013, 8:21am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/25 "2013-12-06T08:21:58Z")
I created a database for a law firm, from scratch. Two horrible things happened that turned out to be really good things. Early in development (thankfully), I accidentally DELETED the most recent version of the database! Luckily, I backed it up about every three days, so I was only missing two days of data input (by about 30 temps who had been entering the data), and luckily they still all had the documents on their desks. Two days of work shot to hell, but the good news was that this just happened to be very slow days and there wasn’t all that much to re-enter. After that, I started to do manual backups, a minimum of twice a day! The other glitch was when I “fixed” the database and added a new feature, which promptly erased the original entire section of that part of the database. I still don’t know how the hell that happened - but because I did save the database twice daily, we only lost a half day of data entry - and it was a Friday afternoon, so once again - not a huge deal to re-enter. And then the crash…the entire server went down in the law firm - NOT my fault - but that was where we stored the database and where their IT stored everything else. However, because of my paranoia of the previous two disasters, not only did I continue to back up the database twice daily, but every other night I made a back up on my own portable hard drive! The night before the huge crash, I had saved everything to my external hard drive and - voila - my database was the only thing we could work on the following day - unscathed and fully intact! Those first two disasters had scared the bejesus out of me, and I am thankful it did - or the database would have been toast. ---
**Author:** ![Caught\_Work](https://avatars.discourse-cdn.com/v4/letter/c/919ad9/32.png) [@Caught\_Work](https://boards.straightdope.com/u/Caught_Work)\ **Post date:** [December 6, 2013, 9:32am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/26 "2013-12-06T09:32:28Z")
rm -r /etc is not the same as rm -r etc ---
**Author:** ![Senegoid](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senegoid/32/6606_2.png) [@Senegoid](https://boards.straightdope.com/u/Senegoid)\ **Post date:** [December 6, 2013, 11:02am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/27 "2013-12-06T11:02:09Z")
Here’s one I didn’t cause. I was apparently the first to run into it and report it to our software vendor. I was a Berkeley Unix BSD sysadmin for a smallish high-tech firm. (We made laser printers in the days when hardly anybody had ever heard of laser printers.) We obtained Bky 4.x BSD and maintenance (both software and hardware) from a company that was in the business of doing that. One fine day we upgraded from 4.2 BSD to 4.3 BSD (circa 1975 or so). The procedure was basically: [ol] [li] Do full disk-to-tape backup of entire 4.2 BSD system, other than the root stuff.[/li][li] Wipe the disk completely.[/li][li] Install 4.3 BSD system from scratch (from distribution tape).[/li][li] Do full filesystem restore of all the rest.[/li][li] Recommended: Run fsck (file system consistency check; like Microsoft’s Scandisk) to make sure everything got installed/restored properly.[/li][/ol] In that upgrade, they did a major overhaul of the internals of the file system, including a lot of the data structures and layout of the data on the disk. Since the backup and restore programs worked on the low-level data, they had to be largely re-written as well. We got all the way to step 5. Well, surprise, surprise. Running fsck, we found that the restored filesystems were largely trashed garbage. Turned out, the 4.3 restore program couldn’t restore dumps written by the 4.2 backup program. Oops. We managed to patch up the trashy file system and get it running, but it took us many hours, plus the assistance of a more experience outside consultant expert than I, whom we had to hire and bring in just for the purpose. The problem was all with symbolic links, where were totally different in 4.3 – we ended up just deleting them all manually, which took a long time, as there were hundreds (thousands?) of them. Then anybody who wanted those link just had to re-create them as they needed them. ---
**Author:** ![Snowboarder\_Bo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/snowboarder_bo/32/229_2.png) [@Snowboarder\_Bo](https://boards.straightdope.com/u/Snowboarder_Bo)\ **Post date:** [December 6, 2013, 11:12am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/28 "2013-12-06T11:12:03Z")
I can’t believe neither **Marley** nor **Hal** has made an appearance yet; I know they both have excellent stories they could contribute to this thread. ---
**Author:** ![ekedolphin](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ekedolphin/32/10003_2.png) [@ekedolphin](https://boards.straightdope.com/u/ekedolphin)\ **Post date:** [December 6, 2013, 11:16am UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/29 "2013-12-06T11:16:05Z")
The day after 9/11, I was the opening CSR at Blockbuster, and for some reason that I can only ascribe to stress, I angrily blew off a customer when he quite correctly pointed out that we hadn’t turned the lights on yet. I have no such excuse for the next story: I responded to a customer’s request for something (that would have violated company policy) with the _sotto voce_ inquiry at her back, “You want to pay my bills?!” Which turned out to be not nearly as quiet as I’d hoped. Boy, did I catch it from her. Thankfully, a manager made sure I didn’t dig myself any deeper. ---
**Author:** ![black\_rabbit](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@black\_rabbit](https://boards.straightdope.com/u/black_rabbit)\ **Post date:** [December 6, 2013, 12:41pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/30 "2013-12-06T12:41:43Z")
> [@Mangetout](#): > > But, actually, I am. For next time. As part of your apology letter, include a plan on how this will be prevented from happening again in future. Oh for sure. I’m well practiced in how to give a meaningful apology. > [@Senegoid](#): > > Turned out, the 4.3 restore program couldn’t restore dumps written by the 4.2 backup program. Oops. Ugh. Reminds me of the time we upgraded from JFS to JFS2 without considering the implications on random I/O. > [@Fubaya](#): > > Why doesn’t it ignore the snapshot directory and not delete anything from it? I’ve never been in that line of work but it seems like you could never have too many snapshots. That’s an excellent question that I’m sure as hell going to ask both the vendor and the consultants who validated our configuration. At the very least, I want to know why the hell the warning message isn’t more explicit. ---
**Author:** ![Colophon](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Colophon](https://boards.straightdope.com/u/Colophon)\ **Post date:** [December 6, 2013, 1:05pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/31 "2013-12-06T13:05:08Z")
A chap on Twitter recently asked for “work screw-up” tales, and collated the best on a [Storify page](http://storify.com/rhodri/work-screwups). Read 'em and chuckle. ---
**Author:** ![yoyodyne](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@yoyodyne](https://boards.straightdope.com/u/yoyodyne)\ **Post date:** [December 6, 2013, 2:14pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/32 "2013-12-06T14:14:14Z")
> [@Snowboarder\_Bo](#): > > I can’t believe neither **Marley** nor **Hal** has made an appearance yet; I know they both have excellent stories they could contribute to this thread. **Marley** #7 and **Hal** #4A & 4B? Those are great stories. ---
**Author:** ![wolfman](https://avatars.discourse-cdn.com/v4/letter/w/a8b319/32.png) [@wolfman](https://boards.straightdope.com/u/wolfman)\ **Post date:** [December 6, 2013, 2:19pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/33 "2013-12-06T14:19:25Z")
> [@ZipperJJ](#): > > Raise your hand if you’ve ever forgotten the “WHERE” clause in an UPDATE statement in SQL. And then died. I didn’t forget the WHERE. I simply fucked it up.:smack: We had an auto load program that read data from a vendor for one of our state databases. But one time they sent it in as csv rather than pipe delimited. and our program simply ignored the commas and smushed the first four numeric data fields together. So the new bad records were created with record numbers like 1286868685586585865865865861212331. No problem for me though . Simply delete all records with a record # greater than a billion; so easy I didn’t need to test, just throw it in the midnight cron and head off home Friday night. So boys are girls, can anybody tell me what happens when you fat finger \< instead of \>, but don’t notice? That’s right. You get a frantic call from the night support at 12:15 AM wondering why nobody can find Minnesota anymore, except a couple dozen screwed up records. ---
**Author:** ![SiXSwordS](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@SiXSwordS](https://boards.straightdope.com/u/SiXSwordS)\ **Post date:** [December 6, 2013, 2:34pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/34 "2013-12-06T14:34:45Z")
I too developed the habit of writing the WHERE clause before the UPDATE statement. ---
**Author:** ![Doctor\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/doctor_jackson/32/32_2.png) [@Doctor\_Jackson](https://boards.straightdope.com/u/Doctor_Jackson)\ **Post date:** [December 6, 2013, 4:36pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/35 "2013-12-06T16:36:06Z")
Long ago, when the world was still DOS with nary a Window in sight, I was working at a large Georgia bank. I was just getting good enough with DOS commands to be comfortable with my work. I needed to delete a bunch of files from a particular directory on a hard drive. Because I’m brilliant, I had used a naming convention where each file started with the letter “O”. One simple DOS command would get rid of all of them (del c:\drctrynm\O\*.\* - delete every file in this dierectory that starts with thte letter “O”). I typed it and hit \. Only I forgot to type the directory. I entered "del c:\O\*.\* (delete every file starting with “O” from the entire hard drive). On the computer that stored collection attempts for every overdrawn account for the previous 5 years. Guess what letter the overdraft files started with? Fortunately, someone in IT had a one of the first copies of Norton Utilities in the bank. We got back _almost_ all of the files. My boss was livid, but couldn’t do much to me because he had to explain to his boss why he had not kept backups of such critical files. If you had an account closed for overdraft at a Georgia bank in the 80’s and said bank suddenly stopped trying to collect - you’re welcome. ---
**Author:** ![treis](https://avatars.discourse-cdn.com/v4/letter/t/bc79bd/32.png) [@treis](https://boards.straightdope.com/u/treis)\ **Post date:** [December 6, 2013, 4:46pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/36 "2013-12-06T16:46:19Z")
> [@Helena330](#): > > This isn’t a fuck up like yours, but I was plenty embarrassed at the time! I was about 20 and working my way through college waitressing at a pizza joint. I had the banquet room with a birthday party of I’d say about 15 30-something guys. They drank a LOT of beer. Our restaurant had this deal where the birthday person would get a small free pizza. When I brought him his, the guys started saying, “Give him a kiss! Come on!” Completely oblivious to what I was saying, I replied, “No, but I’ll blow him one”. > > I really DID mean I’d blow him a kiss! The room erupted. I was mortified, but held my head up, took all the jokes and got a great tip. 😃 _cough_ ---
**Author:** ![Shoeless](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/shoeless/32/7099_2.png) [@Shoeless](https://boards.straightdope.com/u/Shoeless)\ **Post date:** [December 6, 2013, 6:11pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/37 "2013-12-06T18:11:35Z")
> [@ZipperJJ](#): > > Raise your hand if you’ve ever forgotten the “WHERE” clause in an UPDATE statement in SQL. And then died. Oh yeah. Once. Now whenever I do an update I include a ridiculous number of WHERE conditions so that I’m guaranteed to only update what I want. I even do a select with the same conditions just to double check. I also updated the Production database once when I thought I was in Development. Oops! ---
**Author:** ![beowulff](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/beowulff/32/542_2.png) [@beowulff](https://boards.straightdope.com/u/beowulff)\ **Post date:** [December 6, 2013, 7:05pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/38 "2013-12-06T19:05:43Z")
> [@Caught\_Work](#): > > rm -r /etc is not the same as rm -r etc Isn’t that the quintessential typo? Or, maybe the root typo, from which all others are descended? ---
**Author:** ![AlsoNamedBort](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/alsonamedbort/32/3213_2.png) [@AlsoNamedBort](https://boards.straightdope.com/u/AlsoNamedBort)\ **Post date:** [December 6, 2013, 7:55pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/39 "2013-12-06T19:55:24Z")
I once sold a $600,000 mortgage for $60,000. Thankfully, it only cost us a couple thousand dollars to get fixed, but that was a “so should I pack up my stuff now or at the end of the day?” moment for me. ---
**Author:** ![AnalogSignal](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/analogsignal/32/1085_2.png) [@AnalogSignal](https://boards.straightdope.com/u/AnalogSignal)\ **Post date:** [December 6, 2013, 8:08pm UTC](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631/40 "2013-12-06T20:08:59Z")
> [@Nanoda](#): > > It’s my favourite preventative. OTOH, make sure you don’t get distracted by something else and forget the COMMIT, otherwise you’ll spend the afternoon debugging the Work Order module before figuring out that the WorkOrder table is locked for your update. :smack: Uncommitted transactions don’t block queries on Oracle. They do block queries by default on MS SQL Server but you can fix that by turning on [Read Committed Snapshot Isolation](http://blogs.technet.com/b/sql_server_isv/archive/2010/12/21/using-read-committed-snapshot-isolation-mini-lab.aspx). I like Read Committed Snapshot Isolation because of this and several other issues it fixes. [Previous page](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631.md?page=1) [Next page](https://boards.straightdope.com/t/today-i-royally-fucked-up-at-work-commiserate-with-me/675631.md?page=3)