So I’m on this “urgent rotation.” There are three “urgent teams” so two out of every six weeks, I have alarms sent to my BlackBerry (PDA/phone) regarding any downed servers, networks, or systems in the data center. Mind you, there are no procedures or policies that establish what the “urgent rotation” does. We’re to respond – restart services, reboot servers, troublshoot networking equipment, etc. There are four people on my “urgent team.” Whoever gets to the issue first is the one to handle it (lack of policy and procedure here). Keep in mind, there is no extra compensation for this duty and not everyone on the team has to do it.
I’m a network guy, so if anything happens with the network, regardless of my “urgent rotation” status, I have to fix it. My everyday workload has nothing to do with servers and frankly, I’m a little lost on how these crappy homegrown applications function. My head and hands are full with our complex networking environment. Since I’m the only network guy on my “urgent team,” I by default get any issue that even remotely looks like it could be network related. That’s fine. It annoys me but I don’t complain.
Beginning Friday afternoon I began getting pages every couple of hours. There was a problem with the MONITORING system, which is under the direct charge of another team member, who is also on my “urgent team.” It’s his baby. He set everything up and declared that we should not use anything else. And it was sending about five false alarms every two hours. I sent five e-mails since Friday night asking if something could be done. No response from anyone. The monitoring system was not fixed until this morning.
My wife said, “I can’t take another night of that thing beeping every 2 hours. Either sleep downstairs or turn it off.” Done. It’s off. Naturally, all night last night, there was a system that was really down. It’s another one of our crappy homegrown applications that needs an IISRESET every three or four days. My BlackBerry was in silent mode. The other three people didn’t respond, either. The boss asks if we are receiving alerts. I responded that I was, but I put my BlackBerry in phone-only mode due to the monitoring system issues and subsequent e-mails failling on deaf ears, so I didn’t get audible alerts. His response? “That’s not an option.” Mine to him: “Expecting people to get beeped over 250 times during a single weekend, causing spousal distress and generally household disharmony, when the problem is correctable by another team member is also not an option – especially when e-mails to that team member are ignored. It’s your choice. Either we get genuine alerts or we turn off the beeping.” No response. He’s all pissy.
Fuck him. This “urgent rotation” has been a complete mess and source of confusion since he decided to establish it a couple years ago. Out of the 11 or 12 people who are on the rotation, there are about 6 that actually have their PDAs configured to audibly alert. He ignores suggestions to improve it. I’m done with it. I’m on it until next Monday. Guess what’s going to stay in Phone Only mode?
If it’s an option for you try writing your own “urgent team policy”, and make it realistic. I did that when an Intrusion Presention System I was in charge of kept alerting me because a network device was unavailable. Since the device had been removed by the network team, which is why it was unavailable, there was nothing I could do about it. I wrote detailed response and escalation policy, which in my case would have redirected alerts from my pager to the network team, and if they did not respond to the technology director. It took a while to get it approved, but as far as I know that company is still using it.
Ditto.
Sure I miss driving 35 minutes in a Corolla during a blizzard to the remote site because a server decided to destroy its secondary hard drive at 2:00 AM, who wouldn’t?
Dude. That “urgent team” concept is, how shall I say this…totally BS, which of course, you know. I agree that getting documented processes and policies in place is a must. I wish you luck.
I always enjoy seeing that others have problems at work. It makes my bullshit problems seem less important. Yesterday I watched a documentary on IFC channel about where they make beads in factorys in China for the Mardi Gras. The workers there get fined a days pay if they talk while they work. Kind of puts things in perspective for when I feel like bitching. Not to say getting woke up in the middle of the night is anything to sneeze at. Not in the least. But shit’s tough all the way down the line.
Second that. Write them yourself and push for it. Best advice I ever got was to never go to the boss with a problem, go to him with a solution. Also, find the guy who was supposed to fix the monitoring system and beat him with your pager.
My last job was at a google-type company, working as a contract programmer. It was a start-up though, so essentially anyone who was in a tech-position was potentially going to be on call. I’d also built essentially 40% of the whole system with the other 60% split between four or five programmers.
So we start going around the table during a dev meeting to decide who takes “the phone” and on what rotation and who they call if the system sends a warning mail… And as they come to me, they skip past to the next person without even commenting on my presence! Woot for being a contract employee! I didn’t have to do anything at all. I was ready to jump up and down in joy, but of course that wouldn’t have been cool.
