Verizon Data Services is headed into a ditch at 140mph
Verizon has recently (last December) been required (by the FCC) to break into lots of little subsidiaries. The result is that now it takes 3-6 hours, sometimes 3-6 days to troubleshoot & resolve problems that I used to be able to fix myself in under 30-90 minutes. I used to have remote access to the public data switch, I could make administrative changes in settings, correct faulty mapping issues, send out test patterns to localize faults all from my workstation.
Now we have become strictly a call center. We no longer proactively monitor the T1s & T3s for alarms. All monitoring is done up in Boston. If the guy in Boston falls asleep, we miss the alarm & nobody knows the customer is down until the customer himself notices, by which time it’s too late- the customer is out of business for at least a day and our metrics are shot to hell.
If the alarm center does catch the alarm, they open a trouble ticket & drop it into our ticket pool. As long as nobody in my group is asleep, we eventually (maybe an hour later) notice that a new ticket has been opened & then we start acting on it.
If we observe that the trouble appears to be in the central switching office, we get on the phone & wait in queue for about 30-40 minutes for the central office dispatch group to take our trouble report. If it’s an unmanned CO, we have to call a special group of “on call” dispatch techs who may or may not return our page (before, I could hop into a company car & zoom over to the CO in question).
Once the central office determines that the trouble is not in the CO, they call the facilities group & alert them about some faulty repeater or something out in the field somewhere. Several hours later, the facilities group determines that a cable technician has to go out to replace a defective repeater in a manhole.
When the cable tech gets out there, he determines that the facilities are on fiber (not copper) and calls the dispatch center to tell them that a fiber tech (with a fiber truck) is needed. Or maybe the repeater is up on a pole and we need to come back with a cherry picker.
Meanwhile, back in my office… I am sitting in the same building as the main gateway gigaswitch for the Patomac region. If this switch goes down, fast packet data in three states & DC comes to a screeching halt. But because I am no longer part of one big company, we have to refer any troubles up to the switch maintenance group in Boston.
Last month, one of the power supplies went out in this switch. There is a backup supply, but never the less we are on full red alert because of the gateway will fail if the 2nd power supply gives up. I’m sitting about 100 feet away from it. I have spare power supplies for it. I can still get into the central office with my key card. But because I’m not part of the switch maintenance group (in Boston!) I can’t do anything about it.
We had to open a trouble ticket with the Boston group, they in turn referred the trouble to a local switch maintenance tech in the CO, who did not know how to replace the power supply in the switch & ended up calling me on my direct line for me to walk him through it.
I could have had the damn thing fixed on less than 15 minutes, but took well over 24 hours with our wonderful new sectionalized subsidiary system. Thanks a lot, FCC.
The worst part of all this is that I used to be able to call directly to whichever group I needed- if I knew the facilities were fiber, I’d call a fiber tech. Now I have to wait for hours, maybe days to find out through the chain of groups that seperates us, that the wrong tech was called out. Then we’re back to square one.
We had to get all new phones (we’re not part of the core company so we had to give back all their hardware). The new phones go dead periodically, usually rigth when I’ve picked up a customer who has been in queue for 30 minutes & I’ve just given him my name. An hour later (after we’ve gotten the telephone hardware group to reset the ACD breakers) the guy calls back & wants to speak to my supervisor because he thinks I hung up on him.
Meanwhile, as I wait on hold for the dispatch center to pick me up, there are 8 or 10 new calls in my queue. I have never had so many pissed-off customers yelling at me…
Would we have a unfair advantage over other Data Service companies (UUnet, PSINet, etc.) by being able to quicky address major network facility outages without having to go through all of these damned hoops & subsidiary hand-offs?
Apparently the FCC thinks so. Just wait until one of their T3s goes down & see how fast I move. You may have to get out your calendar to be able to clock me accurately.
