[QUOTE=smiling bandit]
I suspect we are talking about two completely different systems (cough DSL cough). I’m talking about being able to check the network card and connections directly. No doubt somebody was messing with it improperly, but that happens with several thousand tech support guys. Once we lost that, the whole thing turned into a huge slog as no one could get the real information they needed. More to the point, cutting access totally is really bad “solution” because it makes the situation worse.
[/QUOTE]
Oh, I see, it’s a different system than what I thought. In this case there is an element management system (EMS) that talks directly to the cards, and a network management system that commands the EMS. There are a couple of different problems in play.
In some cases the EMS is poorly designed and becomes crashy when too many people are logged into it.
The more pervasive case is when you get heroes who go in and start manually manipulating cards, ports, and connections in the EMS. The problem is, the NMS is the system of record, and if it doesn’t have an accurate representation of what’s been done in the EMS, then Bad Things Start Happening. NMS talks to Every System, as in billing, physical plant, inventory, tech support. When it gets bound up, it affects the entire world. As in, hundreds or thousands of customers getting randomly disconnected by routine orders to connect, disconnect, or move services, service appointments start getting missed, people get overbilled or underbilled, connection commitment dates start getting missed. Then it takes a Very Large Crisis Team to sort out the problem caused by this one hero who thought he was hot stuff trying a secret ninja move to sort out one customer.
The overarching problem here is that DSL, at least in its early implementations, was way too complex. The original assumption was “We’ll just hire smart people and make sure they only do the right things”. But the support burden grew too fast, and the complexity grew too fast. It wasn’t possible to add more smart people that quickly, and the technology changed so fast that it wasn’t possible to keep everyone as smart as they needed to be. Things have simplified somewhat, but the supporting systems still show all the fingerprints of that initial overengineering.
Not that I know anything about your actual situation, mind you, I’m just taking a random guess based on your description.