San Francisco's computer access nightmare

[XT]

Interesting…that was my thought exactly when I first read the story:

In posts to my blog, I postulated about what might have occurred. Based on the small amount of public information, I guessed that the situation revolved around the network itself, not the data or the servers. A quote from a city official that Cisco was getting involved seemed to back that up, so I assumed that Childs must have locked down the routers and switches that form the FiberWAN network, and nobody but Childs knew the logins. If this were true, then regaining control over those network components would cause some service disruption, but would hardly constitute the “millions of dollars in damages” that city representatives feared, according to news reports.

It’s really not all that difficult to break into a CISCO box if you have physical access. The procedure is pretty straight forward and doesn’t take all that long to accomplish. Sure, there would be a disruption, but if they need their network to be up 24/7/365 then they probably need to make sure more than one person has the freaking password (I know it’s an apples to oranges comparison but sheesh!).

ETA: This part is interesting.

“Because the system was so complex (and also because he didn’t involve any of the other network engineers in his unit), Terry was the only person who fully understood the FiberWAN configuration. Therefore, to prevent inadvertent disruption of this admittedly critical network, he locked everyone else out. I know most of the networking equipment … does use centralized AAA, but I get the impression he may have configured the FiberWAN equipment for local authentication only.”

Again though it would be easy to simply break into the box directly and circumvent the local authentication…hell, once you use the password work around procedure you could blow all that crap away if you wanted to.

-XT

[Spiny_Norman]

@xtisme

Again though it would be easy to simply break into the box directly and circumvent the local authentication…hell, once you use the password work around procedure you could blow all that crap away if you wanted to.

-XT

Well, there were rumours that he’d not saved the running configurations to flash.

I do not know of a Cisco password recovery procedure that does not involve a reboot, so you’d have complete access - to a blank router. Not a good thing for a production network. Incidentally, IOS 12.3 sports something very much like this as a feature (“no service password recovery”). You can still break into the router with the usual <CTRL>-<Break> during boot - but it will erase the startup configuration before the console gives you access.

If no one has backups - or if the backups are of unknown quality, which is almost as bad - I can see why the mayor had a talk with the guy. Breaking in may or may not work, but the risk of failure is pretty darn high, and the consequences could be pretty bad.

I expect the post of CIO for San Francisco to become vacant Real Soon.

[XT]

Yeah, just saw that:

The inner sanctum
Later in the e-mail, my source offered some insight into what may be at the core of the issue: Childs was so paranoid about the security of the network that he even refused to write router and switch configs to flash, which would mean that if the device was powered off, all configurations would be lost.

“At one point he was concerned about the security of the FiberWAN routers in remote offices, so he had them set up without saving the config to flash. “If they go down, I’ll get alerted, and connect up to them and reload the config.” Great, except we have power outages all the time in this city, some of those devices aren’t on UPSs, and what happens if you’re on vacation? And what about the 15 to 60 minutes it might take you to connect up and reload? He eventually conceded and (ahem) decided that disabling password recovery was sufficient security.”

If Childs did this with some or all of the switches and routers comprising the FiberWAN network, then password recovery without significant network disruption becomes a bigger problem. Without first-hand knowledge of the state of those routers and switches, there’s no good way to know, unfortunately.

If the details given to me in this e-mail are accurate, it would appear that this case is not nearly what it seemed originally. Perhaps it comes with the pressure and responsibility of the job, or the belief that the network they’ve built is simply too complex for mere mortals to comprehend, but it’s not uncommon for highly skilled network administrators to become overprotective of their networks, or for networks of significant size to become an extension of the person who built them.

I’m trying to be as generous as the writer of the article linked to earlier but…this guy and I would seriously be butting heads over this. The guy sounds like a nut, and CCIE or no CCIE there is no way I would have let his security policies (such as they were) stand.

@Spiny Norman

I do not know of a Cisco password recovery procedure that does not involve a reboot, so you’d have complete access - to a blank router. Not a good thing for a production network. Incidentally, IOS 12.3 sports something very much like this as a feature (“no service password recovery”). You can still break into the router with the usual <CTRL>-<Break> during boot - but it will erase the startup configuration before the console gives you access.

From the article it seems he must have had a TFTP server out there somewhere with the configs on it that he was reloading (manually) in the event of something like a power failure. Crazy. And he also disabled (though I’m not sure how…must be a physical dip switch or something like that in the router/switch) the password recovery procedure so you couldn’t use control/break during IOS reload.

If no one has backups - or if the backups are of unknown quality, which is almost as bad - I can see why the mayor had a talk with the guy. Breaking in may or may not work, but the risk of failure is pretty darn high, and the consequences could be pretty bad.

Very bad. It’s, simply put, stupid, to rely on one person this much. I can understand why the guy wanted to ensure that no one screwed with his config, especially if he was the only one who really understood how it all works (which is stupid as well, though I’ve known a lot of IT engineers who play this game), but that’s what change control procedures are for. There is no excuse for this lash up of a system…though I don’t put all the blame on the engineer. The admin types were as stupid (if not worse) than he was for allowing this to happen.

I expect the post of CIO for San Francisco to become vacant Real Soon.

According to the article they have replaced him already. I wouldn’t want to be in the new guys shoes, personally. They couldn’t pay me enough to take over what seems like such a complicated system with little or no apparent documentation. Also, there is no way I would put myself in the same position as the sole engineer responsible for every aspect of the infrastructure…that position requires a staff, not a singe engineer, even if he is a CCIE. No way I’d be on call for everything 24/7/365…they couldn’t pay me enough for that.