That was a bitch of a week. (LONG and geeky)

The BBQ Pit
1

Starting just six weeks ago, I have a contract to do network and system maintenance for a couple of schools in my area. There’s something like 1000 users in all. 800 or so students, admin personnel, and teachers.

Since I’ve started there, I’ve learned the hard way why the last guy left - or perhaps should say “was sent away.” There’s a golf course just across the road from one of the schools, and it seems he spent a lot of time there.

He left in a great tearing hurry this summer, and I ended up with the position. That’s already real fun: big fucking mess, and no transition. I got passwords and stuff from different folks around the schools, so that now I can at least manage the stuff that’s here.

The start of the school year is always somewhat chaotic, what with getting the students in to the system and keeping normal glitches cleared up.

This year has been a real topper, though. Besides a huge fucking mess in the network and the system admin stuff (students could actually read and write to the teachers’ home directories, amongst other sins,) we have several major software upgrades and a district ordered switch from Novell to AD.

This makes the job a real bitch - and a half.

Top this off with a district wide “service desk” that mainly serves to document the sysadmin work load ,and you have a real fun situation.

With all that has to be done, I am naturally way behind on my assigned requests. Never mind that the folks in the school practically worship the ground I walk on. If the service desk is unhappy, there ain’t no joy for me (read, my temporary three month contract won’t get renewed.)

To avoid this unhappy circumstance, I worked last Friday from 0700 until 0400 Saturday morning. The nighttime was especially productive, as I could zip around fixing things without worrying about disturbing classes or getting buttonholed by a teacher with “dire emergency.” I reduced my list from 28 tasks down to about ten, and all but one of those were waiting on something external - parts, delivery of new PCs, etc.

So, I’m pretty well caught up.

Comes Monday about noon, and I get a really snide message from the head of the service desk to the effect that I’ve got over ten requests over a week old and that I should get cracking.

I start to write back and set the fellow straight, and get interrupted by a user with a real problem. The rat race heats up again, and I don’t get a chance to finish the message. About 1500, I’m in the tech committee meeting and the power flickers, goes out, comes back, goes and again and then finally stabilizes. Oh, joy.

I am immediately deluged with calls to the effect that the internet is down and god knows what all else.

First problems were easy - a switch locked up and needs reset, people’s PCs didn’t really like having their juice taken away, etc.

The internet connection is really down all across the school, though. I go check, and my modems are both showing green across the board. OK. We’re connected through another school across town (don’t ask.) It is now 1800, and of course there is nobody left over there. I call the principal, and he sends the secretary (who lives just across the street from the school) to let me in. The modems are all green there, too. Shit. Now what.

Here follows two days of calling every one who could possibly be of help, and getting nowhere. I fianally manage to get hand books for the modems (Pairgain RS 530 Fractional running at 1544 KBPS, if you must know) and find that the previous sysadmin was a hound and a bastard - he had shut off the fucking system alarms. You could pull every damned wire off the bastards and they wouldn’t complain.

The service desk finds someone who knows something about the modems, and the guy who has access to the routers. With one guy coaching me for the modems and the other telling me what the router says about the state of the modems (conference calls can be so handy,) we finally find that the connection from the modem to the router at the other school is buggered. Well, fine. There’s a spare right there. A second modem that used to connect to another school that finally got their own real connection. I plug it up, and we find that the power surge got it, too. 'Cause see, nobody bothered disconnecting it after it was taken out of service. Its been sitting there merrily sucking juice and waiting to get toasted for months.

So, now what? Well, the other school where I normally work has a couple of unused Pairgains laying around. I go over and pick them up. They used to connect my two schools together, but they’ve got glass fiber now. I plug in the one modem that can connect to the router (physically, since one PG had ethernet and the other serial connection matching the one on the Cisco 3600.)

Hallelujah! The Cisco guy now has a good connection to the modem through his router. The modem guy and I go through the settings and get it ready for the connection to my school. I drive back over to my school and check the settings. Plug the crap in, and me and the guy at the other school both reset the modems, and looka there. All green (for real, I reactivated the alarms) and the Cisco guy can connect to my router for the first time in almost three days.

Hot damn. A quick announcement at both of my schools, a short explanation to the principal, and its time for a well deserved lunch.

And now for the pisser:
The first message I get when I come back is from the head of the service desk reminding all of us system admins to please keep our task lists upto date and try to clear as many old items as possible.

Gee, fucking thanks. Two days long you clowns leave me with the problem, and now you are bitching at me to get the other shit done.

Ass wipes. My job description specifically says that that kind of stuff is out of my hands, and will be handled by contract techs from the communications company - except these relics of the last ice age belong to the school, not the comm guys. The sorry assed service desk doesn’t (officially) know squat about Pairgains, and there ain’t shit for documentation in any of the schools.

Oh, and you know why the power outage killed my modems and caused so much hell?

The putz I replaced didn’t bother connecting any of the equipment to the installed battery backup units. There’s a fuckin’ huge UPS built into the bottom of each rack, and every goddamned power cord on every piece of equipment goes out to a normal outlet.

One day very soon, I am going to have to take the switches out of service for a few minutes on each rack and reconnect all the cables - just not tomorrow. I’ve got to take care of all the dicky shit little requests so that the service desk will be happy.

Wish me luck and pray that nobody knocks over a telephone pole before I get the service desk off my ass.

PS:

I’d report all the problems I had, and put in requests to get the work done on the racks, but I can’t. The fuckin’ service desk software isn’t accepting new requests right now. They’ve decided to add a mandatory field that only accepts predefined values - and haven’t gotten around to actually setting up the list. Since the list is empty, I can’t possibly enter an acceptable value.

I called them up to complain. I’d have given them a piece of my mind, but as punchy as I am I need all that I’ve got left.

Update:

I couldn’t post this the night I wrote it, as the SD boards were down by the time I got finished writing it.

It took another day and a half for the service desk to get straightened out.

I’ve got five gazillion new tasks that have been added, and the ones that I put in are rather far down the list.

Oh, well.

2

Ouch!

Query: modems? So you’ve got some sort of high speed connection at one location, and you have the other locations dialing into the hub? I’m not a network admin by any stretch, but that gives me a headache just thinking about it.

As someone who has worked with computers for the computer-inept, I have worn those same shoes. Putting out the big fires, and then getting grief when a few small fires get put on the back burner (I’m sure I just broke some mixed-metaphor rule there).

You’d think that the service desk people, who I assume are also at least mildly technically savy, would understand and cut you some slack.

3

Yeah. One school (where I don’t work) has this tremendous high speed leased line that connects to the school district’s main network, and they have a direct connect to a backbone somewhere. My two schools are linked together via glassfiber. The Highschool then connects over to that thord school via a rented analog telephone line that has these two extreme high speed modems talking over it. 1544 MBPS is not your average modem. There’s no dial in as such. The line is dedicated, and these two modems only talk straight to one another.

Anyway, I’ve caught up on some Z’s this weekend, and I’m making way against the flood of small shit. There’s a couple of large projects to finish, and a new one that cropped up Friday. At this rate, I’ll be the last one still working when the good Lord calls us all home.

4

Are you being fairly paid for doing this doing all this clean up work? Is the work you’re doing within the spec level of expertise called for in your contract?

5

Truthfully, no.

All told, this level of crap is probably way above anything that is contracted for.

The problem is that this is a three cornered deal with a real squishy chain of command.

There’s the schoold district which supplies the network and sets the specifications and manages a lot of the network crap - I don’t have to worry about filtering the network to keep the kids off of bad sites, and the routers and switches are all remotely administered.

The district has a contract with a fairly large company to provide on the spot techs, and I am subcontracted to them.

Then comes the schools where all of us actually work. They don’t have any direct say in the matter, except that we’ve got to keep the end users happy.

I spent the first few days trying to find out who my actual boss is, and the answer keeps coming back “Just the help desk.” Except of course, that there are several other people whose suggestions I’d better take as commands.

The position itself actually only requires an A+ certification, which isn’t all that big a deal.

As best I can tell, though, through talking to people who have contact in other schools, I’m doing a pretty bang up job. I hear horror stories of things being much worse (as far as the changeover goes) in other schools. Hey, I must be pretty hot shit to manage all of the old problems and make the transition so well ;).

Anyway, yeah. Most of this comes under the heading “Doing what it takes” rather than “fullfilling the letter of the contract.”

6

Heh, schools can be the worst! I don’t know about where you are, but here in the land of oz, many of our schools’ “IT departments” consist of a teacher who knows how to type better than the rest of the teachers.

That’s fine most of the time, at least they know not to mess with anything.

I recall one school that had a guy called umm… let’s call him Darryl (as in, my name’s Darryl, this is my brother Darryl, and this is my other brother Darryl). Darryl thought he knew everything there was to know about networks. We gave him a perfectly working Windows NT 4 network (hey, it was current back then!) and within a week it was broken beyond salvation. Darryl insisted it was a virus. It had deleted all his users’ desktops, they couldn’t click on anything, no programs could run etc etc. I asked him if he’d run a virus scanner. He’d run three, but none of them found anything.

But it was definitely a virus!

After a bit of exploration, we determined he’d configured a system policy that removed all shortcuts to everything, prevented you from running anything and, then, just to top it all off, prevented clients from getting registry settings from the system policy.

But it was definitely a virus!

We ended up having to do a registry hack on every single workstation (a hundred or so), followed by rebuilding the PDC because Darryl had, in his wisdom, deleted a bunch of system files. And yes, of course, disjoining and rejoining the workstations to the domain.

But it was definitely a virus!

Sure was. We came to call it the “Darryl Virus”.

:smiley:
Max.