It ain't the network ya bastards

The BBQ Pit
1

So, I am a network admin at a casino. We have a pretty slick network. It is brand spankin’ new and is rock solid.

So, of course, every time a vendor has a problem the very first thing they blame is the network. It doesn’t matter that everything *else *is working just fine, it is (according to the vendors) a network problem.

The latest battle in the war against dumbass vendors went like this. The video display system that controls the marquee and a bunch of other things was having issues. And by ‘issues’ I mean it wasn’t working at all(#1).

Vendor Dumbass #1: Hey I heard you talkin’ about VLans(#2)…
Me: Yeah, what about it.
VDA#1: Well, the problem we are having is a network issue.
Me: How so?
Vendor Dumbass #2: Well, we keep losing communications to our boxes and that causes them to go down. It’s 'cause you got multiple VLans.
Me: How so.
VDA#2: Well, ummm, it’s cause our stuff won’t work with multiple Vlans.
Me: Let me get this straight. Your boxes have been working for 9 months. Our network hasn’t changed at all. Now, suddenly, the fact that we are on a multiple VLan network is the root cause of your system failure?
VDA#1. Well, its…
Me: Let me drive <takes over the vendor controller PC> Let’s run a trace. Hey, check that out, we can talk to your box from your controller. Would you look at that, there is no latency on the two hops to get the first box. Lets check the second. Hey, it’s the same thing, works just fine. Let’s telnet into the box. Would you look at that, I just got in without any problem at all. So I got a question for you.
VDA#2: Yeah?
Me: You say this is a network issue, right?
VDA#1 and 2: Yeah
Me: Well, we can talk to this box right now, right?
VDA#1: Well, yeah…
Me: M’K make it work now. We know we have a good network connection so that is out of the question, right?
VDA#2: <Fiddles around, loads a program that doesn’t work> It’s still not working.
Me: M’K. Ping the box.
VDA#2: I can still ping it.
Me: Then it ain’t the freaking network. <stomps off>

:::later on while I am punching down cable for an expansion in the same room that VDA1 and 2 are toiling away in::

VDA#1: Hey, VDA#2, I just found a bug.
VDA#2: No shit?

Gah!

The next time a vendor tries to blame my network for a problem with their software I might have to find the nearest Cisco router and beat the vendor silly with it. This isn’t the first time it happened. A couple weeks ago I had a vendor tell me that it was impossible for their system to work on our network. I called this vendor because 1 station out of 70 was having a problem. The other 69 stations were up and running without a problem but, according to the vendor, it is our network that is the problem.

Slee

#1. Actually, the outside marquee was working but only because if there is a problem it fails over to a local copy.

#2. For those who don’t know, Vlans are quite cool. Pre-Vlan, if you had multiple networks running out of a closet (think one network for POS, one for gaming data. You don’t want people on the POS network to see anything on the gaming network) you had to have a switch for each network. With Vlans, you can assign each port on a switch to a VLan so that a) you don’t have to buy multiple switches for a closet to run multiple networks. Port 1 can be POS, port 2 gaming and there is no way for anything plugged into port 1 to see anything that is on the gaming network. It is rather spiffy.

2

Don’t do that. Those routers are expensive. At my old job I kept a slide rail from a dell server next to my desk. It was my “user correction tool”. If I tried to ‘correct’ a user, and they ran I could just snap my wrist and the slide rail would expand out, and I would have better reach. No user got away.

My wife is starting to work on computer help desks. I keep telling her the first three steps of troubleshooting.

  1. Reboot the user.
  2. Reboot the user again.
  3. Reboot the computer

99% of problems go away after those three steps. She is starting to believe me.

-Otanx

3

As one of those pesky contractor vendor type service persons, I often get the same thing, but coming my way.

Customer: My whole line is shut down because the damned scale isn’t working!
Me: Let me check it.

: puts weight on scale, scale is dead-on-nuts :

: puts I/O Simulator on Output Card, all I/O firing at Presets :

: Pings scale from workstation, all packets sent/received loud-and-clear :

Me: It’s not the scale.
Customer: It must be the scale; we get interrupt faults at the conveyor exit! The scale isn’t weighing right or transmitting the weight data to the work station!
Me: It’s doing everything it’s programmed to do.
Later…

Customer: Uh, we found the problem. One of the workers adjusted the photo-eye reflector at the conveyor exit to the weigh station.

:smack:

In another instance, one of our customers bought a complex “kitchen unit” (multi-pot “cookers” with heating/cooling elements for precise formulations).

I hooked up the loadcells to the indicator and calibrated it for 25 Kg by .002 Kg resolution. It weight perfectly.

Then I get a call back from the customer several months later. It seems none of their formulations are coming out right, and it’s all the scale’s fault. So I hop in my truck and toodle on over to see what the problem is.

The customer had connected a high-torque mixer motor, a pneumatic valve and air line, and a water line to the “pot” scale. All with fairly inflexible lines. It was now only accurate to ~0.5 Kg. When the mixer motor started up, we were lucky to get it to repeat at ~2 kg.

:smack::smack:

4

Seconded. Use a UPS instead.

And if you land it right you can pin them to the ground so you can wail on them with a cat6 flail til they get a clue or pass out.

5

Hey sleestak, can I ask what sort of devices the vendor in your scenario was using? I’m not asking out of doubt, but rather out of a suspicion that I may have worked for that vendor’s company within the last coupla years…

6

I feel your pain. I’m an anti-virus admin. If it’s not the network, obviously the antivirus software is the problem. If the vendors had their way, we would never have anti-virus enabled on anything.

7

I’ve certainly had any number of cases opened with a basic premise of “Prove it isn’t the network!” - which serves as a convenient excuse to make the network team do the root cause analysis as well.

I’ve also had people doing their damndest to break every rule laid down for use of our network and then get very huffy that it must be a very poor network that won’t gracefully tolerate them connecting a wireless router with a DHCP server, for instance. It works in their living room, so why can’t they just take it with them to the office? People seem to understand that they can’t hook up their Mickey Mouse phone to the corporate phone network, yet they seem to think that they should be allowed to plug in the Belkin router they got from $5 bin at Fry’s. It’s to despair.

Heck, I’ve had a manager form the PC service department insist that our office network somehow broke the DHCP (not that she knew of DHCP, natch) on her laptop, because it won’t connect to her home network, only in the office. So obviously - clearly - the network that lets it connect is at fault for “screwing it up”.

Then of course, sometimes it is the network.Could be because a middle manager in a remote office has decided to purchase an app that works just great locally at 100 Mbps, but the overseas users take forever to download a 1GB file, and what’s up with that? :mad: Why can’t we just FIX it?

Gah.

8

They are called Symons. I don’t know much about the things other than they do signs and they are on our network*. Oh, and I’ve rebooted the suckers numerous times.

GAH!!!

On our opening night someone attached a little router to our network after the IT staff went into great detail to everyone that nothing was to be attached to our network except by IT.

Well, the router that they connected took down a large chunk of the network, the multi-vlan cisco routers and the Belkin router didn’t play well together. This, of course, caused a bit of havoc. Luckily we were able to trace it down to a certain area. At that point all the IT staff descended in mass to hunt down the offending device and remove it. The person who attached the device learned the hard way to *never *do that again.

Slee

*Someone at another of our properties deals with these things. We make sure they are on the network and powered on, everything else is out of our hands, thankfully.

9

I’m on the other end of things. I work for a vendor in the medical imaging business. Networks in hospitals and small imaging clinics are often some of the most horribly set up things you can imagine. And what’s worse, is that we often have sites where there are mobile PET/CT trucks that visit various hospitals that don’t have their own scanners, and trying to troubleshoot those connections is the most horrifically painful thing you can imagine. First of all, you have systems set up on tractor-trailer trucks that connect to a different hospital every day, usually using some cobbled-together set of batch files to change the network configurations.

Generally, in these situations, the hospital IT staff are completely worthless, the techs on the truck don’t know anything about networking (their job is to inject patients with radiopharmaceuticals and run the scanner software), and IT staff for the mobile company is hundreds of miles away. The network jacks are outside, constantly getting plugged and unplugged, so hardware problems are often involved as well.

So yes, sometimes it is the network. I love it when a customer complains about the transfer speed on their “high-speed network”. We run netperf, and it turns out they have a 2Mb connection between their central and remote sites, and wonder why it takes so long to transfer 500MB of data…

10

I’m with QM - it often is the network, but because the organisations I deal with have usually outsourced their WAN links to some deadhead comms company, my customers are always told “there’s nothing wrong with the network” as that’s the response that costs the comms company no time, effort or money. So I’m often in the impossible position of having to prove it’s not our products and is the network, over which neither I nor my customer have any control or visibility.

I can’t recall a single case in 20 years where a customer asked the outsource (or just as bad their own internal networking department) and got an immediate admission there was a network problem.

11
  • it’s not as if the network is the one magical IT component that never breaks, but it certainly gets a lot of flak for issues that end up being located elsewhere - or it turns out that people who really should know better roll out solutions based on unrealistic ideas of bandwidth and, (much) worse, latency.

Any organization that uses an application across any link without first getting a clear idea of the actual link performance as compared to the application’s demands has failed to plan. Link capacity, particularly WAN links, is expensive and time-consuming to expand - unlike disk space or RAM, it’s not something that can be corrected more or less on the fly. And in the meantime, it’s those network wankers holding everything back - again.

Interestingly enough, I used to make a lot of money (for my then-employer, natch) troubleshooting networks for hospitals and city offices. Until an organization reaches a certain critical size there’s no way it can afford to keep a narrowly focused network specialist employed. The days where you’d plug in the cables, turn the box on and get acceptable connectivity are pretty much gone.

12

<hijack>

I got introduced to this 3 years ago when we replaced our flat network with VLANs. We were replacing our physical infrastructure with VMWare and SAN so we actually needed more subnets. We got some Cisco 3750s and stacked them, then carved out our VLANs. Since we’re using layer 3 switches it meant we didn’t need a router for all the networks to talk to each other, but I guess since you don’t want your networks talking to each other that it wouldn’t matter much for your network. :slight_smile:

Just wanted to agree that VLANs are sweet.

</hijack>

Yeah, I fuckin hate vendors coming onto my network and blaming me for their product failures.

13

I have never heard of Netperf, I looked it up on Google and it looks interesting. How does it work?

14

Basically, you start the netperf server on one end, and the netperf client on the other. It does inbound/outbound bandwidth tests, and gives you a report of what kind of throughput you get across the link. Very useful.

This article tells how you might use it, and also includes a helpful link to a Windows version of netperf (it’s open source, but they don’t provide Windows binaries, so the article author built his own).

The most interesting thing we’ve discovered through netperf here is that our Macs and Linux machines can transfer data twice as fast as Windows across our internal gigabit network.

15

Is it wrong that I don’t know what you guys are talking about but it’s making me fluttery inside? Yum, geekspeak.

16

You did exactly the right thing. If you can ping it, and assuming you don’t have your VLANs firewalled or are using filters or some kind of quality control that is messing with the streams, it’s NOT the networks fault. I love vendors who try and pull that crapola. ISP vendors try this all the time…‘well sir, it must be your network’ ‘But I can ping your friggin DNS server you idiot…it’s NOT RESOLVING!’ ‘Um…let me get back to you…’.

I second an earlier posters suggestion that you not pound the vendor with a router…to valuable. Unless you have one of the old decommissioned big Cat (like a 6000 series chassis…fully loaded of course with extra power supplies), I’d stick with dropping the battery from one of your UPS on them…from a fairly high height. You could sit them down in a chair and then have someone drop the battery on their head through one of the over head tiles.

If that doesn’t work, assuming your computer room has a raised floor, you could set up a false tile, invite the vendor to walk over it, and drop him into a pungie pit…

-XT

17

YES!

YES!

and YES!

The number of times I come across customer sites who have no reverse DNS lookup zone is astonishing. Granted, thats not a network problem in terms of hardware, etc - but the same people who administer the network usually have a hand in DNS and basic name resolution. Once in a while I reach into the grab bag of customers and end up with a sharp tack - but most of the time I get a marble.