Allrighty, here’s a weird one.
I’m at a client’s site. This place runs about 400 IP cameras scattered across their gaming floor. Everything generally runs pretty well. All but some specialty cameras run off of PoE. All the units with issues were running PoE.
Yesterday, about 1:30PM something weird happened. About 10 of their cameras started acting strangely, coming and going in the recording system. I checked out the switches, and I could see that it wasn’t a connectivity problem - rather, the switch ports on these things were registering the links going up and down.
I came out today, and looked at it. What I found was that these units were doing (according to the lights) was losing power, and then recovering it, once or twice a second. That’s a lot. The switches generally weren’t even registering the up/down on the links unless a camera managed to get to the correct sequence in their boot phase. Not all of them are doing this the whole time, of course. Some of them are staying up long enough for the switch to establish a link, and some are even staying up a few minutes as evidenced by the fact that I occasionally get a few minutes of video.
So, weird. I tried moving some stuff around, and the problem didn’t stay with the switch ports, so it wasn’t the switch port. That means it’s either the patch cable, patch panel, horizontal cable, or the actual camera. Just to make sure, I put an external PoE injector and it still did the same thing.
Checked out the connector at the camera end. It looks fine, but I change it anyways. At this point, this is where I notice the actual on/off once or twice a second. These units have a power light and a status light and a network light. The power light (which shows green when there’s power and nothing when there isn’t) was flashing like crazy.
So, it’s some sort of weird power issue. Gotta be. Hmm. The cameras are PoE, and they’re on three different switches. So it can’t just be the switch. Those switches are all plugged into the same UPS, BUT there’s a fourth switch plugged into that UPS as well. That fourth switch has no cameras giving problems.
Now, what fixed the problem? Something that makes even less sense than this whole problem to begin with. I was checking the patch panel and, this being one of the type that is a frame with slots for jacks to be snapped in, I popped this particular cable’s jack out and take a look at it. Looks good. I just squeeze the boot on a little tighter and snap it back into place. Plug up the patch cable. Good, solid link light. Pings coming back 100%.
Success! All of this crap is because something funny happened with the patch panels, apparently. Someone must have been poking around back here and leaned up against the panel or something. The important part is that I know what the problem is and how to fix it. Who the hell knows why 10 of them went bad at once? At least I’ve got it figured out.
Except I don’t. How do I know I haven’t figured it out? Because now ALL TEN OF THEM ARE WORKING PERFECTLY.
These things are on several different patch panels. They are on several different switches. They are all, obviously, running on different cables.
What. The. Hell?
Anyone?
