What is going on with my network connection?

Factual Questions
1

I think I need a serious network engineer for this. I’d file an IT request but it takes them forever to escalate things.

I noticed some strange network issues on my work machine. I popped open Task Manager (this is Windows) and see that the adapter is receiving 80 Mb/s of data… constantly. It sometimes cycles off, but when it’s on it’s at a virtually constant 80.

I look for which process could be causing this, and… nothing. Nothing in Task Manager is using anything like this amount. I load up Process Explorer. Still nothing. TCPView–nothing.

Fine. Get out the big gun–Wireshark. Do a capture. At first I’m confused–I can still see almost nothing, even at the packet level. No IPv4 packets (except the usual noise), no IPv6, no UDP…

Explore around a bit (I’m a noob at this) until I run across the Protocol Hierarchy Statistics. Bam. But what the hell?
Imgur

“Data”, 99.3% of traffic and 78 Mb/s. But what is this? I searched around a bit and found that it could sometimes represent encrypted packets, like maybe WPA over WiFi. But this isn’t a WiFi adapter. It’s a 10 Gb Ethernet connection. Plus, the packets themselves look weird:
0000 00 01 74 00 00 01 74 00 00 01 74 00 00 01 74 00
0010 00 01 74 00 00 01 74 00 00 01 74 00 00 01 74 00

It’s like that over and over. Different packets have different data, but it’s all repetitive 32-bit chunks. Another:
0000 e0 00 b3 94 e0 00 b3 94 e0 00 b3 94 e0 00 b3 94
0010 e0 00 b3 94 e0 00 b3 94 e0 00 b3 94 e0 00 b3 94

It’s not all coming from the same address. But I do see only a couple dozen machines. Nothing obvious.

Is there a DDoS happening inside our network somehow? Any advice in diagnosing this further before escalation? Any of it look remotely familiar?

I should add that I have a couple of machines on a 10 Gb switch, which is itself connected via 1 Gb switch to the rest of the network. The traffic isn’t coming from the 10 Gb switch or anything on it. It’s all coming from outside. I’m not entirely sure why the 1 Gb switch is letting the packets into the local network. Shouldn’t it be doing some MAC filtering or something (it’s just a dumb, cheap switch, but I thought it would do filtering at some level)?

2

Sorry, been a while since I played with this, and never too in depth.

play with perfmon and resmon. (command line tools, run in the start box) Maybe they will tell you what’s happening.
Resmon seems to list bytes sent/received by task.

Has a task gone wonky and started sending excessive broadcasts?

A switch only does 1 thing - it looks at the incoming packet, determines which port on the switch the destination address is, and sends the packet out that port when it will not cause a collision. Broadcasts it repeats out on every port. (The switch will build a MAC table from listening to devices on the network, so it knows which of its ports it can find each device.)

Are you sure there is not a loop in the network, so the broadcasts run around in circles until they hit maximum hops or time to live? Many years ago that was a problem, but smarter hubs nowadays (what most bigger establishments have) will detect an error and shut a redundant port. Dumber hubs, just repeat Try unplugging parts of the network and see if it lessens or stops the problem, or go the other way, add pieces bit by but until the problem starts.

(That was usually the extent of my diagnosis procedures, and it seems to mostly have worked)

3

That’s just the thing–there is no task. Resmon just shows minimal usage. Perfmon does show the usage because it’s at the adapter level, but doesn’t have any detail beyond “there’s traffic”. Wireshark is the only tool that shows any further detail, but it’s confusing. They aren’t IP packets at all, let alone TCP or UDP. Just junk.

It does seem like maybe there’s some network misconfiguration. It’s possible it’s something to do with my 10 Gb switch, but it’s supposed to be in a dumb switch mode rather than a managed mode. The other switch has no interface at all; it’s just some cheap Netgear 1 Gb whatever.

4

Kinda rusty at this so large caveat required.

We know the traffic is not (directly) caused by your machine. IOW, these aren’t legit responses to legit requests your machine has made. So further digging inside your machine is futile, except as we’re using your machine as a network probe.

As you’ve said, the traffic is originating outside your personal mini-LAN behind your 10GB switch. And from further beyond the 1GB dumb switch as well. And seems to be in the nature of broadcast traffic. So stopping it at the source is utterly beyond your control. If you could filter it using firewall rules at the 1 or 10 GB switch you could at least get your personal LAN segment unclogged. Depending on your actual wider LAN network workload that may or may not improve the performance you need to do your job.

Rogue crap on a corporate network is a never-ending headache. For the people who’re paid to fix that. Which ain’t you. You’ve done your due diligence. Forward your findings to IT, implement traffic filtering at the gateway into your personal LAN if you can, then wait for IT to eventually solve the problem.

This might be the leading edge of some sort of malware attack and they might appreciate the head’s up. Depending on how far their own head is up their butt.

5

If you’re flooded with broadcasts that your machine apparently sees but ignores, then most likely somewhere feeding your switch is a loop where broadcasts go around in circles until they die. Just be sure your switch is not double-connected to the rest of the LAN, and then if you can’t do anything about that part, it’s up to IT.

Unplug your switch from the rest of the network and see if the traffic dies. This tells you the call is coming from outside the house.

I remember the days of hubs, where collision detection was an issue and so heavy traffic would significantly slow the network, and broadcast storms made network actions like reading server shares sloooooowww. In those days, the blinking light for traffic would give a good gauge as to how busy the network was. A solid very bright light was a bad sign.

I also saw slow performance once when one computer was at the far end of an extreme cable run. It was technically below the 100m / 300’ maximum for ethernet cabling, but having a patch panel halfway apparently caused the packets to fail more often than not. This was not so evident because the computers will retry over and over befor displaying an error. This was solved by putting another switch at the patch panel.

6

Is that the entire packet? It’s just the same 4 bytes over and over (different bytes in different packets, but the same general pattern)? There’s no real data in the packets, no source and destination address, just garbage?

My first thought is that an ethernet interface somewhere on the network has completely futzed itself. Somewhere on your network there is a broken machine that is sending out garbage.

It might not even be a computer. It could be a network drive or a copy machine or a web-enabled coffee machine or who knows what. But somewhere there’s a nic with severe brain damage.

Disconnecting cables in strategic locations to isolate different physical areas of the network should help you narrow down where the problem is fairly quickly.

7

Yep, that’s it, except repeated for >1kB. Sometimes the pattern changes partway through the packet, but it’s all that repetive junk.

It does seem like there’s some rogue device on the network sending out garbage. Though being outside my little localnet, I have limited means of debugging.

IT did actually escalate my ticket relatively quickly, after the usual rigamarole where they ask for the results of “ipconfig /all” and to perform a tracert to the thing I can’t reach (i.e., they didn’t read the request at all). The networking team knows what they’re doing, though.

8

That’s for sure. DHCP used to be a recurring problem–people plugging in routers, and other people’s machines would get their address from those. Haven’t seen that recently, though. They probably set up some filtering.

The constant 80 Mb/s is what gets me. That’s a nice even 10 MB/s. There’s no “natural” reason to see that number since it doesn’t saturate any of my links (though it does still cause a performance problem). Almost like something out there is intentionally sending out traffic at a specific rate for some reason.

9

IT gave me a graph of my port activity:
Imgur

Yep, that’s it. Bursts of heavy activity with short gaps. Didn’t realize it was quite that periodic, though. They see 100 Mb/s but I’ll chalk that up to differences in how they count packet size, etc.

I took a closer look at the junk packets. Since they aren’t IP, all I get are raw Ethernet frames with a dst/src MAC address. At first I thought they were legitimate. But on closer inspection, the addresses are the same junk data as inside the packet. Since a MAC is 6 bytes and the junk repeats every 4, the src and dst look to be different and not obviously wrong at first glance. But now it’s clear that the junk frames cover the whole packet, including the address part and probably the rest.

But the junk can’t cover the preamble, can it? Wouldn’t the adapter throw the frame away in that case? Not to mention a bad CRC.

Well, I guess maybe it is throwing it away since I can only see it in Wireshark. Still. And I’m not sure how they’re making their way through my port.

10

I wouldn’t think so. Whatever device is misbehaving may be framing the datagram correctly, but populating it with the same 32 bits.

Have you tried plugging your PC directly into the 1G switch and disconnecting the 10G?

Similarly rebooting both switches just to rule out a loop inside your Strangelove network.

Where is IT tracing at? Before your 1G or on it?

Since the MACs are garbage I’m not sure why the 1G sends to the 10G. Unless they are part of the issue.

11

That may be the most valuable insight of the thread to date.

12

The network topology is that there’s a single 1 Gb port to the corp network, which is all they can directly monitor. Into that is plugged a cheap 1Gb switch with a few misc devices on it like a KVM and networked power switch. Into that is plugged a 10 Gb switch, and into the 10 Gb switch are my two main machines (which constantly sling large files back and forth, hence the 10 Gb).

I have tried power cycling both switches to no effect. Haven’t tried changing ports, due to me not actually being at the office. Though if the problem continues I may have to go in.

I agree that it’s very mysterious that both switches are sending those frames to my machine. The answer to that may answer the rest.

13

in a DOS box, >ARP -a will list the ip and MAC addresses of all known devices your PC has encountered within the default gateway. (If you were getting that much traffic on the WAN, I’m sure IT would have been all over it. Usually that stuff costs $/MB)

Possibly it’s being seen as broadcast, hence the repeating to everhwhere. Also note that switches build a MAC table to determine what port each local MAC is on, so the when they see an unknown destination MAC they will send it out every port until some machine replies “That’s me!”. Switch relies on the sending device being smart enough to give up soon if it does not get a reply from the target MAC. So switch gets these packets, assumes a MAC “to” field from the gibberish(?), and waits to hear the reply on one of its ports. But somehow these packets keep coming. Loop or wonkey device.

Yes, unplug the connection between your switch and the rest of the equipment (i.e. between the 1GB and the 10GB.) Does that stop it? Tells you what side it is on. Repeat, by unplugging each connection to the switch it is coming from until you find the culprit.

It could be anything, so much is network connected these days (printers, copiers, alarm systems, wifi hotspots, …)

Unless you can get meaningful data from the packets, it will require hands on.

I remember one time finding a device - network card in a 486 computer, that’s how old it was - that just wouldn’t shut up. But because it was spouting valid packets, they flooded the local plant-wide WAN. Found it in those days by pull-and-check…

14

Do you have any other devices plugged into the same network or even electrical circuit that could potentially be emitting RF noise? Powerline networking, MoCA type stuff (digital over coax), etc.?

If you connect that one computer to the 3 switches and unplug everything else, including the other computer, does it still happen?

Does it still happen if you bring the one computer and 2 switches into another entry point into the corp network in a different physical location, preferably a different building, does that still happen?

Also: Cheap test: Does it happen if you use a different network card (like a cheap USB ethernet dongle) with your computer?

15

And the problem disappeared just as abruptly as it appeared.

“arp -a” doesn’t show anything because there was never any IP address. No way it could have been arriving via the internet–had to have been something on the intranet. But a broken device of some kind does seem plausible. Or possibly some kind of malware or even just a strange internal stress test gone wrong.

The machines and their network are rather… large, not to mention my primary work machines, so I can’t exactly cart them off to a different building. Though if the problem comes back, I might grab a long network cable and plug in somewhere else that’s not too far away. The ports are numbered so I might be able to find one on a different switch, at least.

Something like RF noise isn’t totally impossible but the fact of the junk packets repeating every 4 bytes makes me think it’s not that. Plus, it wasn’t exactly random looking… more zeroes than I’d expect if it were random.

Sorta amusingly, one thing that confused me at first is that Wireshark does MAC vendor decoding, so some packets said “Cisco”, “Ampex”, “CyberOptics”, etc. This made me think they were at least partially legitimate. But no, it was just random overlap between the MAC vendor prefix and the junk.

Anyhow, thanks for all the ideas. I will have a few experiments to try if it comes back.

16

Like I said, when the MAC information is not useful, the simplest quick-and-dirty step is to unplug stuff from the network - even just for a minute - and see if that stops the problem. If you have a local and other switch, disconnecting the connection between them for a minute will tell you if the problem device is on your local switch or elsewhere. If it doesn’t stop immediately, plug that item back in and try the next, working your way through the connections to that problem switch one at a time.

I’m sure just about anything except the gateway and the server would not be severely disrupted by being disconnected for a minute at a time. Just make sure someone’s not in the middle of a big print job when you disconnect the printer.

Fancy smart switches will usually disable any port that shows problems. Most lower end switches, won’t.