Bizarre Ethernet & Routing Occurence - Looking for ideas on WHY this happened this way - Nerds only

Factual Questions
1

First of all, the problem is taken care of, so that part isn’t important. The thing is, I’m pretty good on the whole R&S thing, but I can’t come up with a good explanation as to why this happened the way it did. So, looking for suggestions.

The setup:

A number of devices on a basic class c subnet. Devices include .1, .6, .7, .8, .9, and .18. The gateway is .254. All devices are on the same switch.

Now, I ping .18 from outside the subnet and my ping times are <1ms. I ping .254 and my ping times are <1ms. I ping anything else, and my ping times vary from 40ms to 500ms to timeouts.

From inside the subnet, all the ping times are <1ms.

As part of “I don’t know what the hell is going on so let’s see if we can figure out all the weirdness” I disabled the port going to the device at .1. And here’s where it got interesting - from outside the subnet I was no longer able to ping .1 (obvously), nor could I ping .6, .7, .8, and .9. I could still ping the .18 and the .254.

Now, the actual problem was that some fool (not me! yay!) configured the .6, .7, .8, and .9 devices with .1 as their gateway. Once those were corrected to .254 as the gateway everything was happy and good all around. So, problem solved.

The layer two stuff, sure, that all makes sense. Nobody’s pings on the switch should have left it anyway, so all of that acted as expected. The gateway is irrelevant.

But, here’s the thing that I can’t figure out. How the hell were devices responding to pings from off the subnet when they had the incorrect default gateway? They should have received the ping, seen that the source was from off their subnet, ARP’d the (wrong) gateway, sent the ping to .1, and then…

As far as I know the .1 device should have just dropped the ICMP packet. Which obviously didn’t happen.

Ideas?

-Joe

2

The .6, .7, .8, and .9 devices didn’t have an alternate gateway set, did they? What about the device at .1–could it forward a ping response?

3

Depending on what the device on .1 is, it could have either forwarded the packets as the poster above said, or it may have sent an ICMP Redirect to the host with the misconfigured gateway, informing it of the proper route.

4

goldmund’s suggestion sounds reasonable, so I won’t speculate farther on that.

Next time, when diagnosing the issue, you might try running a tracert to see what (if any) extra hops the pings are taking.
And, since this is a geek-only thread, I’ll mention this video which purports to show you how to use “tracer-t”: http://www.youtube.com/watch?v=SXmv8quf_xM

5

The trace routes didn’t show anything interesting. Last hop was a core switch where the routing happens, final hop was the device itself.

The device at .1 was a access control module that managed doors. So, nothing special that I know of. I wonder how I can tell if a device is even capable of an ICMP redirect?

-Joe

6

Ah, that’s unexpected. I would have thought you would have seen the route going though the .1 gateway, instead.

7

How old is that device? A lot of devices like this are embedded Linux devices nowadays. If it’s an embedded Linux device, it would quite likely send ICMP redirects. This behavior is controllable in Linux by the net.ipv4.conf.all.send_redirects sysctl variable or the /proc/sys/net/ipv4/conf/all/send_redirects file. This is generally on by default. I just checked my Motorola Xoom Android tablet, and send_redirects was even enabled on it.

When this is enabled, if another host is attempting to route packets through a Linux host, the Linux host will respond with an ICMP redirect if it has a valid entry in its routing table for the destination. This behavior is independent of the packet forwarding (routing) behavior in the Linux kernel, so a Linux host will do this even if it’s not configured as a router.

8

Interesting. I don’t know for certain that the host was a Linux host, but I’d put pretty good money on it being so.

Well, what do you know. I learned something new on this one.

Thanks for all the input, folks.

-Joe