I’ve seen weird stuff like this happen a few times, and each time it was due to asymmetric routing… either there were dual network cards, or one of the hosts was configured to reply through a different pathway than they received the packets. Get your network admin to put a sniffer on both ends and see where the routing tables are boned up.
Application wraps data up to send out a TCP/IP port. The data is put into a TCP packet. The TCP packet is wrapped into an IP packet. IP does not guarantee orderly arrival. IP does not guarantee arrival at all. But magically, TCP does guarantee orderly presentation to the application layer.
joelonsoftware has a lecture-y article that talks about TCP which may help.
As has been said, if you’re using TCP and the segments arrive out of order, the worst that can happen is that you’ll get a RST and the connection will terminate. Under no circumstances would the network driver actually hand over a reordered data stream, that would be completely terrible.
If you’re not using TCP, and using UDP, then you can certainly get them out of order. That’s why you shouldn’t use UDP.
Also, programmers blaming the network without so much as a network trace to back them up really grinds my gears (no offense meant to the OP). Honestly, it’s almost never a network problem. Fire up wireshark and see what’s really going on.
That is not correct. TCP/IP can not re-asemble packets that arrive out of order. What it does is discard packets that arrive out of order. The protocol will cause the packets to be re-sent. This is one of the weaknesses of TCP/IP.
ETA: this is all occurring below the application layer of course.