Good points, but bear in mind, people were smaller on average during this time. According to this link average male height during the 12-14th centuries was about 5’7, as opposed to 5’9-5’10 as is the average today.
You’re going to tell me that a chunkster of this type (a Highland Pony) cannot easily carry a 5’6" man and all his gear?
I’ve seen horses collapse under too much weight - they’re strong animals, but they have limits.
One man traveling light can go far and relatively fast on a single animal. Moving a large party with lots of gear… not nearly so fast. And terrain counts, too.
Also keep in mind that in the bad old days most animals were NOT championship material. They were, to put it bluntly, mongrels whose health and nutrition were likely no better than the average person’s. A king’s horse was superior because he could afford to buy superior stock and superior feed. The average traveler’s horse was not so well coddled.