I think we may be missing something in the men/women comparison.
On average men will win physical competitions, but what about the men who end up ranking lower than the above average women?.
Say you have a race and it ends up:
1st a man
2st a man
3rd a woman
4th a man.
The man placed fourth may, specially a century ago, feel specially slighted at being “beat by a gurl”, and that may have caused extra trouble and influenced the eventual segregation of sports.