The human species in it’s natural environment (hunter-gatherer bands) is weakly polygamous, meaning most people only have one mate at a time but a few highly successful men might have multiple women attached to him at a time, but it’s not common as hunter-gatherers don’t amass the resources needed to have actual harems. It takes civilization - specialization, cities, and amassing of resources - to have multiple men with multiple wives being fairly common, and even then it tends to be the wealthy and powerful. Poor men might have no mate at all as the wealthy and powerful are keeping them all to themselves.
The upshot is that the size of men and women are less divergent than, say, in gorillas where one large male has a harem of females (for as long as he can hold that position - eventually he gets deposed by a younger, stronger male).
Sexual dimorphism ratio in great apes:
lowland gorillas: 2.37
orangutans: 2.23
bonobos: 1.36
chimpanzees: 1.29
humans (on average): 1.15
(It should be noted that different human groups range from 1.09 to 1.28 sexual dimorphism ratios, but in all cases that would still mean the size of adult men and women are less divergent than for all other great apes).
It should be noted, though, that mating strategy is not the sole determinant of sexual dimorphism. Horses are strongly polygamous, but have a roughly 1.1 ration for that, which means they’re less sexually dimorphic than humans, which are only weakly polygamous as a whole and often monogamous as individuals (well, as pairs).
When less food is available everyone who is smaller tends to survive better. See insular dwarfism for a classic example.