I think it pretty much can be. Contraception and paternity testing and a society so rich that even single mothers can survive without the help of their family are all pretty recent developments, on a timescale relevant to the development of our brains and our culture.
Females, through childbirth and breastfeeding, are unavoidably associated with their children. Males not so. Until recently, the extended family was the basic economic unit:
1/ If your daughter or sister got casually pregnant, you would have had to support her if the impregnating male didn’t.
If your son or brother got a female casually pregnant it may have caused you economic stress but the female probably wouldn’t have been able to prove anything and anyway even if she could, she may not have been able to force you to support her, meaning you pass on some genetic material without having to “pay” in terms of providing support.
2/ If the mother of your children got casually pregnant you may not even have known that you are bringing up someone else’s genetic material, and even if you knew, if she’s also the mother of your children you may still have had to support her thus diluting the economic support you could give to your own offspring carrying your own genetic material.
If the father of your children gets some other female pregnant, the other female probably wouldn’t have been able to prove anything and anyway even if she could, she may not have been able to force you or the father of your children to do anything about it, thus no dilution of the economic support the father of your children can give to your children.
3/ If your mother gets casually pregnant you may not even know that you are helping bring up siblings that share only half your genetic material, and even if you know, she’s still your mother so you may still have to support her children as part of the extended family thus diluting the support you can give to your siblings who have more genetic material in common with you than your half siblings.
If your father gets some other female (not your mother) pregnant the other female probably wouldn’t have been able to prove anything and anyway even if she could, she may not have been able to force you or your father to do anything about it.
The whole thing is too asymetrical for it to be surprising that we might be culturally and mentally less comfortable with females having casual sex.