You are. The average adult American woman is 5’ 4.1" tall. The average adult American man is 5" 9.5" tall. There are also many countries where the averages are slight greater.
I’m a “slightly short” too, being female and just under 5’4", but the experience I remember most in my white-American journey to the borders of maybe understanding something about what it’s like to be a member of a disprivileged minority group was from the first time I lived in India.
It wasn’t a height thing; I’m actually taller than average for an Indian woman, so visiting India (especially in the south) is an enjoyable opportunity for me to feel tall. 
It was the “second-glance” thing.
I had heard Black Americans mention the discomfort of being “looked at like you don’t belong there” in majority-white spaces, without really understanding what they meant. Now I (sort of) get it. In India I would be going about my business wearing typical Indian dress, not doing touristy stuff like messing with maps or cameras, not behaving unusually in any way, . And yet local people who initially perceived me as just being part of the urban landscape in terms of size/shape/general appearance, on getting close enough to see my skin, would very slightly double-take and look again, maybe stare a bit.
It was seldom or never an actively hostile reaction, and I certainly don’t think anybody was scared of my looks, much less dazzled by them (ha!). But even that very slight sharpening of attention was surprisingly exhausting to deal with on a constant basis.
AFAICT, humans have an instinctive alertness-spike reaction to being suddenly looked at, probably going all the way back to “oh no the predator noticed me” situations among early hominids. When that alertness spike is being constantly pinged by people who are low-level surprised by your presence in their familiar environment, it is tiring!
I can definitely sympathize with the OP; making fun of short men seems to be one dickish behaviour that society still largely tolerates. And shouldn’t.
I did want to say though – and I don’t think this is true of the OP – but a lot of short men fixate upon it and blame all their problems on their height.
The reality is that it is one factor among many. I’m above-average height, and I do think it gives some advantages, but I’d trade it to be handsome, for example.
And while it’s clearly a hassle needing to use a footstool to reach things that are designed for taller people, there are people who have to duck every time they walk through a doorway (and can’t scooch the seat back far enough).
Many manual labor tools have a man’s strength in mind and can be difficult for women to operate.
My point simply being that we all have to adapt and improvise in one way or another; the world isn’t perfectly designed for any individual.