Wrong. (IMO)
You are forgetting about sexual preferences. The human female prefers bigger/stronger/taller men so sexual pressure will cause future humans to be so.
Also, humans react more favorable to these attributes allowing them to achieve higher SES easier so such people will be more successful allowing for more mating opportunities.
I understand your logic, however, people underestimate sexual preferences. Ask the Peacock. 
By the way, dalovindj, are there many transhumanists around here?
I first learned about the Singularity when I asked AltaVista “What is the meaning of life?” That was back when they used Ask Jeeves’ software. I was a bit surprised to see it say “AltaVista knows the answer to this question.” But I’m certainly glad it did.
Now it seems silly to me when people talk about the far future as though it will be basically the same as the present. There are certain things that one must realize for one’s long-term predictions to be anywhere near accurate, and a lot of people just aren’t aware of them.
In this case, just let me reiterate that humanity’s future form will not be determined by natural selection. It will be determined by artificial selection. Technology moves must faster than the traditional mode of evolution. We’ll have a very advanced understanding of genetics and be able to alter our own genes extremely soon on an evolutionary timescale. After that, our genetic makeup will be determined by deliberate human choices. We won’t become weak or stupid unless we want to.
While there will be an obvious reason to (re)design oneself or one’s offspring to be more attractive, I think there is also a compelling reason to alter sexual preferences so as not to value physical appearance. So I think little gray men might still be plausible under some scenarios.
I have a sort of fictional history I have made up for several stories I plan on writing but never do, which covers app. 65 million years. Part of your idea is the origin of one of the races in my universe - app. 14,000 years ago, during a period of chaos in our part of the galaxy, an alien race sets up an illegal colony on Earth and starts taking humans and genetically modifying them to make them warrior-servants. Order returns to our part of the galaxy, the rogue aliens and their colony are destroyed (it is illegal to modify sentient beings in my universe), and the modified humans are all removed from the planet. They are given an uninhabited world by the galactic government to keep for themselves but are not allowed to colonize other star systems because it would be unfair to the unmodified races (competition for territory between sentients is encouraged). The modified humans are allowed to travel (as long as they are sterilized first) and act as law enforcement in our area of the galaxy, covertly protecting and guiding human society on Earth until we can join galactic society, hoping that with time we will become powerful enough that we will be able to have the restrictions placed on them cousins removed.
Of course, my modified humans are not short grey baldies with big heads and eyes, they are big (7’ - 8’ tall, 400-600 lb) black (adaptation to a bluer star than ours) psychic badasses.
I personally don’t believe aliens have ever visited Earth, but it makes for a good story.
According to the story I heard, Leo Szilard was present when Marconi made his famous remark to the effect that if outer space people exist, where were they?
“We are here, Enrico, but we are disguised as Hungarian physicists”