Asking an African-American woman to give up a bus seat in the handicapped section makes me racist?

I always ask the youngest looking adults. And when the elderly white male driver asked them to move, they ignored him.

Finally, a white middle aged woman got up and gave me her seat. As I sat down, I said “Maybe they’re pregnant.”