Does size really matter?

I’m about 5’11", and though I’ve never dated anyone taller than 5’5" or so, I don’t think I’d have a problem with any girl under 6’1" or so. Above that, yeah, I start to get irrationally uncomfortable.