I get WAY distracted if I’m talking to someone, and their eyes are flicking back and forth as they look into one of my eyes, then the other. It drives me nuts. Captain Janeway does it all the time on Voyager. My mother does it.
Me? I stare at the one eye, or look at one, then the other for a long time.
Just. Plain. Weird.
I’ll admit to being one of the eye-flicky types. If I don’t keep moving my eyes then eventually I end up slipping out of focus and I’m afraid my conversational partner will notice that and think they’re boring me, or that I’m not listening.
At least that’s my excuse. What’s yours? 
I look into their right eye if I’m actually listening. If I don’t give a hoot, I look between their eyes or at their forehead hoping they will take the hint.