I accidently left myself an answerphone message yesterday and was shocked at how my voice sounded over the phone. I don’t often hear my own voice back to myself, and it sounded so so different to how I think I sound in my own head. Why is this? Is there a psychological or a physical reason?
Not only do you hear your voice through the air like everyone else but also through your skull(sinuses, bone, cartilage).
When you talk a lot of the sound travels through the bone and flesh of your head to directly vibrate your inner ear. This gives it a different sound than if you just heard it through the air like when other people speak, or when you hear your voice through a recording.
Ah okay. This is probably more of a mundane point, but are most people suprised by how they sound?
I think so, yes.
I can be. Mostly I think I sound like myself, but I have heard recorded examples of me speaking where I felt the same way you did and wondered “Is this how I sound to other people? Yeesh!”
This topic was discussed in the Ask the Expert section of Scientific American a few months ago: Why does my voice sound so different when it is recorded and played back? - Scientific American
There’s probably a psychological element as well, where while you are speaking you are also thinking about what you are saying and this factor might influence how you hear yourself. But when you hear a recording of your voice you are hearing it out of mental context and in its pure form like everyone else.
Just a wild guess.
When singers like Billy Joel or Paul Simon or whoever listen to their own records, do you think they go, “man, this isn’t how my voice sounds!”
What surprises me when I hear my own voice (as recorded) is how much like my father I sound - not just timbre, but mannerisms as well. <shudder> 
And singers do do that <huh> thing, at least initially. However, a singer who regularly uses foldback and listens to their own takes does get adapted to the sound of their “external” voice.
Si
Also an odd experience: listening to a recording of yourself talking with other people, and you say “everyone sounds normal but me”, and everyone else also says “everyone sounds normal but me”.
I can give you something from an even more impeccable source: Why does my recorded voice sound different from my "live" voice? - The Straight Dope