If you hold up your left hand, the man in the mirror holds up his right hand. If the man in the mirror looks to his right, I’m looking to my left. So left and right is inversed when looking into a mirror. But why isn’t up and down inversed? Up is still up to both myself, and the handsome man in the mirror. Why is that?
A mirror doesn’t swap left and right - your right hand stays on the right side of the image, just the same as your feet stay at the bottom.
A mirror swaps front and back - your problem is rooted in the idea that the image in the mirror is a solid person with a right and left hand - it isn’t - it is an image of you that has been reversed in such a way as to make the parts of you that are closest to the mirror appear closest in the image and make the furtest parts of you appear furthest, and no more.
Some links:
http://math.ucr.edu/home/baez/physics/General/mirrors.html
http://psycprints.ecs.soton.ac.uk/archive/00000161/
http://ddll.sdf1.net/archives/001827.html
http://www.redrice.com/ci/mirrorProblem.html
http://www.hants.gov.uk/hantsweb/stoat/mirrors/answer.html
The most satisfying answer to me appeared in Cecil’s column on the subject (I’ll let you search). When you view yourself in the mirror and compare to what you would look like unreflected, there is an implied rotation going on. People normally mentally rotate this comparison around the vertical axis (the yaw axis if you will), which swaps left and right.
But it’s just as valid to view the mirror as swapping up and down instead of left and right. Just when you do that mental rotation, rotate around the horizontal axis. If this doesn’t make sense, try it with an object you can hold in your hand. Look at in in the mirror, then rotate it 180 degrees so you can see the front side directly. Now turn it back, and rotate it vertically instead of horizontally, and you’ll see what I mean.
I find it easiest to think about this question in terms of absolute directions (north, south, etc.), rather than relative directions (left, right).
For instance, my mirror is on the west wall of my bathroom. So when I’m looking into it, my right hand is to the north, and my left hand is to the south. Well, when I raise my north hand, the image also raises its north hand. Likewise, my south hand corresponds to the image’s south hand. So north and south are not reversed. But if I move my hand to the west, my image’s hand moves to the east. So east and west, which correspond to the front and back of the mirror, are reversed.
This is also why most people tend to think they look odd when they see themseles on video or in pictures. It is the only time you see your own face without it being reversed.
Stay away from mirrors.
They turn you inside out.
Marilyn Vos Savant was asked this question in her column a few years ago, and she gave the same answer - you look different in photos because you’re used to seeing yourself reversed. I think this is wrong.
I’m used to looking at myself straight-on. In photos, when I look “different,” the angle is from something other than straight-on, which I’m not used to looking at. If I see a photo of myself that’s taken like a mug shot, I think it looks like me.
I’m amazed by now many people get hold of this weird idea that mirrors somehow “swap left and right”. 
They don’t. At all.
Say you have a wardrobe to one side of your mirror. Point sideways towards that wardrobe, your reflection does too. What’s weird about that?
Just think about how a mirror works. It reflects any image that falls upon its surface. So your right hand is nearest the right edge of the mirror - it’s image will be reflected on that side too.
What would be really weird would be if you raised the hand closest to the right edge of the mirror, and your reflection raised the hand closest to the left side. If that ever happens, call the exorcist.
You can see yourself reflected the ‘right way round’ if you arrange two mirrors at right angles. Feels weird, and good luck shaving with that setup.