Well?
Because the mirror is convex.
At least, the one on the passenger’s side is. If it wasn’t, you would have a very narrow field of view. The driver’s side mirror is closer and is usually not convex.
heresiarch is right; in order to provide a better field of view, the mirrors are (often)slightly convex, making everything look smaller (but getting more of it in).
Even if that were not so, the image would be slightly smaller as the distance from the object to the mirror to your eye is longer than the distance from the object to your eye if you were to turn your head. But not to a noticeable degree.