SQUEE! You have a deer mouse!
At least I’m pretty sure you do, from the photos. The deer mice have larger eyes and different build/coloration from regular house mice, and in fact are more closely related to hamsters than to house mice. They are very pretty, sleek, elegant little things-- smart too-- and they “sing” to each other in the wild. (The Peromyscus Genetic Stock Center has a lot of great info about the various deer mouse species and their ranges.)
From what I understand, deer mice naturally prefer the outdoors but will come inside if they have the opportunity for a better living spot (as distinct from house mice). Your mouse looks young enough that he may have been born inside the house.
We caught a young deer mouse (actually, I believe, a* P. leucopus*, or white-footed mouse) in our kitchen in a very similar way in the summer of 2006. His mom had ventured inside because of a very rainy summer, and had her babies under our stove. We caught and evicted the momma mouse and three babies, but left the live trap out for good measure, and caught Ne-taro (“Mouse First Son” in Japanese, a proper samurai name). He was young enough that we decided to try raising him ourselves. Although he was never really tame enough to be happy being picked up or anything like that (for that I blame not handling him enough when he was little), he got accustomed to us, and enjoyed watching us (and we him). We kept him quite happily until he escaped one day early this year-- I think he may have gotten out through the same hole under the stove, which we hadn’t patched very effectively. 
A note of warning: deer mice are potential carriers of hantavirus, depending on where you live, so please do some investigating and factor that into your decisions. We live in a region where it is not endemic (Northern VA) so we decided to press on-- but you may want to check with your local Fish and Wildlife department to get the full story in your area.
If you do decide to keep him as a pet, check out here (awesome baby pictures!) and here for advice and tips, and here for a Flickr set about a wonderful rescued baby deer mouse named Stu.
Sometime down the road, you may want to consider getting your mouse a “friend” eventually. We and other folks have had some luck keeping deer mice and domestic (house) mice together: they generally get along very well, and can’t interbreed. We ended up adopting a trio of lady mice from a local rescue, and (after an initial few days of nervousness on both sides) Ne-taro ended up totally smitten with one of the ladies, Allie, and vice-versa. They slept together, ate together, and were pretty much inseparable.
I don’t claim to be an expert, but we did really enjoy the unexpected addition of our little guy into our lives, and I’d be happy to try to respond to any questions/concerns you might have (and the above-linked Deer Mouse Ranch page has a great message board that can provide better information than I can).
Good luck with your new little friend, and may he be as good a companion as Ne-taro was to us. I miss him.
ETA: our guy liked to 1) chew, and 2) climb-- so it would be a good idea to make sure the enclosure you are using is amenable to both. We found a Martin’s Cage to be a good solution.