I just bought a Garmin GPSmap 76s, and it is well suited for geocaching. It has a built-in compass (handy, because GPS doesn’t know which way you are facing when standing still), is light, has a decent sized screen, and is waterproof. I hate supporting Garmin because of all their proprietary crap (like their memory) but they do make a decent GPS. It was about $370, but you don’t have to spend anywhere near that much to get into it.
I used to have an Etrex Vista, and its a decent GPS also. TINY. The little yellow etrex is probably more than enough for Geocaching anyways. I am a geek, and always want more features.
If you do get a GPS, make sure to get a data cable and download a copy of ExpertGPS or USAPhotoMaps. They will download aerial photos from Terraserver, and show your tracks/routes/waypoints superimposed on the photos.
Talk about addictive. Maybe this deserves a thread of its own…