I’m interested in making something along the lines of an electronic dictionary, where basically I would be able to program some handheld device with a keyboard to respond to text input from the user with specific text. I’d also like it to be cheap (I found this dictionary for $20, so I’m guessing the hardware by itself would be around the $5 range). I’m a fairly adept programmer, but mostly in third-generation languages (I hacked my way through assembly) and I guess at the minimum it should have the capacity to store a text file to look up “definitions” in. Is there anything like that in the market place?
Usually anything like this will have the information stored in ROM and not easily editable. Do you want to mass-produce this, or is it just for personal use?
If it’s just for personal use, you could look for a used Psion Series 3 or Psion Series 5 on eBay, as these are real handheld computers which can store data on CompactFlash cards and so on. I know there are various reference toolkits you can download to put on a Psion.
Not so much mass-produce as mini-produce; in other words, I’d like to make about ten. 
Those Psion seem interesting. I’ll have to research them more.
You can also get an old HP 100LX or 200LX. I’ve programmed quite a bit on these, and if you’re just doing text-based stuff, it’s DOS compatible so there are a jillion tools for developing, for free.
Both handhelds hold possibilies, esp. the DOS compatible one, but I had really been hoping that some kind of generic raw new device that I could get about ten at a time or so, kinda like the above-linked item, only blank and programmable. Kinda like a laptop kit you put together yourself, on a much smaller scale, obviously. Heck, with the right instructions, I’ll hack an existing device, a la XBox! I probably could even handle using chips, since I’ve worked with EPROMs before (and I know some company produces Linux on a chip).
But I don’t even know where to begin to look.
Raw new device? You would have to write the entire OS from scratch. Just use an existing, consumer model and put whatever you want on top of it. Some of the sharp handhelds run linux which is probably the closest thing to a raw device you can get. Their is an ongoing linux on palm project but I’m pretty sure you still cant do anything interesting on it yet.