I’ve been looking into this for a couple of weeks. I was hoping to buy a simple digital camera this weekend, but when I calculated out the money, I decided it was more important to, you know, eat. But I’ll get it soon. And at least now, my Dave Ramsey-style emergency fund is in place! 
I was looking at, among others, the Canon SD1100IS ($280) or the Casio EX-Z77RD ($160).
From reading reviews at www.dpreview.com and other places, it seems that small point-and-shoot cameras are loaded with all kinds of whiz-bang software features that may rarely get used. Several models I was looking at have a face-recognition feature which apparently picks faces out of the image and focuses on them. There are also things to tint pictures, modes that take movies, and all kinds of other things.
But the biggest thing that improves picture quality is a better-quality lens, and there’s no way to get that without adding expense. I suspect that a better-quality lens needs to be larger and have more parts, and that requires more mechanical and optical complexity, which is not getting cheaper the way electronics is.
Another thing I’ve read is that you can have too many megapixels. Apparently, on small point-and-shoot cameras, the image sensor is physically small, and as they cram more pixels onto it, the individual pixels get smaller, and therefore more sensitive to electrical noise.
Larger, more expensive cameras, such as digital “single-lens reflex” cameras (DSLRs), have physically larger lenses that don’t introduce as much distortion or colour artifacts. They also have physically-larger sensor chips that might not have as much noise on each pixel as a point-and-shoot with the same number of pixels. Of course, they are also larger in general, and more expensive. The cheapest one I’ve seen is around $600 Canadian, and the one I’d eventually like to get is around a thousand.
Electrical noise in a picture looks like snow on an old TV screen. When taking pictures in low-light conditions, a camera with more megapixels in the same size sensor will take a noisier picture than opne with fewer. For that reason, I’m going to get a camera with around 8 megapixels, rather than something with 10 or 12 megapixels.
Another thing to look at is what kind of memory card the camera uses. There are a squillion different kinds of memory cards, but the most common seem to be the Compact Flash card and the Secure Digital (SD) card. DSLR cameras often use the Compact Flash card. Sony cameras use the Sony Memory Stick, but not many others do. There are also the xD card, and probably others.
Each type of card has a number of variants as well. Newer variants are physically smaller and/or can store data faster.
SD cards seem to be the most common now among small cameras, and they’ve really come down in price. I saw a 16-gigabyte SD card on sale at Future Shop for $200! (For reference, that 16-gig card will hold roughly 4000 highest-quality highest-resolution pictures from the Canon SD1100IS camera (each picture is around 3.5 megabytes, say round it up to 4, so that’s 1000 pictures in 4 gigabytes, and 4000 in 16 gigabytes)).
The store also sells one 8-gig card at the same price as a two-pack of four-gig cards ($99).
Here’s the kicker though: many cameras can’t access all of a large memory card. It’s maddeningly-difficult to find out what the maximum card size is for each camera. The camera I’m thinking of buying seems to have a 4-gigabyte upper limit; that will decide which size of card I get.
SD cards 4 gigs or larger say SDHC on them; the camera has to be SDHC compatible to read the larger size. 2gigs or below are okay with both SDHC and non-SDHC cameras.
Now, the preceding was all to do with SD cards. If you’re getting a Sony, you’ll need a MemoryStick card. The same store has an 8-gig MemoryStick card for $169, so they’re a little more expensive, but not much.
Here’s the Wikipedia article on memory cards, but many of the formats it lists are not used in cameras.
Hope this helps!