First, an aside to Eve: water in the NYC area is some of the purest around. That may be why your friend is talking down about household water filters. Outside of NYC, household filters, and water-softening systems can be very effective.
Now, to Kalhoun: First, off, are you talking about getting a filtration system, a water softening system, or a ion-exchange system? These are the three basic household technologies I recall from my time as a travelling water thief. You can get systems that use all three methods, and two pronged approaches are very common.
Each one deals with some of the problems you have through different methods.
[ul]
[li]Filtration - basically the water passes through something, anything really, that interrupts the flow of the water and removes solid particles from the water. This will address some of the problem with rust in your water, which is one common source of ground water problems. It does nothing (by itself) for dissolved contaminants, however. The simplest filtration units use a core of basically string through which the water has to pass to do this. And, again, it can get some very good results, but it’s not going to do anything about calcium carbonate - the most common source of ‘hard’ water.[/li][li]Water Softening - this is a particularly odd term, since it implies just the opposite of what it does, IMNSHO. It usually works by adding something to your waterline to neutralize the calcium carbonate that is the biggest factor in forming lime. I can’t recall the exact chemicals used, but I’m sure any company that deals with water softening systems will be able to answer that. It’s another fairly old technology, and works for what it attacks. But leaves the particulate in place in your water, which means it will not do much for taste of your water - but it will leave you, your dishes, and clothes feeling a LOT cleaner.[/li][li]Ion exchange - this is where you pass the water through an artificial plastic ‘sand’ that is covered with a chemical that will exhange either H[sup]+[/sup] ions or OH[sup]-[/sup] ions with the ions in solution. This is a bit more expensive than most solutions, but it’s also very complete about removing dissolved chemicals from your water. It will have the effect of making your water softer by removing the ions that make water hard, and will also act as a filter, just by passing the water through and around the solid beads that the exchange chemical is bonded to. And, with most resins you can ‘recharge’ them simply by passing a commercially available salt solution through the resin housing. [/li][li]finally, the old reliable: Activated charcoal - not quite fish, nor fowl. It still works damned well, and if it’s not as complete as an ion exchange resin for removing objectionable dissolved ions, it’s still very good at it, and because the charcoal is so irregularly shaped, and usually used as part of a filtering system in a two-prong approach, it works very well at both removing ugly stain making solids, too. [/li][/ul]
Personally, unless you’re looking for laboratory grade clean water, or have specific allergy or chemical sensitivity concerns, I’d suggest going with a simple filter with an activated charcoal core. These systems can be installed, as danceswithcats says, very cheaply and easily. If your well water is as bad as your OP implied, you should see some pretty dramatic results with any system you choose, actually.