I recommend you look into using a Mini-disc (MD) recorder for this purpose. MDs will let you do all that you say you want to do, and have several advantages over other digital systems.
The first is that your recordings will be on a permanent (although erasable) physical medium, rather than just stored as files on some device somewhere that could be misplaced or unintentionally erased. Also, you don’t have to worry about filling up your recording space and then having to copy to your computer before continuing. Just slip in another disc.
The latest edition of MD recorder, using the HD format, allows you to record more than 24 hours of high-quality stereo digital audio on a disc that costs about $1.50. You can connect the unit to your computer with a USB cable and transfer the recordings to your computer, then burn them as CDs for distribution, if desired. (Note that you must get an HD MD recorder to have this capability. The old standard HD recorders can’t do it.) You can also transfer files of any kind to he minidisc, and use it for storage, if you want.
Learn all you need to know about MDs at http://www.minidisco.com. They have a good selection of recorders and discs, but you might find better deals by shopping around a little. For recorders I can recommend the Sony MZ-NH700S, which Minidisco offers for only $150. I think I paid almost $100 more for it a year or two ago. If you get a different model, make sure it has a microphone input jack.
Also, I can’t exaggerate the importance of buying a good mike. Your recordings will only be as good as your mike, so don’t scrimp here. I recommend the selection available at Sound Professionals, and specifically the SP-SPSM-10. I’ve been using this mike for years, and it’s great. I’m a journalist and use this setup for recording interviews and conference sessions in addition to taking notes.
You can get that mike, the MZ-NH700S, and a box of discs for about $300. I haven’t checked the latest prices, but I think that’s in the ballpark of a medium-priced iPod plus the recording add-ons.
If you have any questions, feel free ask.