Ok, I think its time for a little more clarification from someone with professional background.
There are, in the realm of typical third-party software, two different sorts of registry changes that will be made as a result of installation. If the software controls a specific hardware device, there may be more for the device driver itself (this can differ depending on whether you run 9x/NT).
The first kind of change, is what has already been talked about, which are the addition of keys, subkeys and values that store configuration information for the specific application. This should usually be found under HKEY_CURRENT_USER\Software<Vendor Name><Product Name>
It can also be found under HKEY_LOCAL_MACHINE… - entries that are specific to a user profile (for example, most recent file access) typically go under HKEY_CURRENT_USER, and settings specific to the (you guessed it!) machine, go in HKEY_LOCAL_MACHINE. Pretty nifty.
These entries are absolutely harmless. The only way they could ever, ever cause you a problem is if you had different versions of the same program on the same machine.
So what is the problem? One problem, a very old problem, is DLL Hell. DLL hell is the problem where common libraries are stored in the system folder, where they can be efficiently shared. The problem is that sometimes new versions are introduced with a new program that are incompatible with existing software. Generally it is not an interface incompatability as much as it is a practical incompatibility - the new version exposes a bug in older software.
COM made this worse in many ways. Used to be, you could keep copy of all your DLLs in the directory with your program, and be sure that they would be safe. Well, COM classes have to be registered, in the registry. This is so a program can instantiate a COM object without having to know what DLL it is implemented in, or anything specific about the DLL. This is important for a lot of reasons of theoretical purity, but the practical result was registry hell, which is truly not any different than the old DLL hell. New installations, even though they might have put their COM dlls in their directory, when they registered them it forced every program on the computer to begin using that COM dll.
There are many documented problems of these types. One major one we are struggling with at work currently is that certain LDAP functions do not work with ADO 2.5 (a database API library used by most modern software that uses DB) installed unless the Microsoft Active Directory Client is also installed. Since we have programs that require ADO 2.5, we cannot install ADO 2.1 (which works fine). Our only option is to roll-out Windows 2000 or the MSADC (which is probably what we will do).
This is a gross oversimplification, and I don’t want to say that every problem you’ve ever experienced is due to COM hell. There is also plenty of device driver hell, but I’m not very experienced with that side of things so I’ll let others elaborate if they wish.
In conclusion: registry good for program settings. COM bad.