The download here will do it for you. It sounds techical but all you have to do is unzip the file and double-click on the registry modification that you want to implement for your caps lock key.
In Control Panel’s “Accessibility Options” (it’s called that in XP, maybe something similar in Vista), there’s an option called “ToggleKeys” which makes a beep if you toggle the caps lock or num lock. I have mine set to that. At least if I accidentally hit the caps lock button, I know it immediately.