Your “word on the street” is wrong.
When people say “radiation” they are generally referring to specific types of electromagnetic radiation. Electromagnetic radiation in general is a whole bunch of stuff. At the low frequency range you have long waves (useful for communication through polar ice but not much else), radio waves, microwaves (which are just higher frequency radio waves), infra-red light, visible light, ultraviolet light, x-rays, gamma rays, and cosmic rays. The lower frequencies are basically harmless. People generally don’t freak out if someone shines a flashlight on you, for example. People freak out over microwaves from cell phones and such, but they really don’t have any reason to. Radio waves aren’t any more harmful than visible light. Get a high enough power of radio waves and you’ll cook things (that’s how your microwave oven works), but if you get enough visible light on something you can cook it too, as kids with magnifying glasses have often demonstrated by frying ants.
At higher frequencies though, things change. Part way through the ultraviolet part of the spectrum, the electromagnetic radiation becomes “ionizing”, meaning that it can strip the electrons off of atoms and create ions. When people say “radiation” they generally mean ionizing radiation (x-rays, gamma rays…). This is the stuff that’s really bad for you, because it causes cell damage and cancers. Your body really doesn’t like having its molecules ripped apart and turned into ions.
Ionizing radiation is a lot like the lower frequency radiation though. When you turn off the flashlight, the visible light is gone. As soon as you remove the x-ray source, the radiation from it is gone too. Your “word on the street” seems to think that the “radiation” is some sort of solid, tangible thing that builds up in your body. It isn’t.
Your body can repair a certain amount of damage from radiation. It has to do it every day. Every day you walk out into sunlight, and the sunlight smashes into your skin and wreaks havoc on some of your molecules. Too much radiation though is going to kill your cells or cause lots of damage to the point where you’ll end up with some sort of cancer.
Obviously, if you get a whopping big dose of radiation all at once that can be very bad for you. Long term chronic exposure can be bad too. If you happen to be a radiation worker of some sort, national regulations don’t allow you to be exposed to more than 5,000 mrem per year. The safety regulations for us common folk are much lower, only 100 mrem per year. If you get x-rays and such they are going to total up your dosage, and try to keep it under 100 mrem per year. A typical x-ray is going to be about 10 or 20 mrem, so you don’t really want to go having bunches and bunches of x-rays done every year. But this stuff doesn’t accumulate in your body anywhere. Just because you had 5 x-rays last year doesn’t mean you can’t have any this year.
To put things in perspective, most of us get exposed to about 300 mrem per year from natural sources (sunlight, radon, the background radiation of the entire universe, etc).