(N.B. Not a physician. I’m a professional singer who has studied vocal health in depth. Take this for what it is worth)
The OP is a particularly difficult question to answer – impossible really – because many different elements contribute to the sound of a voice, both in speech and singing. Also, everyone’s voice has its tolerances for abuse and stress, and everyone is genetically predisposed to age vocally at their own rate (some people maintain healthy, flexible voices well into their senior years, while others may age vocally while still in their forties and fifties – the cartilages of the larynx gradually ossify as we age.)
Some people are just born with “gravelly” voices; maybe the best example is Julie Kavner (voice of Marge Simpson), who supposedly has vocal folds of unequal length – freaky! Some people think the same was true of Louis Armstrong. Many people are just born with perfectly healthy voices that have a husky quality to them.
Prolonged abuse of the voice is the biggest cause of the sound you’re describing, however. It isn’t so much as you suggested in the OP – that the gravelly singing results in vocal damage (though that can be true) – rather, the gravelly sound often results from misuse. Any use of the voice that habitually exceeds the laryngeal muscles’ comfortable stamina, or which frequently exceeds vocal tolerances of pitch, volume, and pressure can eventually cause irrevocable damage and change the sound of the voice.
Smoking and drug use can have the same effects over time. Smoking especially – think of the “Weezy” Jefferson voice, for instance. Ray Charles sounded like a gravelly old man in his early twenties, and some attribute this to his extensive drug habits as a young man.
Potential problems include chronic tension in the vocal mechanism (which is very complex, involving dozens of musces both within the larynx and without), calouses on the vocal folds themselves (usually called nodules, because they create nodal points in the vibrating edge, thus changing the sound emitted), and chronic swelling/stiffening of laryngeal tissues.
The most common victims are pop/rock singers, who often sing well outside their natural limits for an entire concert, and who often do this for years on end in the process of touring, etc. This is further exacerbated by the loud volume of rock concerts – even though they are being amplified heavily, most rock performers still sing loudly enough to hear themselves over the noise. If you’ve ever gone to a noisy party and come home hoarse afterwards, you know the effect; what seems like normal usage is actually very loud.
If the resulting damage does not impair vocal function too much, the change in the sound can actually become a singer’s trademark. Rod Steward is a good example, as is Bing Crosby. Crosby didn’t have the sound you describe, but he actually had vocal nodules – “million dollar nodes,” as his doctor is fabled to have called them – that contributed to the unique color of his voice.
Opera singers often suffer similar damage over time, but the effects are usually more subtle. Instead of gaining an overtly rough timbre, their voices usually just lose their characteristic sheen and beauty and become less desirable as they age.
Another common factor in vocal roughness can be acid reflux disease (GERD). Many people have it. Some experience heartburn along with it, in which case they are likely to have it treated. But others have no obvious symptoms, and over time acid irritation (minor burning, mostly at night and after meals) can lead to chronic swelling and hoarseness.
Sorry to ramble, but I hope that at least gives you some idea of things to pursue on the subject.