It was a victim of the Reagan-era “War on Drugs.”
Crime rose steadily through much of that period, and one accepted explanation for crime was the need for drug abusers to finance their habits. Then cocaine (originally treated as a rather benign drug) picked up a lot of press regarding its nastier side effects and addictiveness. Shortly after that, crack was developed as a way to deliver cocaine more easily than as a powder and there were, originally, lurid tales that crack could cause total and irretrievable addiction with only one or two hits.
The rush was on in the legislatures and Congress to save America from drugs. When the basketball player Len Bias died of a cocaine and alcohol while celebrating a March Madness victory, the impetus was provided to simply pass laws regardless of reason or logic. (For example, Bias died of powdered cocaine, but his death was used to justify making the penalties for crack ten times as heavy as those for powder.) The propaganda and advertising that accompanied that fever simply overwhelmed any other voices, creating an impression that has continued to this day. (E.g., the fears of crack addiction potency were disproved in less than a year from the passing of the law, but no one in Congress has had the guts to get the laws evened out in the last 16 years.)
Marijuana simply got caught up in the movement.
There are various movements out there to change things, from the marijuana-specific advocacy of NORML to the more general Libertarian approach that drug laws are, in general, unnecessarily intrusive. However, the press and propaganda was so thorough through the late 1980s and early 1990s that no such efforts have made any headway.
Additionally, crime rates did begin to fall shortly after the draconian drug laws were passed. While many people have pointed out that the number of males aged 15 - 35 (the segment of the population most inclined to commit crimes of property) have fallen throughout that same period and others have noted that the economy was on a steady upward cycle throughout that same period (and crime tends to rise and fall with the fall and rise of the econmy), opponents of drug legalization can point to imprisoned drug users and falling crime rates more easily than drug decriminalization advocates can point to correspondences in the economy or the age of potential criminals. It “feels” right to many people that we have locked up more people and we have less crime. And, of course, decriminalization advocates have rarely been able to put together coherent and simply presented rebuttals to those claims–either because the material is too complex for sound bite news segents, or because the anti-drug hardliners were correct.
In this environment, separating attitudes toward marijuana from attitudes toward cocaine, heroin, or numerous other drugs is simply harder to advocate.