It’s always supply and demand, and the fears of future disruptions and/or gains to the supply plus expectations of future increases or lack thereof in the demand.
After OPEC enforced a rise in oil prices in the 1970s, oil prices hit what is still the all-time adjusted high in, I believe, 1980. This caused for the first time improved conservation measures in the U.S., including a mandated increase in CAFE, the average gas mileage in U.S. car production. The U.S. was still driving the worldwide demand for oil in those days, so any measures we took had a disproportionate effect on demand. Companies also tried to increase supplies by new drilling and new resources.
With lowered demand and increased supply, oil prices eventually dropped.
The reverse conditions hold today. U.S. demand has steadily risen. The growth of low gas mileage SUVs - which don’t count in CAFE ratings because they are treated as trucks - along with the growth of light trucks themselves mean that the vehicle fleet requires far more gas than future expectations once thought, especially since small and efficient cars sell even more poorly than anticipated.
Refining capacity has not increased because the low oil prices of the era you’re referring to meant that oil companies then did not have the huge profits of today to sink into expansion.
And China, along with India, have changed the worldwide demand equation. Japan was once supposed to do this, but the implosion of the Japanese economy - which took down much of the rest of Southeast Asia with it - meant that the demand increase never materialized. This fooled the shortsighted and the extremely large and quick Chinese ramp-up caught them by surprise.
Today oil demand is running as very nearly oil supply, as least for reasonable expectations of what can be pumped and refined. That means that oil futures - which are the price per barrel numbers that are featured on the evening news - are driven by fears of any possible disruption. Many of the major sources of oil - Iran, Iraq, Nigeria, Venezuela - are iffy in near future terms. Almost anything could disrupt supply from them. Katrina put a hurt on domestic refinery and pipeline infrastructure that we are still recovering from and forecasts indicate an even heavier hurricane season this year. SUV sales are down sharply in the first quarter, true, but the vehicle fleet is still biased heavily toward gas guzzlers and will require years to change in any appreciable fashion. Commuting times continue to increase nationwide, causing more gas consumption and stop and go traffic, the worst kind. Expects can probably name a hundred more problems with supply and demand.
What can or should be done about this is a topic for GD, not GQ, and I’m sure there are a dozen threads already there. But supply and demand really is the answer to your question. And lack of foresight, lack of political will to make unpopular decisions, and unwillingness to make sacrifices (albeit for an uncertain future payback) on the part of the public explains why supply and demand have not changed sufficiently to alleviate the current situation.