The extreme optimism about the prospects of manned spaceflight in the 60s was a product of the times, and turned out to be unwarrented.
As far back as the late 1930s, the handful of people seriously interested in rocketry calculated the cost and performance of chemically-fueled rockets, and concluded that they would almost certainly be impractical; given that you would need a huge rocket with only about 5% of it’s mass as payload. Most assumed that manned spaceflight would not come about until some form of controlled nuclear power made it possible to build high-performance rockets. This was the presumption by most scientists and science fiction writers into the mid-50s.
But by the late 50s, a number of innovations combined to make chemical rockets suddenly look more promising. The first was nuclear weapons. Although many were intially skeptical, it was an obvious idea to put nuclear warheads on some sort of super-V2. This stimulated interest in developing rockets, and fifteen years after Hiroshima, ICBMs were a reality.
Secondly, two important revolutions in electronics took place. One was solid-state electronics, which made possible first the transistor and then the integrated circuit. The second was the maser, essentially a fantasticly sensitive microwave receiver. Together, these made it possible to build an automated probe that could send back photos from Neptune. Almost all science fiction had presumed that we’d have to send a spaceman with a camera to snap photos and bring the film back.
In short, by 1960 we’d developed high-value payloads that made chemical rockets worth the cost. Still, there would never have been an Apollo program if the Soviet Union hadn’t shocked the hell out of the US government by being the first to launch a satellite into orbit. In the context of the Cold War, it was presumed that manned spaceflight would have immediate vital national security applications, and the Apollo program was instituted primarily to ensure that the US had a massive spacelaunch capacity.
By the end of the 60s however, the original impetus had petered out. Neither the US nor the USSR found it expedient to base nukes in space, and it became apparent that there was little a man could do that couldn’t be done almost as well and a lot cheaper by remote-controlled satellites. Treaties banned basing nukes in orbit or nationalizing other planets, and the national security justification for manned spaceflight waned.
In the 1970s, the USSR took the next logical step in spaceflight, a series of small-scale spacelabs culminating in their Mir complex. IF the Soviet Union hadn’t collapsed in 1990, it would probably still be in orbit today, but now Russia is simply too poor to continue a first-rate space program. The US manned space program languished after 1973, using up leftover Apollo hardware while waiting for the Shuttle to become operational.
The Shuttle, alas was the Edsel of launch vehicles. A compromise design from the start, it was neither as cheap nor as reliable as was originally hoped. After the Challenger explosion, the original ambitious goals for the Shuttle were drasticly scaled back.
Today, there are only two really profitable venues in space: (1) Communication satellites, which ironically are now facing stiff competition from earth-based communications networks; the ambitious Iridium and Teledesic programs have been cancelled. And (2)Earth-imaging, which was deliberately hampered for years by the US government not wanting high-resolution photos available on the open market until foreign satellites forced the issue. Space tourism might be the next venue. So far though, other prosposals such as zero-g materials processing or powersats have not panned out. In short, there isn’t any “killer ap” which would spark a gold rush into space.
It is absurd that we’re still paying thousands of dollars a pound for payload into orbit, when much cheaper launch systems such as “big dumb boosters” could be built. But spaceflight is currently the domain of the entrenched interests - NASA, the Pentagon, the defense contractors- who have no real interest in changing how things are done. Some conspiracy-theory buffs even claim the government is deliberately squelching all attempts to commercialize space. One concern is that the same technology used to launch satellites is applicable to military missiles.