Why do commercial jets cruise at 35,000-40,000 feet?

The silver lining of the 9/11 attacks was that the grounding of commercial air traffic over the US for several days provided an opportunity to study the climate effects of contrails:

First, why go so high: Jet planes are more efficient at high altitude because the engines themselves are more efficient.

Just like a piston engine is more efficient at higher compression ratios and temps, so is a jet engine. Why not just run the jet engine at slower rpm and temps?

(1) Unlike a piston engine, the pressure ratio of a jet engine varies greatly with rpm. It produces far too much thrust at lower altitudes for level flight so rpms and pressure ratio must be reduced. This reduces the efficiency, resulting in higher fuel consumption for given thrust.

(2) Unlike a car, a jet engine doesn’t have a gearbox, and unlike a piston or turboprop aircraft, the turbine blades are fixed position. That single fixed position, number and arrangement of turbine blades must be designed to operate most efficiently at cruise rpm, altitude and speed – where it spends most of its time.

Some jet engines have variable stators or movable non-rotating vanes to try and regain some efficiency across a broader rpm range, but no production jet engine has variable pitch compressor or fan blades.

If efficiency increases with altitude, why not just keep going higher? Because to gain propulsive efficiency almost all civil jetliners use turbofan engines which lose more thrust at higher altitudes than a pure turbojet. This caps the achievable altitudes.

They could use pure turbojets and go higher (like Concorde) but those are less efficient than turbofans. This is because for a given thrust level, it’s less efficient to accelerate a lower volume of air to high speed vs accelerating a high volume of air to a lower speed (like a turbofan or turboprop engine). So they would lose more overall efficiency using turbojets to go to 65,000 feet than they’d gain from the higher altitude.