Well, it’s not necessarily true that overall drag decreases as you go higher. Parasitic drag goes down as the air gets thinner, but induced drag may increase because indicated airspeed goes down. In essence, the airplane thinks its going slower for the same groundspeed as it flies higher. If the indicated airspeed drops below best L/D, drag will start to increase. At the highest altitudes, some aircraft are flying only a few knots faster than their stalling speed! This is the infamous ‘coffin corner’.
The way to reduce induced drag is to have long, high aspect-ratio wings. But those are harder to to build, take up more space on the ground, weigh more, etc. So there’s a tradeoff involved. In general, airplanes that are designed to fly very high will have long, thin wings. Airplanes that fly low will have short, stubby wings.
So when you’re designing the aircraft, you’ll have to trade off service ceiling with aircraft weight, ease of manoevering on the ground, low-level drag, structural integrity, etc. When all that comes out of the wash, you’ll have a fixed optimum cruise altitude at which the airplane is most efficient.
But wait, there’s more! Airplanes don’t always fly at the same weight, so the optimum altitude will vary. The air is not still, so the optimum altitude will vary (winds generally get stronger as you go higher, so if you have a tail wind you’ll usually want to climb high and take advantage of it. Conversely, if have a headwind, you’ll often want to fly lower.
But wait, there’s more! The certification requirements change if you want to fly really high. Above certain altitudes which I can’t remember offhand, both crewmembers must be on oxygen at all times, and the consequences of an unintended decompression become more severe. At some point, a decompression becomes non-survivable even with oxygen masks, because the partial pressure of oxygen around you is so low that your lungs won’t function properly. So there’s a pretty hard limit at to how high a conventional airliner can fly.
But wait, there’s more! Engines are designed to be efficient at certain altitudes. Above or below that, efficiency drops off. In the case of propeller aircraft, long thin props are more efficient at high altitudes than short stubby ones. But long thin props are a bugger to deal with. You need longer landing gear for prop clearance, which adds weight and complexity which may overshadow prop efficiency gains. And that long, thin prop might not have as much efficiency at sea level as a fatter, stubbier one. So climb efficiency decreases.
Every time I right one of these paragraphs, I think of more factors that come into play. In the end, the aircraft designer has to juggle a million different configurations and come up with the one that represents the best collection of tradeoffs for the mission the aircraft is being designed for.
