It is the heat and pressure that the steel of the barrel can tolerate. Even though there is other fuel, the oxidizer in the powder is also working like a cutting torch on the steel of the barrel. The barrel life is fairly short at the top end of velocity spectrum. They erode at the throat, just ahead of the chamber where the heat and pressure is the highest. The steel is very thick there, so not a safety issue, but it kills accuracy.
If you don’t care about barrel life, then the combustion rate of the powder becomes the limit. This is where gas guns come in. They use pressurized gas, usually hydrogen to save mass, instead of powder. One of those puppie would probably get you to escape velocity.
The sabot helps, because the pressure gets to work on a light bullet with a large piston area.