A diamond sphere, with a 2 m crew sphere on the inside and 2 m walls, would be able to survive a pressure of around 600,000 atm, or 9 Mpsi. In that case, the thick-walled pressure vessel equations hold, but since the crew space radius is fairly small compared to the total radius, most of the terms drop out and you end up with the compressive strength of the material. Which is about 60 GPa for diamond (though it could be higher if you’re careful with the crystal orientation).
The sphere would sink in water. But that’s ok; you can use the same trick as the vessels that went to the Mariana Trench did and add flotation devices. The Trieste used a container of gasoline. Whether that continues to work further down depends on the compressibility, but I suspect it would be ok; water tends to be more compressible than hydrocarbons. And there are other solutions (new vessels use syntactic foams, though I don’t know what their limit is).
Anyway, it looks like you can go as far down as you want in water before it turns to solid. You can go down further if you are diving in a sea of hydrocarbons, or even just liquid hydrogen. Diamond will still do well, but the buoyancy problem is trickier. I wouldn’t try it.
But that’s just normal material science. Dumb materials. Active structures, on the other hand, can have virtually unlimited strength. Consider a cylinder divided into rings. Each ring has a set of masses inside it, revolving around the perimeter. They’ll exert a centrifugal force against the outside. And they can be made to revolve as quickly as necessary. They still need to be braced against the outside, but what this means is that your wall thickness is limited, which means you can scale up the whole vessel as much as necessary.
A cylinder would need to have its endcaps braced. That could be done with a linear accelerator, where masses “bounce” back and forth between the caps, exchanging momentum to counter the crushing forces. Or, you could build the vessel like a torus, with rings along the major and minor radii.
Sure, there are some engineering challenges here, but you asked about theoretical limits…