This is going to be a little confusing without sketches, though here is a site that shows and discusses the issue WRT to the SRBs. (I don’t agree with his ultimate solution–IMHO, o-rings just aren’t a correct design for this application–but his basic details are correct.)
O-rings are intended to seal around a shaft, or in an inset channel like an O-ring Face Seal (ORFS) joint against a flat mating surface. In these applications, the pressure of the hydraulic fluid or gas trying to escape deforms the o-ring, stretching the normally circular or oval cross-section into a kind of a pointy “rugby ball” shape. (Some o-rings are epitrochoid in section but I haven’t seen any evidence that they’re actually superior.) This is what makes it an effective seal; the more pressure, the more the ring distorts, and the stronger its reaction up until it fails or degrades. However, in order to make an effective seal an o-ring must be compressed within a certain range; too little, and you don’t get enough axial pressure to make a good seal. Too much, and it can’t deform enough to stop leakage.
When you put an o-ring in a joint that sees bending across its axis (normal to the axis of the o-ring), then one side of the ring tends to be compressed less than the other. If that difference is enough (as it was with the Shuttle, owing to joint surface misalignment and bending actions during firing) then gas can leak past at that point, which not only allows the gas to escape but also causes the o-ring material to degrade (wear) if the gas is, as was the case with the SRB, hot and high pressure. Similarly, a torsional motion (about the joint axis) will distort and stretch the o-ring so that it has less “give” about its section, sort of like how your sock is much tighter if it gets twisted when you put it on. Again, this can lead to a lack of sealing and ultimate failure.
The problem with o-rings is that, while they’re self-reinforcing within their design parameters, once they get in an out-of-design condition they tend to fail rapidly and often dramatically. I recall in a former job having some valve blocks with an ORFS groove that was machined unevenly; repeatedly, when the hydraulic systems were subjected to max operating pressure (3000 psi) the joints would spray hydraulic fluid in increasing quantities until the o-ring parted and fluid just flowed out. And this was a non-catastrophic (if critical) joint.
There are a number of design solutions for an SRB-type motor that do not use o-rings; all, however, have the downside of being significantly more costly, somewhat heavier and more intenstive in fabrication and assembly time, and would require a nearly complete redesign of the SRB casing, including (probably) the casting molds for the propellent, which is a cost that nobody wanted to bear. (This was, in fact, the Morton Thiolkol argument against making any change to the joint design prior to Challenger’s demise.)
That being said, any “blank sheet” motor design is going to have its own unforseen problems and development setbacks. The joint redesign, while not ultimately remediating the root problem (IMHO), has reinforced and reconfigured the joint sufficiently that o-ring failure and problems stemming from that are significantly less risky. Given the age of the system and its short remaining deployment lifespan, it’s kind of like complaning about rattles in a Yugo; sure, the thing shakes and shudders at 56mph, but on the bright side, you’re only going to be able to drive if for two years.
Stranger