I don’t really know other than people, as a whole, don’t have a great sense of time and as the clapping goes on, there’s some sort of error propagation that occurs that drifts people off.
I don’t know why that happens, but I did want to share this fun video of Harry Connick Jr throwing in an extra beat to flip a crowd’s clapping on the 1 and 3 to the 2 and 4. The magic happens around 40 seconds. Note the drummer in the background at 0:44 raising his hands in victory.
That crowd actually does a reasonable job of keeping the beat, they were just on the wrong one, and when Harry turned them around, they just kept going, not even noticing they were on the 2 & 4 now.