The most compelling argument is that last minute schedule changes are disruptive and potentially dangerous for athletic training regimes. The best-in-the-world class of athlete competing in the Olympics is likely engaged in a multi-year program that is carefully and specifically designed to peak with their appearance in these particular games. If you introduce a one-year delay at this point (after already having delayed once), they have to rejigger their practice routine accordingly, which will impact their performance and possibly increase the risk of injury. Some will choose not to participate, or will be entirely unable, especially if they’re privately or sponsor funded and can’t manage yet another year of maintenance training. And so on.
The point is, these are not just casual games played by the duffers in the corporate softball club that you can easily shift from one weekend to another. Major schedule changes will have major consequences for athlete participation and performance.
The question, obviously, is whether the positives outweigh the negatives. I perceive an emphasis on apparent positives in the OP, and a dramatic understatement of the negatives, which are legion. It’s not a slam-dunk debate.