The entire challenge rating system is completely ineffective, a lot of things are seriously under or over CR’d. Plus, CR is meaningless when considering the different classes and how vastly different in power they can be. The Druid, Cleric, Wizard, and Psion group can probably stomp encounters five or six CR higher than them, if not more, and do it reliably. The Fighter, Monk, Paladin, and Rogue, on the other hand, will have a hard time even with even-CR monsters.
Consider monsters of varying CRs that are either far too powerful for their level or far weaker. The Mind Flayer, a CR 8 monster, can probably obliterate an 8th level party unless the group has a hard counter to his once per round conical stun attack. If the battle starts out pretty even rather than with the party having some massive advantage like catching the illithid by surprise, they are unlikely to win or even survive if the mind flayer is played competently.
A Tyrannosaurus is also CR 8, and yet can be defeated by a 3rd level wizard in a single spell, with minimal chance of failure, and can be defeated by a 5th level wizard with zero chance of failure. These wizards may be able to take a mind flayer if they were extremely well optimized, but defeating the tyrannosaurus requires nothing more than using simple spells in a straightforward manner.
These are quick and easy examples that came to mind, but the monster manuals are absolutely full of things that wildly vary in power but have the same CR. I don’t know if a CR system could work well if it were done better; it may be too complex a question to boil down to a challenge rating. But at the very least it can be said the designers didn’t do a very good job at assigning CR to their monsters.
Well, if you want an explanation of my reasons for it, one of the main factors is that I like the world my characters live in to remain consistent. That means that the rules are, to them, equivalent to the laws of physics in the real world. A DM that arbitrarily alters the rules means that my characters’ world is no longer consistent, and that harms my suspension of disbelief. It becomes a world where the laws of physics (and magic) arbitrarily change in order to add drama and excitement to your life.
Additionally, I put effort and time into making my characters actually match up to their concept, mechanically. Whatever my concept is, I go to great effort to make sure the mechanics of my character will support it, so I’m not making unfounded claims of what my character is capable of (of course, the character may boast, but that’s part of the concept too). The DM changing things on the fly makes me feel as though my effort in doing this was meaningless, because it means my character is not as capable as I have made her; she is as capable as the DM decides she should be, since if my success is deemed to be too great, it will be made to not work for the sake of making the plot work the way the DM wants.
As for improvisational DMing, it’s a very touchy issue. I know it is impossible for a DM to prepare everything beforehand. If I veer off into an unexpected course, the DM must improvise. But I think this can be done while maintaining consistency and adhering to the rules. A DM can have a selection of premade statblocks prepared to assign to NPCs they need to improvise into existence. A DM should decide beforehand what resources each significant entity (organization, boss, etc) in their campaign has at their disposal; even if they haven’t decided how, precisely, those resources are allocated, I think this works much better than the DM suddenly giving a character or organization resources specifically to counter a clever plan I came up with.
More than anything, though, I want to be able to trust that anything that occurs in the game - especially things that affect my character, or that my character is trying to affect - follow rules that are consistent, that my character and I can understand, at least theoretically, even if we never in practice get around to finding that information. Such consistency goes further than almost anything else in establishing a world for my characters that I can suspend disbelief in and enjoy playing in, confident that my success and my failure are both in my own hands, that the arbiter of this world, my DM, is fair and impartial, not trying to mold my actions into his story. The story, whatever it turns out to be, should emerge on its own from fair results.