There isn’t a lot of middle-kingdom stuff. Much of the spectacular stuff we see isn’t so much New Kingdom as Ptolemaic rebuilds and embellishments.
The pyramids were sheathed in smooth limestone, but to be fair, yes a lot of the wall reliefs, especially indoors, were shaped into wet plaster rather than carved in stone.
Note that Akhenaten’s Brazilia clone capital city was dismantled, along with much else he had built, after he died, by the priests of the traditional religion and their minions. During the interregnums between high points in Egyptian society, priests would have fled the temples and the sand might have covered them. At Karnak in Luxor, there is still remains of a mud ramp leading up to the top of one of the major “pylons” (front walls) where the blocks were hauled up, and the wall is somewhat unfinished. (For a bonus, beside that two of the columns are still rough blocks, so you can see how they made nice neat columns - they piled and mortared the squared blocks and then carved the final round shape.
Google David Roberts paintings for what things looked like 200-plus years ago. Sand covered everything. Some of the most spectacular sites still have paint on the columns and reliefs, because the sand covered them up.
Consider that just like the Roman emperors, the Greek tyrants, and just about everyone else even to present day - the prestige lay in building something new, not in routine maintenance. With scarce money, rulers were inclined to either severely rebuild or build from scratch. the priests maybe stayed as long as they got offerings and support from the local community. If for some reason a site was abandoned, likely it stayed that way unless a new monarch decided to turn it into his spectacular new temple.
After all, when the statues at the temple to Ramses II fall over or are pulled down by marauding armies - 200 years later, why would some other pharaoh spend his hard-earned taxation money or restoring and putting back together Ramses’ statue instead of making one of his own?
It’s funny too, you see some of the external blocks with the reliefs and hieroglyphics carved very deeply. Apparently, this was to discourage “recycling” of old building material - what good is a stone block if you have to spend all that effort to shave 6 inches or more off it to get a smooth face for your carvings?