This is what undid the elite class in ancient Sparta*. There was no path for upward mobility, only down. There wasn’t even any way for disgraced Spartiates to redeem themselves; so the elite class shrunk and shrunk until there weren’t enough to maintain the system anymore.
*or for that matter, the Purebloods in the Harry Potter books.