Of course it wouldn’t work. Digestion is an inefficient process. In this closed system, the cats and rats would have to be able to assimilate 100% of the biological energy of their food. If they didn’t the animals would either need to be supplemented with additional food from outside the system, or they’d eventually starve or dwindle in population down to nothing.