Writers not producing books after an advance are more common than people think, but they are almost always handled quietly so nobody outside the industry ever hears of them. I can think of four categories the resolutions fall into:
The writer returns the money voluntarily. Not common, but not unheard of either.
The writer substitutes a different book for the one contracted. Often this will be a collection of some sort, anything enough to bulk out the covers and call a book. Usually the publisher accepts it as a loss to retain good will, but sometimes it’s the end of the relationship.
The writer defaults and the publisher eats the loss. Probably the most common. Used to happen more often in the old days, when publishers weren’t tiny parts of multinational corporations with profit and loss statements uber alles.
The publisher sues the writer to get the money back. Very rare, and not always successful. More likely to happen when they hand out big bucks to some celebrity who finds that putting the bad parts of a life in permanent print isn’t a good idea. A variant of this is a celebrity producing something so totally worthless - i.e. lacking the good parts - that it couldn’t possibly sell and the publisher won’t accept it.
Once the book is published, however, the advance is solid. Publishing is a crap shoot, in more ways than one. The vast, vast majority of books don’t earn back their advances. That’s why you see the blockbuster mentality that’s killing publishers and authors.
Softcover rights may be held by the publisher or sold to another publisher. In either case, the money brought in goes toward the advance. That’s true of every possible right, from audio books to magazine pre-prints to foreign sales to movie rights. Royalties do not begin to be paid out until the advance is fully covered.