I’ve always seen them cap out at the price of the book. At that point, you might as well not return the book when you pay the fine, as they are likely going to buy a new book anyways.
At least, that was the policy at the library I worked at. Oddly enough, they had another policy of never actually billing people if they didn’t return the book. I always thought that was weird.
Here’s a obscure piece of library fines trivia: Prisoners in New York State prisons cannot be charged for overdue books from prison libraries. For some unknown reason, the department’s director of prison library operations decided to enact a policy that prisoners can only be fined for damaging a book not for borrowing it and failing to return it.