Tonight I was looking at the jumble of books on one of my bookshelves, and decided, for the fun of it, to sort them by size.
So, we have the typical height of a paperback, which does vary slightly (my old “Tron” novelization is a bit taller than my copy of Richard Bach’s “Illusions.”)
Then there are the trade paperbacks which also vary slightly in height (Cecil Fletcher’s travelogues being smaller than Heilbronner’s “Worldly Philsophers.”)
Then I have a long series of mainly hardbacks but also some paperbacks that slowly grow about a half-inch in height over the course of my bookshelf, ending in a few books that barely fit (“First 20 Years of Monty Python.”)
Books with lots of pictures, which deserve to be seen at high quality (“coffee table books”) would be big to accomodate the photos. Books which someone would take with them to read when they have spare time (novels) would be smaller for convenience. Books with a lot of content (dictionaries) would be ridiculously thick if they had covers the size of a paperback. So good so far. But as for minute variations that you described, I have no idea. Perhaps it was just never standardized, so the publishers just picked sizes randomly, then never decided to get together and make it consistent.
Former book editor here. There is a standard size for text-oriented, mass-market paperbacks that are intended to be displayed in wire racks. If the book is not intended to be displayed on a standard size wire rack, then there is no reason why there should not be all sorts of different sizes of books. The size and shape of a book is determined by two competing factors: the requirements of its illustrations, and the cost of the book. The author and editor want the book to be large enough, and shaped appropriately (i.e., wide or tall) to reproduce the illustrations properly. Think of art prints or maps — they must be large enough to retain their integrity. Yet the publisher wants to keep a reasonable cover price that won’t hinder sales, and attempts to limit the number of pages or the size of the paper. Between those competing factors, you get compromises of all shapes and sizes.
There is some standardization. Large printing houses tend to quote lower prices for sizes they are used to producing (and can thus do at lower cost). Those whose book’s size doesn’t matter much will thus tend to choose one of the common sizes.
But all tastes are to be found in nature, so book sizes vary for the same reasons that book titles do.
If all books were of the same dimensions, it would be really boring, and longer books would have to be printed in unbearably tiny font sizes in order to keep them from being thicker than say, Dr. Suess’ The Cat in the Hat.
