My crass opinion is that length of books is correlated with money.
In the Depression era, books got very long; even classic mysteries were regularly over 300 pages and fat bestsellers like Anthony Adverse (500,000 words, later split into three paperbacks) and Gone with the Wind (400,000 words) dominated. In this pre-paperback era hardbacks were reprinted by low-cost publishers like Grosset & Dunlap, which retailed for 50 cents, but were pushed by chains like Woolworths for 40 cents or less, even a quarter if stock remained too long. That was many hours of enjoyment for a very low price.
War-time paper shortages forced the slimming of books. The mass-market and digest paperbacks that appeared in the late 1930s had a fixed price of 25 cents and all but took over the book market, but also suffered from the lack of paper and abridged books to make them fit the thinness, espeically the digest which had only 128 pages.
A faster-paced noir world kept paperbacks thin, especially original paperbacks, which appeared in the late 50s. These were only 40-60,000 words. Prices rose but were kept under 50 cents for the decade. Low-cost hardback book publishers died out but book clubs proliferated, giving out cheap reprints. I got nine James Bond novels for a dollar when joining the Detective Book Club in the 1960s.
Everything changed in the 80s when fat books became the norm again. Writers of thin, taut, fast-moving genre, like Ed McBain and his 87th Precinct series, began bloating their books to two or three times the length. That made them hardback bestsellers but the quality suffered greatly.
P.T. - Post-Tolkien - the fantasy market swelled and so did book lengths. So did sf space battle books, like those of David Weber. How they ground out more than one 1000-page book a year is incomprehensible to me, but length again sold and the market demanded longer and longer stayed in the imaginary worlds. The literary market saw the results and followed.
Ebooks did make a difference. Millions of the so-called books on Amazon are actually short stories, priced to the minimum Amazon will allow. That drives sales. Not quality, but sales. There’s a sweet spot in the middle, but like mid-level movies nobody goes to them anymore.
Sorry. Answering a question with “it’s always the money” is a cliche, but so very often is the best answer.