This has been a favorite rant of mine for years. First off, neither the instructor nor the university makes one cent from textbooks, so forget that argument. The authors do well, when the book is widely adopted but it is the publisher that makes out like a bandit. And makes sure there is new edition every year or so so used sales won’t cut into the market.
The basic reason seems to be that the people who choose the text typically pay no attention to the price. (Similar to prescription drugs?) Once I was about to teach a course for which I could have just as well skipped having a text, but a text is a good source of problems and the students feel uncomfortable with no text. So I wrote to five different publishers, asking what the price was. Nowadays, I would just google it, but this was probably about 20 years ago. Two of the publishers answered with the price (and I chose the cheaper of the two), two didn’t bother answering and the fifth said that that information was proprietary(!!). WTF, the price is a trade secret?
Calculus books took a sharp climb in price when they introduced four-color printing. But is there any evidence that that makes a better calculus book. Calculus hasn’t changed in 200 years (almost no book uses Robinson infinitesimals, although that would be a different book). I see no reason the book I used 55 years ago (by someone named Bliss) that I bought second hand for $2.50 wouldn’t be perfectly usable today. I think there are also online books that are free.
About 10 years ago, I had a book reprinted. It was graduate text level and while 1800 and then 800 copies had sold over about 7 years, the original publisher declined to reprint it a third time. So I found a local university press that was willing. We did 800 copies. The local printer they used charged a one-time setup fee of $3200 and then about $5 a copy for printing, so that the entire 800 copies cost $7200. That included printing (from camera-ready copy I provided), binding (in sewn signatures) with a plastic cover that they produced. It was sold for $45 a copy, postpaid anywhere in the world. At the time, that was about $33 US. It is still available.
A few years ago, I met a woman who told me that her father was writing a calculus text. I exclaimed without thinking, “Just what the world needs, another calculus book.” She was offended and explained that her fathers book was going to be different from all those others. I bit my tongue and said nothing. But I know what would happen. He would produce this text thinking that this one would be different from all those others that are essentially clones of each other. He would send it to a publisher, who would send it to 25 (this is not an exaggeration) referees, each of whom would insist on changes. All those changes would be in the direction of making just like all those others. Either the man would make the changes, in which case my comment was justified, or he would refuse in which case it would not get published. He could try self-publishing, lot’s of luck.
Textbook publishing is like a lottery. A few books have gigantic sales and make enormous profits and the rest languish. If someone would pay me enough, I would write a calculus book that could be circulated by the math societies for free online or for print-on-demand. It is an easy task; just take any one of the many books and copy it (in your own words). They all copy each other shamelessly anyway and there is no way they could sue with clean hands. And I think I could write better than most of them.