There was a trend for illustrating a lot of sf/fantasy from the late 1970s into the 1990s. It seems to have died away. But I note that occasional special releases still get illustrations, especially when the text is a bit sparse.
Recent releases of new Jules Verne translations, I note, are illustrated, usually with illustrations originally used in the period.
The idea is very popular in Japan where the format is called a “light novel”. I understand most of them are aimed at teenagers so not quite “kids” but still not fully for adults. Quite a few popular anime these days are based on light novel series such as The Melancholy of Haruhi Suzumiya, Toradora, Shakugan no Shana and Bakemonogatari.
Some are currently available in English translation. The Haruhi books and another series called Spice and Wolf are published by Yen Press and I think there are a few others.
Yes, anything by Stewart & Riddel is worth getting just for the illustrations.
DiTerlizzi’s Spiderwick Chronicles, likewise.
In the 19th Century it was common for adult novels to be illustrated. Dickens, for example, and Victor Hugo. The emblem of the musical “Les Miserables” comes from a picture of Cosette from the novel.
Maybe one reason is that, if modern readers want to see the characters and situations depicted in a novel, they wait for the movie to come out. Pre-twentieth century readers didn’t have that option.
But I’m sure the main reason is economics. Why would the publisher pay to include illustrations in a book, unless they believe that more people would buy it, or people would be willing to pay more for it, than without the illustrations?
Carter Beats the Devil by Glen David Gold is a recent well regarded novel with some illustrations. It’s a historical fiction and there are some great old posters in between some of the chapters along with an illustrated clue.