Generally not, in LOTR for example magic rules are absolutely NOT explained.
You can, generally get away with explaining and detailing magic but if you go too far, IMHO, you run the risk of turning it into science, something that, to me, makes it about as “magic” as an operating system manual.
They aren’t explained or they don’t exist?
This is kind of an iceberg thing. Fantasy writers that I know spend an insane amount of time putting magic rules into spreadsheets and talking about does it work this way or that way? And what if this condition? They go on about it forever. They build their little encyclopedia.
And they write a story that features this magic system while only referencing it in the narrative when necessary. So they create this solid foundation but only show what they need to show for the story.
And that’s my understanding of worldbuilding in general.
I’m more of a story first person. I find the story and then build the world around it. But that’s riskier. And I’ve screwed myself doing that. For example, now that I’m done editing my novel I’m trying to fix the continuity (time, geography, seasons, etc.) So I was like, “oh, I guess I can’t do this without a map.” So I’ve been trying to make a map to fit my story, and I think I can pull it off but I’ve had moments of dawning realization that I probably should have drawn the map first.
As you alluded to an hour ago… any sufficiently advanced technology is indistinguishable from magic. It’s sometimes just a plot device meant to facilitate some important part of the story, but whose inner workings and rules may or may not be critical.
One of my favorite parts about both the Wheel of Time and Dune was their exploration of women, power, and their struggles between their personal morality and wants vs the societal obligations bestowed upon them.
The Aes Sedai in WoT and the Bene Gesserit in Dune were both more or less “witches”. The origin of their powers were explained in the books, I’m sure, but it’s been so many decades I don’t remember exactly where they came from… but it doesn’t really matter. Having those powers allowed them to face off against powers greater than they would’ve otherwise been able to handle, whether it’s big dragons or political machinations, and also to test their personal desires against their obligations (like mother against tribe).
Oh, and giant worms and dragons are always just fun to ride 
Sometimes fantasy spends forever explaining the rules of its magic. Sometimes sci-fi spends forever explaining the rules of its technology — at times believable, at times just pure technobabble, as is usually the case with Trek. Both that’s not what gives them their staying power, outside of nerding out on game nights
Both can address the bigger, deeper, darker questions (or not), depending on the writer’s talent and whims. Magic and technology are both just tools to facilitate the explorations of those questions without necessarily over-focusing on the “engineering” or “channeling”. A Jedi is a space wizard; an Aes Sedai is a magical cyborg.
Then, separately, there are also books of both genres whose magic or technology is the central focus of that story, where they invent some internally-consistent set of magical or technological rules that the characters must then abide by, and that itself becomes the source of the intrigue. “Given these constraints, what would you do in a situation where _____”.
In something like The Giver, for example, the magic (of being the only person able to remember the before times) forces a contrivance where the rest the society lives in ignorance, and explores the moral questions therein. (I’d classify that as fantasy more than sci-fi, but the line is often blurry!).
Whereas in something like Arrival, “technology” allows for giant squids who talk in spacetime and makes linguistics a central, interesting twist in “how we get the world to work together”. Or in The Martian, “stuck alone on a different planet with limited access to technology and civilization” forces it to become a survival story.
But it doesn’t really matter that one is “magic” and the other is “technology”; you could largely switch those around and not lose much in the story, as long as you keep their contrivances. The Giver could’ve been powered by a giant USB drive. The Martian could’ve been a prisoner teleported to a dungeon island.
Science fiction and fantasy are the same, in that both turn a lens upon the present day through presenting scenarios and situations that are sufficiently removed from the present day to make things clear. They just do it through different scenarios and situations.
For example, many Star Trek episodes are social commentary on present day issues. But they use the 23rd century and alien races without genders to do so (to use a half-assed example). By seeing how the genderless aliens treat each other and how the Enterprise crew treats them, we can compare that to how we treat people of other or indeterminate gender in our own lives.
Fantasy does the same thing, but in different scenarios and situations. C.S. Lewis’ Narnia books illustrate a lot of Christian themes, but through the lens of Narnia, Aslan, and the Pevensie children, and Eustace. By not bringing it up as Christianity, people may be more receptive to the ideas and concepts.
(FWIW, I’m paraphrasing Dr. Jimmie Killingsworth from my “Science Fiction and Fantasy Literature” course back in 1995 or so).
Is it possible you have some form of aphantasia (difficulty forming mental imagery)?
I think sci-fi often has beautiful vistas too… Dune, Annihilation, Interstellar, etc.
And yes, I can imagine quite a lot of visual imagery when I read (books of any genre) — it’s part of their appeal, and why I’ve often preferred sci-fi, fantasy, narrative nonfiction, and adventure. They often take place in landscapes or set pieces as dramatic as their stories, whether it’s a cliff’s edge or a bustling dystopian city or a flagship (of the sea or space).
I think both types examine society in a new paradigm. Whether it be one in which a replicator can recombine molecules into dinner without significant time or effort, or in which a magic wand accomplishes the same task, it’s a change of effort or requirement or possibility. The primary difference in fantasy is that the scientific explanations are hand-waved, whereas most scifi media will at least try to account for the physics. Both also occasionally use species differences to drive opportunity or conflict.
What if Moms could have n extra arm installed? = SciFi
What if mating with XCVKians from the planet Trgicx made an extra arm pop out with your baby? = SciFi
What if you could pay a mage to make you a potion to give you an extra arm for the next 11 years? = fantasy.
Or more concisely:
What if you could eat a mushroom that gives you the ability to teleport? = Fantasy
What if you had a teleportation device? = SciFi
I have a theory that people with aphantasia are good at detailed florid prosaic writing. They make up for reduced mental imagery with great literary descriptive passages that achieve the same thing. Whereas with someone like me, I find I write better screenplays than I do stories, as the descriptions for that format can be minimal and functional.
If I’m right, I wonder if that also relates to why you like something more technical like Sci-Fi over something romantic like Fantasy. I mean, you can have prosaic writing for both, but perhaps there’s something more going on.
Would you count George R.R. Martin among writers like that? I loved the series (well, until the last few seasons) but hated the books… couldn’t stand his multi-paragraph descriptions of the food on a dinner table. After the first sentence, I kept thinking to myself, yes, yes, I got it, I can imagine it just fine… why do you have to overly describe such trivial things…
I hadn’t considered that (ability to form mental imagery) as a possible difference between writers (and readers). Before I learned about aphantasia (through some internet meme, I think), I didn’t even realize it was possible for people to not form mental imagery… I had mistakenly thought that’s how we read at all, by forming pictures of the words, that it was the default mode of human thought. I guess I was quite mistaken!
Certainly not true of me. I can not only picture things in my head, but figure out puzzles by laying them over a 3d grid and turning them around. But I love really in depth world building. I don’t just want to make up a world for myself. I want the author to convey the world they are picturing.
When I read the title, I immediately said
“That’s the beauty of it!
It doesn’t DO anything!”
But then I ruined it by saying “Well, okay, The Lord of the Rings does a lot… oh, and Narnia, and Star Wars, and Winnie the Pooh, and the Oz books…”
That’s almost where the Ben Aaronovitch Rivers of London urban fantasy detective series has ended up. But in that case it’s a feature rather than a bug, as the practitioners have a whole “Newtonian” structure to explain and constrain magic.
For the purposes of my theory it doesn’t matter.
What matters is that we, the readers, don’t know exactly how it works, magic should be misterious, otherwise is just technology with another name.
In the case of Tolkien and Middle-Earth: probably the latter.
I’m not at all certain that Tolkien ever set out (even if just in his head, though he seemed to make actual notes on everything) “magic can do this, it can’t do that.” If he ever set parameters around how and why magic worked in Middle-Earth, I’m unfamiliar with it.
Even when Gandalf mentions that he cannot do something, it’s that he’s not allowed to do it, because it would violate the restrictions which the Valar (the “gods,” fundamentally) placed on him when they sent him to Middle-Earth, not because he’s not capable of it.
Tolkien’s elves can “do magic,” but it’s typically not depicted as actively casting spells, like D&D spellcasters; “magic” is just a fundamental part of what elves are and what they can do. An important part of the story of LotR is that the elves are fading away from Middle-Earth, and as they do, the magic which they created is fading, as well.
Elves can make “magic items” (such as the cloaks and rope which Galadriel gave to the Fellowship), but this is mostly explained as being simply how elves are capable of crafting things: to non-elves, they are “magical,” but to elves, they’re just well-crafted items. The magical rings are, maybe, an exception to this, as they are a product of deep lore and study, but again, there’s no “rules”, per se, set out about how they work.
Possibly @Exapno_Mapcase can add some wisdom here. It’s near to, or overlapping with, his professional area.
One thing about Fantasy is that it is very very old. Possibly the oldest form of storytelling. So whatever itch tales of heroes, magic, monsters and quests scratches, it’s a primal itch.
Galadriel had the flashiest magic of the third age elves. Between her scrying pool and apparently throwing down the walls of Dol Guldur 3 days after the Ring was destroyed. While the armies of men were returning from the gates of Mordor. She also laid bare its pits. The only hint is she used song-like powers like Lúthien.
What if you could consume a genetically engineered quantum fungus which allows your the atoms that compose your cells to entangle with atoms in a distant location and then transport you there molecule by molecule? = Magitech
It’s the modern incarnation of tales told round the campfire, I guess?
We as verbal creatures have been creating mythology since long before the written word.
And in some ways it’s a mental escape from the boring predictability of the workday world, perhaps.
IMO two things (caveat I have read a lot of sci-fi, almost no fantasy beyond Tolkien)…
First is it invokes myth. It builds a world around myth and sets a story in it. Of course sci-fi does that too, especially space opera.Though fantasy’s myth is a based on real myth. Tolkien is an imagining of Anglo Saxon myth (that Tolkien believed had been lost in the Norman conquest)
Second it does the same thing as the best sci-fi IMO, namely take a ridiculous premise (e.g. what if dragons and magic were real) and work through the interesting repercussions of that in real world people and societies, in a realistic manner. The early seasons of GoT are a good example of this (I’ve not read the books). The way people and society are portrayed GoT is the most realistic portrayal of medieval life ever shown on screen (speaking as neither a time travel medieval peasant, nor medievalist). It answers that “what would medieval society be like if dragons and magic were real” question brilliantly.
The way it was explained to me was that it’s a way for the author to take the reader out of the present day and present scenarios and challenges that are either hard to conceive of in the present day, or that won’t be well-received if explored by present-day readers in a present-day setting.
Hence the use of aliens as stand-ins for various underprivileged or persecuted groups in Star Trek (although they tend toward the unsubtle and ham-fisted in their writing). Or of what could happen with widespread surveillance (1984).
Of course, it doesn’t have to be so loaded; it can just be a way for a writer to bring the reader along on a fun exploration of a imagined world and how it differs from ours. Lots of fantasy seems to fall into this category- stuff like The Witcher and ASOIAF fall into this category IMO; there’s no larger message, just exploration of a foreign setting and how characters behave within it.
There’s not much difference IMO, between alternative history and fiction for what that’s worth All this stuff is speculative fiction and therefore is presenting an alternate world to the reader, whether it’s the world of Narnia, the Imperium of Man, or the present-day world of The Man in The High Castle.