Why the middle ages more so than other periods of history? Why not ancient times, where there certainly were kings and emperors and all kinds of intrigues?
I think because the tropes of modern fantasy owe a lot to the matter of the medieval cycles - wily wizard, upright (if sometimes morally vulnerable) fighter, lonesome fierce monster (sometimes with treasure), etc. These tales have worked the same for centuries in the same setting and while it’s possible to transpose them to pre-medieval settings, why mess with a working formula?
There is a short scene in the animated movie Flight of Dragons where several wizards are discussing why magic in their world is growing weaker as the people begin to move more towards the utilisation of science and technology.
The Chinese wizard shrugs and says, “Well logic is so…logical…” 
This is basically what I was about to post. Harry was not particularly interested in magic theory or research, but a number of other characters are, and Snape in particular started working to improve existing and create new spells and potions while he was still a student at Hogwarts. It is also made clear in the books that learning to cast spells and make potions takes some effort and practice. It’s only after one has mastered a spell that the “point wand and say Latin-esque word” thing produces reliable results. Something like the “lumos” spell is apparently pretty easy for anyone with magical abilities to learn, but Harry has to practice was Professor Lupin for some time before he can manage a Patronus Charm.
As for technology, it’s specified in the books that electronic devices do not work at Hogwarts. I can’t remember if this is specific to Hogwarts or if it’s true of any location with high levels of background magic, but it’s significant to the plot of the fourth book. Harry speculates that journalist Rita Skeeter may have hidden microphones at Hogwarts, but Hermione explains that this is impossible. Pre-electronic technology, like the Hogwarts Express, seem to work fine. In the third movie there’s also a scene where they listen to music on a phonograph machine.
The groundwork was probably laid by the Victorians, who were into romanticizing the Middle Ages. There was also a lot of popular interest in fairy tales (e.g. the Andrew Lang Fairy Books) and tales of King Arthur (e.g. Tennyson’s Idylls of the King) at that time. Then came The King of Elfland’s Daughter (1924), arguably the first modern fantasy novel, which has a pseudo-medieval setting. So do The Hobbit and The Lord of the Rings, and a lot of post-Tolkien fantasy was heavily influenced by Tolkien. Once it became an established trope, sticking with a pseudo-medieval setting for a fantasy novel must have seemed like an obvious choice for a lot of writers. They may have loved reading books set in these kinds of worlds themselves, and this kind of setting is going to be a lot more familiar to a lot more people than a pseudo-Bronze Age setting.
That said, it seems to me that there is a fair amount of fantasy with settings based on Ancient Greece, Rome, Egypt, and/or Britain, although this is sometimes mashed together with pseudo-medieval elements.
Pretty much all sci-fi I’ve ever seen has magic (usually under a different name). Star Wars definitely has magic, it’s not even trying to hide it really. Star Trek has a number of magical things (empaths, telepaths, Q, etc). Babylon 5 had the psi-corps which is basically magic and a number of species that had magic powers, essentially.
So magic is definitely not always limited to the middle ages. In fact, I’m a little disappointed that so much of it shows up in science fiction all the time.
Exactly. Dumbledore is famous partly for his co-discovery of “the twelve uses of dragon’s blood”, for example. The potion for mitigating some of the most dangerous symptoms of lycanthropy was developed within Professor Lupin’s lifetime, that is to say, within the last thirty years or so. Broomstick technology appears to be constantly producing performance improvements.
It’s true that the “magical community” in the Potterverse mostly shows little interest in modifying the non-magical technology or science that’s part of their routine daily lives. But they certainly don’t have a static or incurious approach to magic itself.
Since a major theme in the books is anti-Muggle prejudice in wizard society, it seems likely that the apparent lack of interest in developing or adapting non-magical technology is due in part to mistrust of Muggles. It’s easy to imagine that people like the Malfoys would believe that nothing invented by Muggles could possibly be better than something invented by wizards, and perhaps also be concerned that non-magical technology developed by wizards would wind up falling into Muggle hands.
I have a pet theory named Fifi that says there was a major cultural split between magical and Muggle communities, at least in Britain, in the 15th through 17th centuries, beginning around the publication of the anti-witchcraft treatise Malleus Maleficarum. That’s when witches and wizards really committed to going “underground” and hiding all traces of magic from Muggles, and consequently it’s the point where wizards’ adoption of Muggle technology more or less stopped evolving.
At some point in the 19th century, probably catching some of the Victorians’ self-consciously progress-minded attitude, there was a cautious re-engagement with Muggle industrial development, resulting in adoption of things like steam trains, newspapers and photography. But of course, there was an inherent physical limitation built into this renewed technology transmission, since electrical and electronic devices don’t work in magical environments.
Currently, decline in anti-Muggle prejudice is making magical/Muggle communication a bit more open although still exclusively unofficial. But the almost total dependence of modern Muggle technology on electricity means that it’s almost completely useless to wizards.
It has a nice balance of primitive technologies and sophisticated culture, and it’s also the era when a lot of fairy tales, myths, and legends were set; from around the time of King Arthur (circa 600CE) through to the days of Robin Hood (circa 1500CE).
The author of that article ignored the fact that technology isn’t a linear progression. It’s a series of happy accidents, building on earlier tech, as well as being able to build on earlier-known ideas. The principles of the steam engine were known for over 3000 years before they were harnessed for energy. The Romans had concrete, windmill power, trade networks, accounting, codified law, and some level of intellectual property protection; all things that were key nearly 1000 years later to the growth of industry in Europe. Why didn’t that revolution happen earlier? Best guess of many many historians: hellifweknow.
Something that people forget about our real-world history is that guilds had a stranglehold on many technologies and specialized knowledge for centuries before the Enlightenment started to break down the walls of customary trade-secrecy. And even then it took a couple of hundred years to make information sharing somewhat normal. Look where we are now with patents — especially software patents — and copyright law. You could easily make the assertion that technological progress is being held back by less open information exchange than in the recent past.
Imagine if magic-users not only had a head start in magic over technology, but also had a vested interest in making sure they were the top dogs. How many centuries of progress did it take for firearms to be practical vs. bows? Multiply that by mages who might be able to do things like blow up your powder supply, turn un-enchanted metal to other substances, scry out your factories and summon dragons to destroy them, or simply call down a meteor storm.
I suspect that the person also hasn’t read a lot of fantasy, because many of the better fantasy authors actually have addressed the magic vs. technology question. They either explicitly state or strongly imply economic, political, technological reasons for the state of their societies. The Potterverse is probably the least rigorous treatment I’ve read. Some authors posit that they’re two sides of the same coin; some mutual interference; others that if the laws of physics permit magic, then technology doesn’t work as it does in our world, or vice versa; and there are probably more patterns that I can’t think of right now. Here are a few examples from series I’ve read. This is by no means an exhaustive list.
Song of Ice and Fire - George R.R. Martin
In SoIaF, with the Winter comes invasion by magical creatures. You don’t know how bad it is, but it’s implied to be very bad, otherwise why build a 700 foot(!) wall to keep shit out? It’s implied that only help from the gods kept humans from getting wiped out by the Others. While the introduction of magic in the series has been slow and subtle we’re starting to see that powerful forces outside of what we’d consider normal reality are actually at work in Westeros. It’s not just metaphor or technological ignorance.
Also, there’s historical precedence of widespread famine and internal unrest on top of the external warfare. Think about how much disruption the Black Plague and the Little Ice Age caused at different times in medieval Europe. The eruption of Tambora in 1815 caused crop failures and famine, flooding, epidemics. That was a drop of less than 1ºC in a single year that stabilized well within a decade. Imagine a 20 year winter with a global drop of 5ºC or so, and what the fallout would be from that.
It’s no wonder they’re stuck at iron-age to medieval social and technological development. Every Winter trade would be devastated and industry would crash. The population would probably drop by 50% from famine and epidemic alone (death rates doubled during the Year Without a Summer in 1816). External warfare and internal strife would kill even more people. Even our present level of technology would probably not survive those conditions, much less something in between.
Guardians of the Flame - Joel Rosenberg
(College students get sent to a D&D world as their characters. Way better than that one-sentence summary sounds. Highly recommended.)
Magic is both extremely powerful and dangerous. Not everyone can do it. You have to have the right genes (or whatever) to even be able to make sense of the magic. The ability is inborn, not acquirable. Part of the conceit is that the world is either engineered or chosen from a multiverse by the mage who sent them there. The natives of the world had explosive powder already, but the mages had a monopoly on it, and very actively discouraged any encroachment on that monopoly.
One of the people sent back was an engineering student. Along with the help of the others and the seed of their society of freed ex-slaves, he started a many-year project to create firearms. The problem was creating a industrial revolution in their lifetime, while under some opposition from the Mage’s Guild, and basically open warfare with the Slaver’s Guild.
While Home (the territory they carved out for themselves) relied more on technology, they did use magic too. If they hadn’t, they wouldn’t have survived past year one. Even so, the situation over two generations or so depicted in the books had stabilized at basically equilibrium with opposing countries, with an early industrial level of tech that probably wouldn’t get to present-day levels for centuries, if at all. Magic was, after all, more useful and powerful for some uses than technology, and the surrounding territories had enough power to keep Home from expanding quickly, plus if everyone around them decided to cut off trade it would weaken their industry to non-sustainable levels.
Darwath - Barbara Hambly
Firearms have been basically useless since one of the easiest things for a mage is to trigger latent forces or exploit elemental energy. In other words, guns go boom at wrong times, don’t go boom at all, or go boom way, way bigger than they should. Even in the real world, misfires and magazine explosions happened often with all forms of early firearms. Powder factories blew up with alarming regularity. Imagine if all you needed was a handful of people with the ability to blow things up from a distance to keep someone from building a stockpile of powder.
In the first books, there are a few people from our world who cross over to Darwath, and she implies that all of her other series worlds are likewise reachable from each other. In related books computer technology is harnessed to magical. There have also been invasions by demons and other dimensional travelers. At least one “demon” in later books is shown to be a high-tech traveler from a different dimension, and becomes a friend of sorts to one of the characters, showing again that magic and tech aren’t fundamentally different.
There was a past history of worldwide domination by various mageborn and this post-mage world is climbing back out of a dark age following the collapse of those empires. Technology/society varies from medieval to roughly mid-19th century industrial, depending on the time period shown, and some remaining ancient magic-tech artifacts imply a past level of progress that is on a par with SF style futuristic tech.
Recluse - L.E. Modesitt
Backstory reveals that at least some high-order technology works on the same kind of forces as magic. Chaos and Order are linked/opposed. There are mages with an affinity for one or another. It’s later revealed that both kinds of mages can, to some extent, work with the opposite forces, but can only channel one. There are consequences for working with either, and the side-effects can be dire.
In the book that introduces how Recluse was founded, the main character goes blind from being around and handling too much Chaos. The laws magic adheres to are strict; any death, any destruction, no matter what the intent or need of the person, causes chaos. Manipulating magical forces directly results in severe backlash unless handled properly, and even then there are undesirable results.
In some stories, there are Chaos or Order mages who attempt to create technological items using both magic and standard tools, but it’s not easy. One mage who makes a steam engine has to keep the thing from blowing up under opposition from mages who could increase local Chaos to the point where the metal would fragment like paper. And that’s after he’s solved the metallurgy and tooling problems starting from basically a high-medieval level of tech, and found a way to get funding.
Secrecy and infighting on the part of the mage societies, a fundamental opposition between the Order and Chaos mages, a lack of codified knowledge about how magic works, and the relatively small number of mage-born as well as the strict limitations on how magic works all keep mages from being too powerful. There’s very little cooperation, and enough fighting back and forth to keep progress very slow. On top of that is the normal-world political wrangling and warfare. There are indications that past societies had reached a high level of magic-technology, but that there were severe consequences with long-lasting environmental impacts.
Ethshar - Lawrence Watt Evans
There are many kinds of magic. Wizardry is probably the most powerful, and taps directly into the chaos underlying the universe. The rules of wizardry are very random, though wizards seem to have figured out some fundamental principles.
Research is…problemmatic. There’s a column of fire hotter than dragon fire that extends from the ground to the heavens in one of the places called the Small Kingdoms. It has been burning for three centuries. Scrying and other research spells the wizards used to find out what happened revealed that it was produced by the wizard sneezing while doing one of the simplest low-level fire-lighting spells taught to beginning apprentices. Oops.
There’s also sorcery, which is the closest thing to technology. It seems to be the creation of magical artifacts, and it’s implied that sorcerers also could do some kind of bioengineering or magical manipulation of humans to enhance or change them. The part of the World that uses mainly wizardry fought a 5000 year war with the North, which used mostly sorcery. During that war, when taking those kinds of risks was acceptable, the average lifetime of a research mage was stated to be about 3 years, showing again that messing around with wizardry is not a particularly safe thing to do. Eventually, the gods stepped in and ended the war by destroying the North. So while technology is not anathema, you’re not going to be seeing a high level of sorcery being used in society for a very, very long time.
Warlockry and Witchery in this world is basically psi, but treated as magic. A technological link for the power for these magics is explained a bit at a time in different books. All magics, including theurgy and demonology, seem to have at least some affinities and opposing forces. For example, some things produced by wizardry are invisible to or unable to be affected by gods or demons. Wizardry doesn’t work reliably on Warlocks or Witches, and sorcery and wizardry seem to work at least a little bit on some of the same principles.
One oddball is Metropolitan by Walter Jon Williams, where plasm is collected by structures so everyone lives in enormous cities of skyscraper-like buildings designed to channel and collect this magical energy. It reads like SF even though the “technology” is solidly F. In a way, it demonstrates why so many fantasies are set in medieval societies. The setting determines a great deal of the feel of the technology.
Star Trek technology might just as well be magic since a great deal of the technobabble is about as scientific as Harry Potter’s incantations are Latin.
So, good reasons are commonly presented for technology or a technology/magic interface to have stalled at about the level it’s portrayed in many fantasy worlds. I would probably have to look harder for those that didn’t feature fairly solid worldbuilding that implicitly explained the reason for that setting than for those where technological progress would outstrip magic, given that magic has real tangible effects in that world.
Asimov keeps riffing on this in FOUNDATION’S EDGE, where the robed scholars with might-as-well-be-magic psychic powers keep priding themselves on not bothering to keep up with the latest technological breakthroughs on other worlds – which is a bit jarring, given that their old-timey stuff is still futuristic to us.
So their faster-than-light starships are hopelessly out of date – taking weeks to span interstellar distances modern vessels manage in minutes – sure as they don’t much care what sort of military hardware a cutting-edge fleet is packing, because, hey, it still just comes down to a guy who pushes a button, right? Or who doesn’t push that button, if he’s under your spell? Or who obliterates third parties for us, if we ever command 'em thusly? So, hey, you guys just keep on with the blood and toil and tears and sweat, building ever-more-impressive gear we can seize through you at will; we’ll just keep living amongst simple farmers, cultivating the power to cloud men’s minds in between poring over predictions made five hundred years ago…
An exceedingly good point, here—in a lot of modern sci-fi, “technology” or “science” often just takes the place of “spells” or “magic,” to the point where you could practically just swap the terms around and make more sense. Even the Sonic Screwdriver from Doctor Who is essentially a “Science Wand” these days!
Maybe this is just an archetypical underpinning of western mythology and storytelling…or maybe it’s just lazy writing. 
On a related note, it kind of reminds me of Gaiman’s American Gods, where he notes “the 'chopper” as taking a role in modern fable equivalent to Valkyries. “They come because they have to come.”
A “Jewish physics” situation, in other words.
Basically, yeah. Rowling’s depiction of the baddies in Harry Potter was pretty clearly influenced by the Nazis.
But focusing on developing magic instead of technology would give wizards a real advantage in conflicts with Muggles, since Muggles can’t cast magic spells. The Nazis presumably would have been delighted if it had been possible to use “German physics” to develop a weapon that could only be deployed by Aryans and could never be replicated or turned against them by other races. Had wizards directed their efforts towards improving things like non-magical weapons or even medical treatments, these could have been used by Muggles to their own advantage. The various curses that appear in the Harry Potter series seem to have been developed primarily for use in conflicts between wizards, but it was probably considered a plus even to the more tolerant wizards that there was no chance of Muggles ever being able to go around Avada Kedavra-ing folks. (We only have guns, which are easier to use but clunkier and less effective.)
There’s a simple reason fantasy writers don’t like advancing technology in their world: loot. When the protagonist finds the sword of the ancient hero, 3,000 years dead, it’s long and made of steel instead of short and made of bronze. When he dons the hero’s armor, it’s made of plate and mail, not leather and fur. Otherwise, what good is it?
In a broader sense, in fantasy, the past is always better than the present. The wizards were more powerful, the monsters were meaner, the towers were taller. It makes sense that technology in the past will be, if not better, than at least no worse.
Well, I’m increasingly inclined to believe that a story’s setting (futuristic vs. antiquity) or attitude towards the past vs. the future is a separate and orthogonal classification from whether it’s set in a magic-based vs a technology-based world.
Past + magic = traditional fantasy (heroic fantasy, swords & sorcery, etc.)
Past + technology = steampunk, maybe some historical fiction
Present + magic = urban fantasy, some horror
Present + technology = techno-thrillers
Future + technology = hard science fiction
Future + magic = Sci-fi involving what are essentially magic powers, of the sort described by drewtoo99 and Ranchoth:
One of my favorite fantasy works is the novel The Case of the Toxic Spell Dump, by Harry Turtledove.
The novel is set in contemporary times, not the Middle Ages. Magic works. It’s a fun story, with characters I like, and would make a hell of a movie.
Who wouldn’t love a vampire hampster, flying carpets, and virtuous reality technology?