Sounds like the Neanderthal Parallax stories to me (And likely many other SF stories I don’t know about). In that triology, a door was opened to a parallel universe, and although there were no demons involved, there were Neanderthals, who had become the dominant hominid species in that universe.
The door was caused by a quantum-computing experiment gone wrong. It was based on the idea that quantum events occur across time and so the computer reached out to other “simultaneous” universes where it also existed while trying to factor a number that should have been prime. The number was too large, however, and a flaw in the computer led to it accessing a universe (ours) in which it did NOT exist, but in doing so it managed to transfer a Neanderthal over.
I do not know the math enough to know how much of this concept is theoretically possible and how much of it came from the author’s mind, but I do know that math allows for many more dimensions (and universes, I guess) than we know of here.
Sounds to me like your f-o-f was just telling ghost stories with just enough facts in them to make you really wonder about it…
the sign of a good story teller, IMHO.