I couldn’t even offer a guess as to the etymology but I can say that at a bookstore I worked at over 20 years ago we called the section with supernatural/paranormal, etc. books the “woo woo” section. That was the first time I had ever heard the word used that way. There were plenty of books of quackery and bad science and the like in the store but they did not get this designation. At the time it was reserved for occult/supernatural type books. I always just took it to mean something like “wooo, scary.” Searching for an etymology online via Google books, etc. is difficult because “woo” has been used to mean courting or romancing for hundreds of years. Its hard to separate the results to find early cites of this use.