A lot of viruses that are not thought of as typically invading neural tissue can do so, and when they do, the tissue sometimes does not recover.
Chickenpox, for example. I lived at Gallaudet University for a year, so I’ve met people deaf from just about every cause you could think of, and I know a couple of people who became deaf after a bout of chickenpox. Turns out that chicken pox is generally thought of as benign, but it turns out that while it very rarely invades neural tissue, it can invade the auditory nerve, the optical nerve (even rarer, but there are a few cases). It also appears to be one of the viruses that can cause autonomic neuropathy, a condition where a number of systems that are not under a person’s conscious control-- digestion, blood pressure, pupil dilation, bowel and bladder control. Again, rare, but several viruses can cause this.
I also know of a case where adult chickenpox is blamed for male sterility. The guy had children already, so not born sterile; caught chickenpox from one of the kids, was very ill, because, adult, and couldn’t conceive with his wife afterwards; a sperm count showed that he was sterile. He had no other illnesses, and no injuries to the groin.
The effects of maternal Rubella and Zika are well-documented, and the seriousness of adult mumps, as well as the possibility of measles encephalitis, which usually shows up within six weeks of recovery, but can take several year-- and is always fatal.
I know there’s some research going on now to see if a maternal virus that is asymptomatic in adults might cause some types of autism.
So it’s not unreasonable to wonder what COVID could cause down the road.