Unless you have a physical deformity or a brain injury keeping you from pressing the keys or reading music,  this is simply not true.  There have been many studies that have tried to disentangle ‘talent’ from ‘hard work’, and they universally find that almost all ‘talent’ is simply hard work.  Wayne Gretzky was a ‘talented’ hockey player - who was playing hours a day on his home rink since he was a small child.  The Beatles were ‘talented’, but had been playing together in bars in Hamburg for years on a gruelling 8 hour a night schedule, and had thousands of hours of experience as a band before hitting it big.
I read a study a few months ago that compared entrants to a music academy - I want to say Berklee but I’m not sure and I can’t find the study now.  But they rated the students coming in to find the ‘prodigy’ students - the ones that were supposed to be extremely, innately talented.   They then measured them against their peers in the class to see how much farther ahead they were.   Then when they graduated they were measured again - and what was found was that the ‘prodigy’ students did NOT advance faster than the others, and in fact the students who advanced the most were simply the ones who did the most work.
When I was young I used to tell myself that I was very uncoordinated.  Always tripping over things, never as good at physical activities as the jocks, etc.  In fact, what was actually going on is that I was a bookworm and I was always thinking about stuff, so I was just careless and didn’t pay attention to my surroundings.  I wasn’t good at sports because I didn’t play sports for fun like the other kids, so I didn’t have as much practice at it.  In my mind, that translated into ‘lack of physical coordination and talent at physical things.’
After high school I was about 30 lbs overweight, and still telling myself that I was a physical schlub.  So, I enrolled in a martial arts class.  And excelled at it.  I discovered that I was just as coordinated as anyone else, and I worked my ass off and very rapidly developed physical awareness, balance, and all the other things necessary.
Talent is nothing more than evidence of accumulated practice.   Now, one thing is true - people will advance faster if they know HOW to practice, and that too is a skill that needs to be learned.  When I was teaching karate to kids,  I could predict almost instantly which ones would advance at the fastest rate - they were the ones actually listening, trying to apply what they were taught, were willing to ask questions, etc.  Just showing up and going through the motions will not get you there.
If you take piano lessons with the attitude that it’s really hard and you aren’t cut out for it or lack the innate talent to do it,  you’ll practice half-heartedly and give it up before you reach the point where you could see the improvement.  That will reinforce your belief that you just don’t have ‘talent’ for it.  If, on the other hand,  you listen to what your teacher is telling you and you focus hard on where you are going wrong and you practice an hour a day between lessons,  I guarantee that you WILL get better, and it will not take long before you find yourself able to play things you never dreamed you’d be able to play.
I know this is true because I started piano lessons when I was in my early 40’s, expecting to fail at it (I took the lessons because my kid was taking violin lessons at the same place, so I decided that instead of just wasting my time sitting and waiting, I might as well learn something).  I didn’t continue with it after my son left there for a private teacher,  but I could see that if I had just continued I could have been a perfectly adequate recreational piano player in a couple of years.  And I would never have believed that going in.
Bad examples.  The reason you have a good head for trivia is no doubt because you’ve been reading books like crazy since you were a small child.  Am I right?  I’ve got a good head for trivia too,  but like I said I was a bookworm.   You may have enjoyed the reading and not thought of it as ‘work’, but it was certainly good practice for Jeopardy. Feynman was a ‘natural’ at math because he was doing math puzzles when he was very small, and by the time he was fifteen had already spent more time thinking about mathematical problems than most people do in their entire lives.
Speaking of that… My kid was told by his grade 10 teacher that he had no aptitude for math and should give up on science in university and go into the liberal arts where his lack of math skill wouldn’t hurt him.  Instead of listening to the old bat, he knuckled down and studied,  and I tutored him extensively.   Last week he started his third year in honors mathematics at university, with a minor in computer science.  He does mathematical proofs for fun now, and has so far a straight ‘A’ average in all his math classes.   I’ll bet the other kids who don’t work as hard think he’s just ‘lucky’ to have that ‘innate talent’ for math.