A lot of music came naturally for me. I have a really good ear and a really good ability to recall a tune in my head, great memory for lyrics. I also have excellent hand-eye coordiantion, high dexterity.
However, learning to make any specific instrument sound the way I thought it should sound was another matter. Getting my hands used to certain movements so I didn’t have to think about it was hard. These troubles were only solved by hard work at practice. Even with my advantages over others I was a sorry musician for several years. I didn’t get good until I was stuck on the farm one summer by myself with no transportation other than a horse or tractor. Circumstances (play music or be bored) forced my hand into becoming an expert musician after practicing at least ten hours a day. That summer I became a good musician on several instruments. (Piano, guitar, drums, and mandolin.)
The difference between me and a lot of others is that one summer. OTOH I know people I have tried to help for years who just lack control of their hands and my memory and the ability to hear what is wrong (or right) in their playing. No amount of practice is going to make these people competent, they just can’t do it. On the other hand, there are people who play perfectly even very complex stuff from a very early age. Extremely rare.
I’m voting for hard work as the result of drive and that any average person can become a competent musician. The best have superior abilities.
As to the O.P. and annoyance for people who do not understand and dismiss hard work by musicians, I understand perfectly and agree that it certainly is annoying. I just politely smile and say thank you.
The single most important thing I think is drive, how badly a person wants to.