IMO, it’s a tossup between the French horn and the violin.
The violin is difficult because it’s all so subjective–woodwinds have keys you push to get the right note, and so do most brass, and even the organ (sorry) is just a matter of mechanical dexterity (or “pedal-ity”?), but to play in tune on a violin/viola/cello requires literally years of practice and a good ear. (Please don’t try to tell me about Suzuki violins–I’m sorry but I have yet to hear a group of them play in tune.) Also, with a violin, not only do you have to get it in tune, you also have to get the right tone, or timbre, i.e. so it’s not that horrible squeaky, squawky noise. You have to make it really sing, which also takes years of practice.
Also, strings have many more “special effects” like pizzicato, etc. that have to be mastered in order to perform the orchestral literature.
The French horn is just a bastard to play, pardon my French. You can learn the fingering, sure, but then you have to continually adjust the pitch by sticking your fist in the bell, and some days it just won’t play in tune and there’s absolutely nothing you can do about it. Some of the notes called for even REQUIRE your fist in the bell, making it an experience much closer to playing a violin than a trombone or trumpet. Also, orchestral parts for French horn tend to be written so as to emphasize its distinctive tone color, and if you’re wrong, baby, you’re 100% wrong, right out there in front of God and everybody. Even housewives from Peoria will notice if, say, the noble horn calls from various Wagner operas are not in tune.
I think the remark about the oboe being the most difficult to learn has to do with the specialized and very difficult embouchure that’s required to pinch that double reed good and hard, but without allowing it to squeak or squawk. A flute, clarinet/saxophone, and brass instruments also require the development of a special embouchure, but IMO it’s not as exhausting. Anybody can play a flute or clarinet well–but not everybody can play an oboe well. I think that’s the difference.