Not the official “I’m Popeye the Sailor Man” song that dates all the way back to the 1930s Fleischer cartoons, or any alternate lyrics. What the OP is referring to is a song Popeye would sometimes sing to himself in the 1960-1961 King Features cartoons (the ones where his enemy was called Brutus). A sort of bee-bop tune with improvised nonsense lyrics (is that style called skat?)
I’m confused too! 
I thought the scat part of the Popeye song went something like this:
No, that’s the Witch Doctor song.
I remember Popeye scatting in many an old cartoon. Certainly in the Max Fleischer cartoon, and possibly in the cartoons of the Famous Studios era. Jack Mercer provided the voice of Popeye from 1935 until his death in 1984, except for a period during WW II, when he served in the U.S. Navy, so it’s possible to hear him scatting during just about any era.
The line “skiddly-bop a doo skiddly skee bop a skiddly bop” sticks in my memory as one of his lines, too.
Thanks for making those long-repressed memories surface up! I wonder what other sorts of flotsam and jetsam was pulled up with it.
Here’s one example of Popeye singing nonsense to the music in 1936. Little Swee’pea.
It’s not quite the same as the scat singing in the later cartoons (which I think the OP is referencing). I don’t recall any particular song he usually sang, I think he just did this to any music (though this meant mean that he can hear his own theme song playing). But I’m no Popeye expert.
That music is from the Popeye cartoon “Brotherly Love”. Original lyrics here: Production Information about the Theatrical Cartoon Brotherly Love
The skat singing the OP refers to occurs in one or more of the color (Harvey? Paramount?) Popeye cartoons, not the earlier b&w, according to my vague memories of the audio.