I don’t believe that can be right, as the word “kazoo” appears in the very next line, and John Fogerty is too smart a lyricist to rhyme a word with a distorted version of the same word. Also, you don’t “twang the rhythm out” on a kazoo; you buzz the melody out. It seems like he’s referring to something along the lines of a washtub bass or a cigar-box & rubber band banjo, but I’ve never heard of any such instrument called a kalamazoo.
And Genghis Bob, I think he means “doubling” in the sense of playing an instrument in addition to your primary one, as in a saxophone player doubling on flute. So Willy’s main role is dancer, but he doubles as the kazoo player.
Then again, at the time this song was in the charts, I thought that line was “And when he goes into a dance, the devil’s on the loose.” Hmm, where’s that So that’s what he’s singing thread?