Possibly some affiliation with this religious rodent reprobate:
In similar (anemic) vein:
Mild but interesting Ukulele Orchestra of GB doing the theme from Shaft
Tony Scott, Hare Krishna
Some really good ones here. My contribution: 365 days of outsider music. You’re welcome.
^ Thanks, an oddity goldmine! Now another take on the Krishna theme, by the Pizzicato Five, who make it sound like an aerobics class.
You know, on consideration, I’m not sure how many of my picks would be “quirky” or just “weird.”
But I think I’ve got one old favorite that still qualifies, even if it’s a little obvious: Roland the Headless Thompson Gunner.
Not quirky at all, just a sweet, beautiful sound. Might as well be here.
Up, Up and Away by Raj and the Sepoy Mutiny
Terrible Operation Blues by Georgia Tom and Jane Lucas
Up up and away, I love it! They do a great version of “the Letter”. Best bogus sitar I’ve ever heard!
Nothing else sounds remotely like this. It’s four people wandering off in four different directions - or some of the tightest synchrony ever devised. Yusef Lateef, Psychicemotus:
I give you… The Bran Flakes
The Residents - The Third Reich ‘n’ Roll
The Residents - Constantinople
More to come. This is just from Ralph Records.
Vic Reeves and Bob Mortimer are a good source of random musical nonsense. Particularly
“prepare a lovely crisp salad and have sex with a kestrel above a caravan”
And of course their compendium of “Mulligan and O’Hare” compositions
“She wasn’t immunised
That’s a legal requirement
She’s increasingly slapdash
since we bought that new hearth-rug”
Try Messer fuer Frau Mueller for some wacky mix of surf, carnival, electronica, cartoon, disco, lounge, 50s rock & roll, pretty much anything under the sun. I don’t know much about them other than they’re out of St. Petersburg and have been around since the early 90s. There is also a related band, Messer Chups, that is kind of horror surf music, for lack of better description. An example of Messer Chups.
Continuing on the weird Russian music theme, there is the experimental rock band N.O.M. that I enjoy. Here’s a semi-cover of Queen’s “Bicycle”. It’s a song called “Apiary” and, from what I understand of Russian, the lines that correspond to “I want to ride my bicycle” translate to “I have an apiary.” Apparently, it’s the title track to a movie called “Apiary.” "Nina and Nina 2 are two essential tracks to give you a sense of their work.
The Moog Cookbook is a collaboration between two Moog synth enthusiasts who record analog synth covers of pop hits. There is wonderful musicianship on their albums, great use of sounds for texture, and a clear, not-taking-itself-seriously quality to what they do. I love how they recontextualize the songs and play with the genres, taking something like Green Day’s “Basket Case” and turning it into something that sounds like an early/mid-80s sitcom theme. Or Soundgarden’s “Black Hole Sun” into some 70s Latin lounge synth cheesefest.
This is Cuban group performing “Hotel California”. There are no instruments. All the sound is done by voice.
The Ghost Frolic, used in two “Our Gang” shorts.
Well, there’s neo-cabaret in general, and Vagabond Opera in particular, fronted by a trained opera singer (not, IMO, a particularly good one, as comes out on some of their songs; he’s best sticking to “normal” singing). Here’s The Transformation Into Marlene, which is extra-quirky since it’s about a guy wanting to become Marlene Dietrich. 
And then there’s Alamaailman Vasarat, a Finnish… um… band. There is no describing them; just go listen.