Might qualify, might not, but the Butthole Surfers have their avant moments.
I think many of the classic “art rock” bands would fit here:
Emerson Lake and Palmer
King Crimson
Jethro Tull
Yes
Traffic
Oh, man, much as I love some of those I have to disagree. Once you’ve got a genre that you stick pretty comfortably within–prog rock–then you’ve left the “avant” label behind.
I think of the artists who are legitimately avant garde as each being in a genre of one. The artists named above are avant *compared to *the previous generation of rock musicians, but none of them is sufficiently unique to be considered avant garde outside of that specific context. (Using “unique” in its literal, absolute sense.)
Not sure what fits in the OP’s category, but I love the following:
Pierre Boulez, “Dialogue de l’Ombre Double” (for clarinet and computer-controlled circle of speakers)
Frank Zappa and Steve Vai, “Dangerous Kitchen” and “Jazz Discharge Party Hats”, from Man from Utopia (for guitar, with notes transcribed from spoken voice “poems”)
several works by Cecil Taylor (piano)
P.D.Q. Bach, “Einstein on the Fritz” (Peter Shickele’s parody of Philip Glass)
The Beatles, “Tomorrow Never Knows”
several works by Laurie Anderson or Diamanda Galas (both already mentioned by others)
Plastilina Mosh (Mexican “experimental” band)
I think John Cale fits in here.
Brian Eno’s Taking Tiger Mountain by Strategy and Ambient 1: Music for Airports
There was a recent reissue of Eno and David Byrne’s My Life in the Bush of Ghosts that’s really freakin’ incredible.
Renaldo & The Loaf
“Is Guava a Donut?”