Yeah, you should go. It’s rare that you’ll find a concert where you love EVERY piece. Stifle yourself during the Bernstein and enjoy the Schumann.
Incidentally, Wagner’s Siegfried Idyll is gloriously beautiful. So there’s that too.
I stopped going to my local symphony years ago because of this. I wanted steak and a baked potato, they wanted to give me broccoli and brussel sprouts. Fuck them.
I can only WISH I could go to hear a symphony. How excrutiating could it be? I would suck it up and go and hold my fingers in my ears during the dreaded hateful part - ‘lalalalala I can’t hear you’. Of course, I don’t know much classical music, so I wouldn’t have to hear the horrid pieces, it’s all good, more or less, to my untrained ears.
“I don’t write music for sissy ears.” -Charles Ives
Did he ever cover anything by someone called Ulysses?
Well, I went and enjoyed the Wagner and loved the Schumann. I survived the Bernstein. I think I’ll probably never be a fan of his music, but some people in the audience certainly seemed to enjoy it.
Lennie’s Overture to Candide is pretty tasty; it shows up on programs frequently, but not enough to have made it to “chestnut” status.
Take it you don’t like West Side Story? 
I think he composed West Side Story for the mass audience, whereas he composed other music in order to impress other composers. I’d be interested in knowing what other composers think of his music.
I am aware of Ives’ fondness for competing melodies (in different keys); it’s kind of his thing. And I do find it intriguing. And if one does some reading up on Ives, there’s plenty of explanation of why he does it.
None of which requires you to like it, although I do. And if you find Ives horrible, you should definitely avoid Stockhausen and Crumb.
At the Insurance Executives’ Banquet, Hartford, Connecticut, early 20th century:
Charles Ives: This music of mine is REALLY going to fuck up the squares!
Wallace Stevens: Wait until you get a load of my POETRY.
Or this:
Charles Ives: This music of mine is REALLY going to fuck up the squares!
Wallace Stevens: People are not going
To dream of baboons and periwinkles.
The advantage of being independently wealthy is that one’s artistic vision isn’t constrained by what other people are likely to want to pay money for, and therefore one can be as “experimental” (read: odd) as one likes.
See also: the Sitwell siblings.