I think we’ve covered a lot of the ground that needs to be covered here, and I’m bored enough to follow up my tongue in cheek first response with something more serious.
There is an association of wealth and privilege with fraternities. Rightly or wrongly, most of the time when you hear about someone who was in a fraternity, they’ve come from a wealthy family. It is my understanding that there’s also an emphasis on “second generation” pledges - you’re more likely to get in a fraternity if your father was in the fraternity. Many fraternities have a reputation for wild parties and hedonism as well.
All in all, it tends to create a stereotype of fraternities as a peculiar kind of American aristocracy. It’s wealthy, hereditary, and living better than the proles. That stereotype is bound to generate resentment.
Now, my own issues with fraternities are much simpler. I don’t like exclusive organizations. I don’t like organizations that I feel encourage ‘groupthink’. I dislike the notion of valuing brotherhood more than truth or merit.
Fraternities. The Army. The Police, to a lesser extent.
I never considered joining a Frat, even for a second. I never considered joining the Army, even for a second and despite prodigious recruitment attempts. I have never considered working in law enforcement, despite my father’s career in the field and a fascination with criminology. While it is most certainly not the case everywhere, the danger that I might be put in a situation that I would have to succumb to peer pressure to lie, else my life would be ruined by my comrades in retribution for my lack of ‘brotherhood’, was sufficient to discourage me from any of these paths.
I’m a strong believer in individualism, and I see fraternities as the opposite of that concept.