Noooo, I love James Joyce. His wit can be obscure, but figuring it out is part of the fun.
Le Guin’s Buffalo Gals was a very tough read for me. At the time, I was having a lot of difficulty in my life and I actually gave up on it. She’s an author I adore and I actually gave the book away (it was laughing at me, I’m sure). I wonder, if I read it now that I am older and have my head on straighter, will it still be nonsensical to me?
I think Joyce takes the cake, but I am compelled to insist that Mason & Dixon is a much, much harder read than Gravity’s Rainbow. GR is Ikea directions for building an MC Escher infinite staircase. M&D is backcountry yokels trying to describe how to build a semiconductor.
Dostoyevsky’s Brothers Karamazov was the first Russian novel I read, and I had a very difficult time with it because of the Russian names, and how seemingly every character has a dozen different names. War and Peace was the second Russian novel I have read. I was more used to the Russian naming style by then, but I think I will read it again within the next year (I originally finished it late last year). Now I’m reading Anna Karenin, which I find is much easier to understand than the other two books I mentioned due to my current semi-familiarity with Russian names.
I can’t believe that no one’s mentioned Atlanta Nights by Travis Tea.
Or does it not count because it was intentionally written so as to be indecipherable (missing chapters, duplicate chapters, random computer-generated text, etc)? I bought a copy of it a few years back just to see if it was as bad as described. It was.
On a related note, I challenge anyone to read aloud The Eye of Argon, in one sitting, without any breaks for laughter (or crying - from the pain brought about by trying to stifle the laughter).
I accidentally picked up Samuel Delaney’s Dhalgren in an airport some years back. Couldn’t get through much of that at all. Went back later and read about it on wikipedia, and realized I wouldn’t have cared for it even if it had been a little easier. 
I love that book. it’s probably in my top five. It’s also a ton of fun to read on the subway, and enjoy the bewildered looks from people who see me turning it aroundandaroundandaround. 
I actually came in here to mention Mason & Dixon, which I gave up on. Just didn’t care enough to keep going. I haven’t attempted Gravity’s Rainbow yet but it’s on my list of things to do before I die. So is Infinite Jest, which has sat on the shelf taunting me for years now.
As is usually the case, Hesse won the NPfL for his entire body of work, not just for that novel.
I found that Radix by A A Attanasio was a pretty hard slog. And I speak as one who enjoyed Rudy Rucker’s book on infinity. It was a long time ago and I don’t remember whether I finished it.
Too many characters and not enough plot.
I thought that was a wonderful book… definitely not as brutal as Finnegans Wake. Not even close. I had to reread some things a couple of times, but that’s what makes a good book.
Simply not true.
It doesn’t necessarily “rape the rules of grammar,” though - read it as an imperative or as a complete sentence!
Anyway, my vote is to William Gaddis’s JR. I read The Recognitions with pleasure, though it took a long time, but this gave me a headache that got worse with each page. I’m sure someday I’ll read it, probably when I’m old and gray, but for now I think it’s just too much.
Didn’t finish Finnegans Wake (did anybody?), but I have read Gravity’s Rainbow and Ulysses, and found GR much more difficult. And I maintain that if you can get through the first 150 pages of GR, you can basically tell people you’ve read it - it’s not as if it becomes increasingly more brilliant in the following 600 pages or anything (IMHO).
I also found John Dos Passos’s USA Trilogy pretty far up there on the difficulty scale.
Needs a comma then: “Finnegans, wake!” as in, Finnegan family, wake up! Without the comma, it sound better to me as “the Finnegans wake.”
I agree that Mason & Dixon is a harder read than Gravity’s Rainbow. I know I read it, but I couldn’t tell you anything about it, other than it wasn’t worth the effort.
I read about 50 pages and gave up. I’m willing to work for my literature, but not that much.
Faulkner’s the Sound and the Fury is the one that does it for me. Jaysus. Even my literature degree didn’t penetrate that one, and I studied it in two different classes!
I came in here to post the exact same book! Exact same grade as well. God, I hated that book with a passion. And this is coming from a person who loves to read. I went back to my old high school and found that they don’t use that book anymore, can’t say I’m surprised.
RIP Logicism