I don’t particularly find him assholish. There’s something about his character that takes the edge off. He seems like a nice guy to me. Obviously, others find his humor mean-spirited, but I don’t.
I have always found him to be very subversive, in that his humor (especially in his stand-up) consists of saying something outrageous with the twist being, “Can you believe that some people would say this and actually mean it! And that there are people who would laugh at it thinking it’s funny? But now YOU are all laughing at it!” So, in a way his schtick is tricking the audience. I find that to be clever.