"We aren't" != "You are"

“Some people say you’re too passive-aggressive, but I have some doubts.”

Just because you say ass-holish things, doesn’t necessarily make you one.