No, I think they hate him for some things that he did do. You see, he raised taxes, the thing that was supposed to drive the economy into ruin. Instead it took off, and we wound up with the first surplus in ages. Now Bush did what the conservatives said was going to improve the economy, and it sat there like a lump. (yes, it is improving now, but it always rains after the rainmaker does his stuff. If it takes three months, he still claims credit, right.)
Lets look at the last few.
Nixon, while he was a paranoid loonie, didn’t steal the '72 election - he would have won without Watergate, which makes it especially pointless. Besides that, he opened China, signed the Clean Air Act, and was in many respects a decent fellow.
While LBJ was responsible for Vietnam, he also got the Civil Rights Acts passed (something Kennedy couldn’t do) and did a number of positive things.
Bush I did a very competent job with the gulf war. Reagan was far more pragmatic than people give him credit for - as mentioned he did raise taxes as the deficit mounted. Ford and Carter, while nothing to write home about as Presidents, certainly never started any wars or lied to people.
Some older ones: Hoover certainly mishandled the Depression, but remember there was no economic theory then telling him what to do. He was not an evil person, he made his name as the leader of relief for displaced people in Europe after WW I. Harding and Grant were surrounded by crooks, but that’s not enough to be the worst. Andrew Johnson, if you remember, got impeached for trying to reunify the country., which was following Lincoln’s direction.
I can kind of see Jackson and Buchanan being in contention - but we’re down to a select few at this point. Buchanan did nothing, true, but things were too far along at that point to stop. I never thought about Jackson being the worst, but I can see the argument.
I tend to rate people who did nothing to help (like Pierce) lower than people who actively make things worse (like Bush.)