I do think there was a long spate of poor Presidents, I don’t include Jackson and Polk among them, frankly. I’ve said in other threads I’ve always been a tad anti-Jacksonian (for his disastrous Bank policies), but he was a very consequential President that did a lot of things that set the course of the country for generations. Additionally he was just a very strong political operator, in an era of weak candidates and big party power brokers, of weak Presidents and dominant Congresses, Jackson dominated everyone through sheer personality and force of will.
Frankly anyone who isn’t willing to give California and Texas back to Mexico is a huge hypocrite to criticize Polk or Manifest Destiny. The U.S. conquest of the land to the Pacific was a good thing, not a bad thing, and it’s hilarious to try to portray something that literally allowed our great country to be great as a bad thing. Mexico has always been a terribly mismanaged, ill governed country (if you actually read the history of Mexico prior to 1850, it was arguably much worse so then than even now.) We could have a whole other thread about why that is, but the simple reality is this land was far butter under American management, and the Mexican Cession and Texas are truly part of what has made America a great country and a great power.
Now having said all of that, I think a major reason we had a long string of middling and poor Presidents is because of how the party system worked and the role of both the Presidency and the Federal government.
The President’s original design really made his handling of matters of war and foreign affairs extremely important because they are almost sole executive powers, and powers in which the Federal government has basically sole authority (not shared with the state.) Domestic policy the President was simply one part of the Federal decision making apparatus, and on top of that the Federal government was far, far weaker relative to state governments in the 19th century.
The way the party system developed in the 19th century is party power brokers would make most of the major decisions, and extreme horse trading would occur to determine who a Presidential candidate would be. Decisions about the cabinet would usually be part of these negotiations–meaning a President might be nominated with a lot of backroom deals and agreements already binding many of his most consequential decisions. “Propriety” barred Presidential candidates from directly campaigning for election, instead it was all done by surrogates, additionally the limited communications technology of the day limited the ability of a “strong personality” (think an FDR type) to establish any kind of true national following. Many times the men who would get the nomination would be only moderately known outside of their home state, and then the voters would vote largely on their party leanings (when this all happened under the aegis of the Democratic-Republican party, people voted based on the factions within that party they belonged to), so the President came to power with little recognition and limited “base” of support.
Juxtapose this with Congress, which the Constitution actually empowered as clearly the most powerful of the three branches in theory, and in the 19th century it mostly did operate as the most powerful branch. Leadership in Congress largely required years and years of experience in national politics, so the leaders in Congress were national figures, powerful within their party and known well throughout the country. So powerful figures like Henry Clay, who never won an election for President despite trying, was many times likely the most powerful man in the U.S. Government, almost a quasi-Prime Minister with the President being rendered into a state not dissimilar from a constitutional monarch.
So basically, until the latter 19th century, the “job” of President wasn’t as big as it is now, the powers of the Federal government were less, the “real” powers of Congress were much higher. However the President did have areas in which they wielded immense power, it’s just those powers weren’t always relevant to the time. To breakthrough this and be consequential you needed a President who was very political powerful, or who was running things during a power in which the Presidency’s constitutional powers were more relevant.
Lincoln is an obvious example–he was actually not that strong politically in his first years in office, often at odds with Congress and the Judiciary, but the constitution and its rules written and unwritten vest immense powers in the Presidency to quell insurrection and to conduct war, and that was the meat of Lincoln’s Presidency tied to the Civil War.
Jackson and Polk both kind of broke through because unlike their contempoaries (the van Burens and the Fillmore types), they had powerful political factions that were personally loyal to them, this tended to be unusual of Presidents in this time period, and possessing that made them forces within the government.
The first five or so Presidents largely get such high marks because there was a mixture of foreign policy, military and precedential activities going on, precedential meaning these guys were kind of fleshing out what the Presidency even was and how government would work. Most people have generally felt the first five Presidents aside from Adams did a good job at those things (Adams is oft considered one of our best Founding Fathers but not a particularly good President.)