Carter was the first president since Andrew Johnson who did not nominate any Supreme Court justices.
Carter won’t be remembered like Fillmore because he was fairly prominent after leaving office. You know, such as winning the Nobel Peace Prize.
However, no 20th Century president will ever be as insignificant as Fillmore (whom I don’t think was all that insignificant if you ever read what he did in office). The 20th Century saw a dramatic increase in presidential power and the size of the Federal government. Anyone holding the office of POTUS got it because they were important. Even Ford, even Carter.
As for Fillmore, he did sign into law the Compromise of 1850, which his predecessor Taylor probably would not have done. That Compromise served to delay the Civil War for 10 years.
It was also Fillmore who sent Commodore Perry to Japan in an attempt to open up some sort of trade with that nation. It’s just that by the time Perry got there, Pierce had become president.
And Fillmore had enough standing to run for President again in 1856, albeit on the Know Nothing party ticket.
Granted, being connected with an Anti-Immigrant Party and signing a beefed up Fugitive Slave Act as part of the Compromise of 1850 may not be the greatest legacy to leave, but he did do things that were significant. They just may not have been good things.