Steady labor “toughens” people only up to a point, after which it starts to break them down. Keep in mind, too, that in the past there were no joint-repair surgeries, no anti-inflammatories, and so forth.
Pre-modern era, the skeletons of common people reveal lives of hard and unremitting labor, with accumulating damage to joints and points. The types of labor engaged in left their mark on skeletons.
Sure, a sedentary lifestyle has a negative side, but so does a lot of physical labor.
But, slightly more directly to the OP’s question:
Way in the past, the skin of the common people aged pretty quickly because they spent a lot of time in the sun without protection. It was the aristocracy/royalty who could either get out of the sun, or purchase parasols, or whatever and they generally looked better in adulthood, although there were exceptions. A king who was outside a lot would wrinkle up just as much as a peasant.
Again, only the wealthy would have access to anything much like gentle soaps or moisturizers which contribute to the relatively youthful appearance of people today. The poor might use olive oil or some sort of tallow-based ointment, but fat was needed for a lot of other things in life so supplies were limited at best.
Hair dye was largely unused by most people - again, the exceptions being the wealthy.
Scars not only from injuries but also from serious infections that had to run their course without modern medical treatment and various diseases were pretty common for everyone. That didn’t help, either.
Nutrition problems no doubt contributed to aging as well. For much of history, even royalty could wind up with scurvy after a bad winter.
Injuries didn’t heal as well - if you broke a bone worse that a hairline fracture good luck getting it set. Injure a joint you were probably screwed not just because they didn’t have surgeries for such things but also because they didn’t have our rehab programs, either. Walking sticks weren’t just fashionable, for a lot of middle-aged people they were necessity.
Dental care sort of sucked, too - losing teeth started relatively early in life, and discolored/rotting teeth were pretty common. Which could lead to problems with eating a good diet.
Basically, there was nothing to ameliorate normal deterioration. For myself, a person pushing 50 in the early 21st century, I get regular dental care, adequate nutrition all year round, was vaccinated as an infant against scarring diseases like smallpox, wash with mild soaps, have cheap and effective moisturizers and sunscreens, medical care for the skin problems I’ve had, the joint injuries I’ve suffered, and I don’t have to labor to the point of degrading my body in order to survive. As a result, if you put my picture next to someone of similar age from 50 or 100 or even 150 years ago I look younger than they do.
Pre-1800 someone 40 years old typically had grey hair in part or in whole, was missing teeth, probably had some significant scarring, sun-damaged skin, and probably some chronic injury in the hands, feet, elbows, knees, wrists, or ankles due to overwork. The wealthier folks maybe a bit less so, but the vast majority showed their years.