I’m sure you’re right. But if you look at some of the competitors in the archery events, for example, there’s no way they approach the fitness level of the coxswains. So the bar has already been lowered.
But the question about defining “fitness” is spot on. Unless everyone agrees on what we’re measuring there’s no way to answer the question.
That woman shotgunner that I’ve seen pop up in the news for medalling in consecutive Olympics doesn’t seem “fit” at all. That sport seems like more pure skill than athletic conditioning.
I competed for years in equestrian. My college team went to nationals. I knew riders who competed internationally. If you get up at 6 am you can even see some of these folks. 
One can be a serious rider and carry a little gut (look at say William Shatner, 80+ years old and still an avid rider). Even then, I would not say he wasn’t fit, just less fit. He also wouldn’t qualify for the Olympics.
You will not find an out of shape rider at the international level. You need a lot of leg, back and core strength. Any extra weight is also carried by the horse, and can cause injuries in the sports like cross country, where the horse is jumping large obstacles and cantering for longer distances. It also takes years and years of training to compete at that level. It’s also expensive. Mitt Romney expensive.
I went down the list of sports included in the Summer Olympics to see what likely candidates were. I remain astounded by some of the things we include (badminton, slalom canoeing, and trampoline).
I’m going to vote trampoline, maybe, or weightlifting. Weightlifting because I think you could specialize. So you would have to be in shape, but just for that one lift. That’s my theory. Really I think your best bet is trampoline. Go team trampoline.
The Olympic lifts require speed, co-ordination and balance besides strength.
If a trampolinist screws up, do they get bounced from the team?
Have you looked at pictures of any of the trampoline athletes? They are incredibly fit, as all the gymnast are.
As to weightlifters, until you get to the superheavyweights they are all quite fit by most standards. Yes, they are in shape for lifting (rarely one lift, most do both) but the lifts are full body exercises. They’re probably not the best long distance runners but I’m sure they do a variety of cardio to keep the weight off.
Does Sarah Robles qualify as “fit”?
Yeah, the reality is if you look at the list of Summer Olympic Sports, you really aren’t going to find anything on there that doesn’t require much more fitness than your average citizen possesses. Plenty I’m willing to mock, sure, but they’re all physical sports. So what’s on the lowest end?
Take a look at Morghan King or Kendrick Farris, they both look quite fit.
Robles is a superheavyweight, where carrying extra weight isn’t an issue. All superheavyweights have greater body fat %, but that doesn’t really mean they aren’t fit. It all depends on your definition of fitness but you might be surprised at how those lifters can move. But I could see putting the heaviest weightlifters in the conversation for least-fit Olympians.
Again, to the OP, define fitness and it’ll be easier to get an answer that people can agree on.
She’s fat, but she can also throw 300+ pounds over her head. I would still call that “fit”, if a rather specialized fit
I will quibble with that, though, for two reasons. One, extra bodyweight has to be lifted by the lifter along with the barbell after they’ve gone into their initial squat. I can’t imagine that weighing less isn’t an advantage in the second portion of a lift. Second, competition rules stipulate that in the event of a tie, the lifter with the lower bodyweight wins. As such, heavier lifters need to lift more weight.
Robles won bronze, but the gold and silver medalists were both considerably slimmer.
Very good points, I should have placed some caveats around the statement. Superheavyweights have noticeably higher body fat percentages because cutting to a weight limit has its costs. Where the cost/benefit lies isn’t clear cut and you correctly pointed out where extra weight hurts the lifter.
You haven’t watched world-class golf in a long, long, time, apparently.
Those guys at your local muni aren’t Olympians.
Eh, there’s superheavyweight, and then there’s superheavyweight… All of the women in that competition (at least, the ones that were actually in contention), went over 100kg; Robles is the only one who was over 125kg (something like 143, IIRC). The bronze medalist outweighed the silver medalist by nearly a hundred pounds: I feel like that’s a whole other class of “super heavyweight.”
I’m not a little guy, especially for a coxswain- I’m 5’8".
I competed at 115 pounds from an out-of-season weight of 140, so while I wasn’t particularly strong, I was, I daresay, fitter than some archers, shooters, or (for certain definitions of “fit”) weightlifters.
-Scrappy, NCAA and three-time US National Champion coxswain; current triathlete.
Ah, but some professional golfers aren’t all that fit, either.
The OP asked what Olympic sports could be done by an out-of-shape participant. Golf certainly fits the bill.
And, to back-up Scrappy - my kid is on a rowing team and I can vouch that everyone in the boat has to be in good fitness, even if they don’t have hands on the oar.
Uladzislau Hancharou (Belarus) won the gold medal in trampoline. Here he is performing a routine. I’d say it requires a high level of fitness.
If you are not willing to at least try and be in the best physical shape you can be, how do you expect to compete with those who are? If you are looking for some sort of shortcut to Olympic glory you are probably not ready to compete, mentally, with the best in the world - regardless of the sport.
That’s an awful list. Kevin Stadler, the one guy who could be considered fat and out of shape hasn’t been a member of the PGA tour for a couple of years. But even if he were, half of the rest of the list are Senior Tour (50+) golfers, and the others are listed by weight, which is not a reflection on fitness. If so, most of the US Men’s basketball team wouldn’t qualify, nor wrestlers, or athletes in track and field, volleyball, various martial arts, among others.
You ain’t kidding; as far as I can tell, the Americans who took silver and gold yesterday in the shot put each weigh more than anyone on that list – and, as far as I can tell, it would be a big mistake to mess with either of those guys.
Baseball is coming back in 2020, though. 
An overweight 58-year old Brit with a hip replacement just won Gold in show jumping - oldest Brit since 1908.