The first two names that came to mind for me were Lancaster and Gene Kelly.
And he was certainly was known primarily for his bodybuilding before he was an actor.
And, quite well-known for it, too, as I remember.
A no true Scotsman. Weightlifting requires plenty of skills but you exclude all those it requires.
But have any of those mentioned gone to the Olympics?
Pretty sure Geena Davis has.
.
eta: How wrong I was, on two counts:
Geena Davis ALMOST went to the Olympics…
And Jason Stratham did!
He was an Olympic diver before becoming an actor… in the 1988 Olympic Games in Seoul, the Barcelona Olympiad in 1992 and the Olympics in Atlanta four years later.
An inversion of the topic but if you want to hear a funny story, well told, watch this about an actor who was not an athlete but…
He had a few minor movie roles as he was winning Mr Olympia multiple times. The documentary Pumping Iron in 1977 gave him even more exposure.
Also he was cut from a Canadian (pro) football team.
Cantinflas was also a circus acrobat in his earlier career, as his performance in Around the World in 80 Days amply demonstrates.
A similar comment could be made about Errol Flynn. He was famously physically gifted; audiences knew it was really him with the sword, or on the horse, or swinging out of the tree, or sliding down the pirate sail, or whatever.
Of course, he was also famously a colossal dickhole, whereas Lancaster was always a gentleman, so choose your fan alliance accordingly. 
Singer and actor Chris Isaak did some boxing at college. His nose was broken half a dozen times suggesting he wasn’t really the best… but he claims he was never knocked out.
TCMF-2L
Iron Maiden singer Bruce Dickinson claims at 23 years old he was ranked 7th best at fencing (with the foil) in Britain.
TCMF-2L
In his native UK Vinnie Jones was a (limited ability and yet) very successful footballer with his then team winning the FA Cup and him becoming the captain of Wales. That original sporting success would exclude him from the OP.
Unless, in the USA, he’s only known as the actor from films such as Lock, Stock and Two Smoking Barrels, Mean Machine, portraying Juggernaut in the X-Men and TV shows like Elementary and Arrow.
TCMF-2L
Bodybuilding and weightlifting are not the same thing.
Bodybuilders need to lift a lot of weights to get the type of body which wins competitions. But the competitions themselves don’t measure any type of skill, and hence none are required in order to be a successful bodybuilder.
Vinnie Jones strikes me as kind of a tweener. I think I’m probably pretty typical of an American in my knowledge of him. I know Vinnie Jones as a tough-guy character actor. I think I knew he had been a footballer, but I didn’t know he had been a team captain in the pros or had been on an FA Cup-winning team. I certainly had never heard of him as an athlete when he was playing. From an American perspective, he seems like he fits - and probably wins this thread - but it sounds like in the UK he’d be more in the category of Alex Karras - a famous pro athlete who parlayed that fame into a successful acting career.
Even as a professional footballer there was an air of pantomime bad guy to Vinnie. He was a ‘Hardman’ and his role was to intimidate the opposition. Even though football is supposedly a non contact sport. As I said he was very successful with only average talent.
In the UK his notoriety peaked when he (supposedly discreetly) squeezed the testicles of the far more talented (until alcohol abuse scuppered him) Paul Gascogne.
TCMF-2L
Two singers who were talented footballers (soccer) at pro-level in their youth and had to choose between careers in music or football:
Rod Stewart
Julio Iglesias
He’s been known to embellish stories about himself but Sean ‘James Bond’ Connery claims he was so good at football he was offered a contract by Sir Matt Busby (one of the greatest managers) at Manchester United (one of the top clubs.)
TCMF-2L
I believe football (soccer) is very much a contact sport. Not every contact is a foul, some may be, but shoulder to shoulder for instance is not.
Vinnie’s style of contact was frequently beyond what the rules allowed!
TCMF-2L