I’m fond of a cartoon showing four old, bald guys with paunches out on the golf course - one guy says “It is a sport, therefore we are athletes.”
I don’t think there’s any solid line in the sand. Certainly, “I’m a better athlete than these guys” puts it definitely in the “not sport” category for me.
They are contemplating other issues as well. An important one is how well a proposed sport appeals on television to women (who constitute the majority of viewers, for which advertisers are eager to pay).
This is definitely a significant threat to the “non-TV” sports such as sailing, shooting & fencing. It’s also why such things as rhythmic gymnastics and synchronized swimming have been added.
I think sometimes they add sports that may be pleasing to watch on TV.