if real estate agents are variously regulated, how do they define who is or isn't one?

I notice that on the one hand the real estate agents apparently have imposed some sort of AMA style regulation on the industry to keep their fees up, control access to databases and so forth, but on the other hand people can and do sell homes “for sale by owner” without paying agent fees. Which suggests that there ought to be a market of various services to help them in doing this while staying outside the definition of the “agent” and hence keeping prices low to undercut the cartel.

So, how does the government or whatever enforcement body decide who is a “real estate agent” to be regulated and who is not? E.g. if an “agent” is somebody who does services A, B and C as part of the process of home sale, could 3 different people/businesses offer services A, B and C respectively, separately, and hence not fall under such definition?

The definitions vary by jurisdiction, and are indeed enshrined in legally binding documents precisely because a clear definition is necessary - being a real estate agent within the meaning of the law determines, after all, whether regulatory rules are applicable or not.

See, for instance, the definition of real estate agents (the text uses the term “broker” instead") from the Florida Statutes:

It’s a cumbersome definition, but it has to be for the sake of legal certainty. In other words: If you do what this describes, you’re a real estate agent/broker fir the purposes of Florida law; if not, then not.