As mentioned before, it’s the IOC who picks who it accredits as an independent olympic committee. (Besides, in theory, and in many free-market jurisdictions for real, the IOC/NOC are NGOs, private organizations, not government ministries).
Through half the exitence of the Modern Olympics, better than half the whole world was a colony, protectorate, dominion, mandate or unincorporated territory of a handful of metropolitan powers. The IOC started long ago recognizing separate olympic status to different “countries”, in the sense that is conveyed by the term pays in the IOC’s official French – a place with a distinct geographical space and culture, not necessarily a sovereign nation-state. The tradition is this is purely a matter between the IOC and the NOCs, as long as the Metropolitan Power’s NOC (or government) did not object AND have the means to make an objection hurt. What they generally do NOT do is recognize NOCs from fully integral components of a sovereign state. Thus, the Olympics do not have England/Scotland/Wales teams, as in Euro Football, but one UK team. California would have a hard time getting its own NOC since they are a state in the Congress that passed the U.S. Amateur Sports Act, that grants USOC a protected monopoly in the US of A.
In the specific case of those of us in the US Overseas Territories, the legal fact is that politically, it was the USA’s own Supreme Court (Downes v. Bidwell), that decreed that PR, Samoa, Guam, the Phillippines pre-WW2, etc. were/are “Non-incorporated Territories, belonging to, but not an integral part of the United States” (simplified paraphrase) (IOW, we are “a nation, divisible”) . Yes, we are US Sovereign territory and US citizens, but since we’re not legally a fully integral part of the “USA” polity, we’re eligible for separate NOC status.
This of course raises the possibility that if PR became a state, we may at least transitionally do a Hong Kong and retain the separate NOCs until the systems integrate completely (wouldn’t you want Carlos Arroyo, Larry Ayuso and Eddie Casiano in Team-USA?)