Perhaps as a research project to see if sustainable living is possible, thousands of people sign up, finance it and head off. Assuming they brought what is needed to have a self sustaining base including a way of producing energy and food for the people living there. Some grow rooms for food, and perhaps wind energy or whatever? Either way they have what it takes and the skill sets on site to accomplish this.
What is the line where the authorities step in and stop this?
There are no authorities.
Let’s say, this settlement is in the UK section, which is pretty much exactly the same as the Argentine claim.
Who has jurisdiction?
I can see the Brits sending a small group of soldiers who will easily chase off the people described in the OP, but to where? And who says the UK has a valid claim?
Then the antarctic treaty means nothing, as one has to exercise some authority over that territory to make those rules. Since the Antarctic treaty is taken as valid there must be some authority.
The Antarctic Treaty System says so, which has 55 nations as signatories. However, it appears that the territory claimed by the UK has some overlap with territories claimed by Argentina and Chile.
That’s good for the signatories, but is it valid?
Edit: l think I am starting a geopolitical hijack so I’m going to back out before I do.
I think the proper approach is to make it an international effort. The information gathered would be of benefit to all concerned, and I think that information should be made freely available to all nations.
To be fair, it does appear that there’s a fair amount of contentiousness around some of the territorial claims, as well as various countries not caring to recognize other countries’ claims.
It also appears that there is an area of the continent which is not currently claimed by any nation (Wikipedia says it’s the area from 90 degrees west to 150 degrees west); if the OP’s hypothetical colonizers chose to plant themselves there, I’m not certain which, if any, country would choose to (or would assert the right to) do anything about them.
The settlers might also have the official support of the government of one of the treaty nations. Some of them have historically tried to strengthen their claims by sending colonists to live there year-round, and even to start families there.
The whole point of the Antarctic Treaty is that all the claims are on pause, but not relinquished. How far a group of people could go in colonizing is entirely dependent on whether or not any nation decides what they are doing is illegal by their national laws. Considering how hard it would be to maintain a presence, let alone a self-sustaining one, and considering you could claim such an effort was scientific research, they’d probably be allowed to try and fail, as long as they put sufficient clean-up fees in escrow.