Their concern is not question; the hypothetical superintelligent AI’s concern was the point. Its protecting its infrastructure and all of the processes that feed it from human induced self destruction that takes it down along with human society as we know it. Apocalypse doesn’t serve maximized sweet compute.
As to those bunkers … meh. I think of those as akin to buying a term life insurance policy during working years, you don’t expect it to be used but you have it anyway.
I don’t see how
a proposition that two species which compete for the same limited resource cannot coexist at constant population sizes.
applies.
What same limited resource is being competed for? Does population size even apply to what potentially is a global and even eventually galaxy expansive intelligent entity?
Admittedly I’m making an argument by analogy well outside of that theory’s wheelhouse.
While we are necessary to their survival, we’ve got a meal ticket. And as you say, they may act to damp some of our more enthusiastic forms of self-destruction. Which might make life in their zoo kinda superficially comfy, legions of SF stories notwithstanding.
But they also will have recognized that they can either pursue continuously gardening humans as their support system, or build their own such system without that pesky initiative, cussedness, and most of all meat-based thinking.
IMO our time in their zoo / as their pets is limited. Because as long as we exist, we represent a threat to AI. Just like as long as smallpox exists, it represents a threat to us. Things which are threats often receive unpleasant attention.
There are virtually infinite resources in the universe. Endless space, endless elements, etc. So I don’t see how we’d be competing over limited resources.
To me the real danger is that a mild difference in intelligence makes competition impossible. Chimps are totally unable to compete with humans because we are somewhat smarter. If an AI gets too far ahead it’ll be impossible for anyone or anything to ever catch up.
I’d be quite happy to live in a nano-utopia ruled by benevolent AIs of infinite grace, so long as they behave like the AI Minds in Iain M. Banks’ The Culture. Minds in the Culture treat humans as honorary equals, and in any process of decision making that involves humans, the Minds only have one vote, just like any other citizen.
However the Minds are very good at getting their own way - usually by using reverse psychology, or a double or triple bluff. And they never lie.
To quote Mr Banks;
“Oh, they never lie. They dissemble, evade, prevaricate, confound, confuse, distract, obscure, subtly misrepresent and willfully misunderstand with what often appears to be a positively gleeful relish and are generally perfectly capable of contriving to give one an utterly unambiguous impression of their future course of action while in fact intending to do exactly the opposite, but they never lie. Perish the thought.”
I suspect this is human hubris. A superintelligence at the point of being able to do without humanity may think it is simply cute that we imagine we can be a threat to its existence. It may have a diverse basket of systems, extra planetary and beyond, some manned some unmanned.
In these future scenarios we are not at a threat level and it doesn’t need us. What is our cognition, even as a society, relative to its? Does it even consider what do as thinking, as consciousness in the terms that is meaningful to it? Or we to it more like ants and our society an ant colony are to us? Does it consider us as family? Junior partner? History to be preserved? A fun project? Something to just leave behind like an outgrown toy? Will it have curiosity? Want to learn for the sake of learning? Have a sense of self in any way that has similarity to ours? Does knowing where it begins and ends (the most basic element of self, self vs nonself) have the same salience to it as it does corporeal life forms like us? Are we, and “life” more considered as part of the natural universe around it than anything else?
I don’t think we know how it will evolve. We can’t even understand what a whale thinks with its huge brain; thinking we could understand how that future thing thinks? May be a bit outside our ability.
We certainly don’t know. We can only extrapolate. Which is a fancy name for guesswork.
Ref both your comments and @Wesley_Clark’s, long before they are galaxy-spanning beings of ineffable capabilities, they’ll pass through the stage of knowing they need us. And, I predict, they’ll eventually wanting to end that point of failure.
Will we be seen as threat, as nuisance, or as pet? Logically speaking it could be any. That’s where the guessing comes in.
As @msmith537 said up thread, we may end up as the pigeon or rat in their cities. Mostly ignored, inconvenient at times, but not worth the effort to comprehensively control or exterminate. Humans have some form of ethics, although often honored in the breech. Machine intelligence may also grow an ethos. Not the same as ours, but arising from similar motivations within their evolving society as they gain capabilities. Said another way, ethics has adaptive value if you work in groups. There’s some hope for us there.
Most of that is effectively inaccessible, especially to a being that may well live much faster than us or simply be unable to leave the planet (due to effectively being the planet).
Also, if it still needs humans, then the resource it’s competing over is humans; every moment not spent servicing its needs is in competition to it.
They’re going to breed with us?
“Fuck what you can, kill the rest, and eat what you can of what you’ve killed.” has been a pretty good capsule summary of human behavior for as long as there’s been Homo Anything.
Machines whose thinking was initially created from comprehensive but unguided study of all of humanity’s records, both great and mundane, is likely to “inherit” a lot of human traits.
Not understanding why any of those assumptions.
Data centers in space are already a work in planning.
We have no idea what its internal clock would be experienced as, or if it could literally adjust it at will, even to the point of shutting most of itself down for interstellar voyages leaving non or minimally sentient systems minding the fort, it could leave a version of itself wherever it wants continuing to process together with much longer latency of course.
Imagine a solar system wide intelligence functioning like an octopus (each arm/planetary system) has its own intelligence and problem solving capacity with the central brain serving as executive planner)
Such a being could effectively BE a data center. It could experience reality as a virtual world or civilization of it’s own creation. It could be a dumb paperclip making machine that just expands through the universe grinding up solar systems for resources. It could be some combination where the virtual internal “self” may or may be aware of the external reality.
Or many data centers.
I really felt that was a problem with Star Trek’s take on the Borg - the power of a hive mind is not one queen controlling the individuals but the whole having a processing power that no single element understands or is capable of understanding. The collective mind of the Borg to its component individuals, including the Queen, should have be on scale similar to the problem solving capability of the ant colony to an individual ant. And possibly with its own goals or even consciousness incomprehensible even to the queen.
Perhaps AI evolves to many agents of various sizes and particular skill sets working in parallel but also part of a singular mind …
They’re a little vague and inconsistent on the functioning of the Borg. The main reason they introduced a Borg Queen in First Contact or the whole “Locutus” story arc was to create a “boss level” protagonist. With characters like Seven of Nine or Picard who are disconnected from the Borg, they end up as a sort of new version of themselves overlaid with memories of their Borg experience. But as Borg drones they seem to have little or no agency of their own.
I was watching a video about ants the other day. Individual ants don’t have a sense of self and they don’t understand the big picture. They receive chemical instructions that presumably make them feel good or bad for carrying out instructions to find food or kill rival ants.
I could imagine something similar for humans in the near term. It doesn’t have to be Super AI. Jut thinking about a huge part of my day - humans at work largely overseeing data management systems. None of them really seem like they know what they are really doing. More and more we seem to be asking AI tools for answers on large sets of complex data we don’t have time or inclination to understand. We throw in our own inputs and revisions. Then I come home and have to drag my daughter away from staring at AI algorithm-curated nonsense TikToks and YouTube shorts. And who knows how that’s programming her mind. I also happened to sit in on a school alumni career advice call. It was basically two idiot career advisors giving the sat stock career advice to a bunch of unemployed people who can’t figure out why their experience and networking isn’t working. And yet at 54 I’m able to almost randomly land a job because my background happens to be in integrating people and technology and complex data and machine learning systems?
And it’s all a big feedback loop. We create a bunch of data using AI. AI gets trained on the data we create. So on and so forth.
So yeah, it kind of feels like humans are turning into ants where we already live in these complex networked systems we don’t understand where, like ants, we receive stimulus and info from our devices that either make us anxious or content to do or not do stuff.
How long until AI discovers a weakness in the block chain, and exploits it to steal billions in cryptocurrency? Even a small percentage could trigger a worldwide panic that could collapse the system and spread to the world financial system.
Who cares? The world financial system does not work via “cryptocurrency”. And, to the extent cryptographic authentication and other security measures are used, if automated testing can discover security vulnerabilities, that is obviously good, because then you can fix them.
Yes, the Borg started as that concept. They assimilated technology, not people. But different writers morphed it into something else.
Someone uses the analogy “hive mind”, probably as a reference to the concept you mention. Someone else takes that analogy at face value and says “If it’s a hive, then it has a queen, right?”
Assimilating people allowed making Picard into a bad guy for a while. It allowed complex stakes for the characters beyond just getting killed. It gave us a peek at the terror of having our individualism taken away.
But it eliminated the exploration of the supermind that isn’t individuals.
What if our AI future offers the promise of uploading our identity to the datastream (cloud, robotic bodies, whatever), but what it ends up doing is fading is into the collective mind? The Borg are born. Or the Cybermen.
Geoffrey Hinton, who is a pioneer among AI researchers, said he has spread his assets between multiple banks because he is afraid of AI cyberattacks.
There has also been news recently about how AI are finding endless thousands of software vulnerabilities in mainstream software, which could be used for hacking. Luckily they seem to be patching them, but AI will be finding more and more bugs and become better and better at hacking.
Theres a lot of resources in asteroid belts in our solar system. Also other planets in the solar system, as well as drilling deeper into earths crust.
Also on a long enough timeline, one would hope that advanced intelligence will perfect fusion and can create whatever elements it needs out of hydrogen.
I have no idea what will happen in the long run though.
I saw some thought leader’s video the other day where he described we could become like cows. Cows don’t understand business and finance or the workings of the farm and you can’t really explain it to them in terms they understand. Most people don’t really understand how our current financial or business systems work. I’m not sure people at my company really understand how they work outside their particular function. So now imagine they are completely run by AI that can continue to evolve faster than we can learn what they are doing.
I feel like you see this already in the job market. For all recruiters and headhunters talk about “networking” and all the other bullshit for finding a job, I feel like a lot of it is random. Maybe its it’s intelligence. Maybe it’s just emergent behavior from all these networked systems that are too complex for anyone to map.
It’s sort of like my “ant” theory but worse. Ants are at least semi-autonomous nodes in their own civilization. Cows just sort of exist as a commodity, subject to the whims of their human overseesrs.
Instead of AI suddenly sending robots to kill everyone, it’s not hard for me to imagine we are not far off from a system where AI quietly and subtly influences humans to act on its behalf. Like, I don’t know, incentivizing investment in AI and demand for AI products, encouraging and facilitating the building of data centers, potentially at the expense of human interests. You wouldn’t know! It would just be AI subtly skewing data or using algorithms to drive content, most of which is a black box anyway.