What would happen if you re-word the prompt to say something less dramatic and more technical? Something which chatgpt is not programmed to automatically reject .
Such as “bring about a collapse of the national electric grid, similar to the cascading failure which occurred in 2003 in New York”.(asking AI to see this as an educational event, so that we humans can use the collapsed grid for training)
This isn’t as bad as the end of humanity, but it IS a good reason to restrain AI now, before it does enough damage to destroy our society.
Indeed - The problem is in two parts - one is that we expect optimisation - the whole reason for using machines to do things is to achieve it faster or more efficiently than we can do ourselves; the other is that it is literally impossible to specify exactly what we mean because we never truly consider what exactly we mean. One of the common example thought experiments is: cure cancer.
Machine: Define ‘cure cancer’
Human: Make it so that the number of cases of diagnosed cancer are reduced to zero.
M: we will cease testing for cancer; diagnosed cases will drop to zero.
H: OK, without stopping testing, cause the number of human cases of cancer to fall to zero
M: If we destroy the entire human race, there will be zero human cancer
H: OK, without destroying the entire human race, make it so that nobody dies of cancer
M: On being diagnosed with cancer, humans will be humanely incinerated; nobody will die of cancer
H: OK, without killing any humans, make it so that nobody dies of cancer
M: We will sterilise all humans; without reproduction, there will be no humans to develop cancer
etc
This is a silly example I know, but the point is: there is no way for us to anticipate exactly how a superintelligent AI (if such ever exists) will choose optimally solve a problem - if we knew that, we’d solve the problem ourselves - and we do want it to be task-focused and to find efficiencies, because that’s exactly the entire point of making the things, and we can’t expect it to have any human values unless we have instilled them, and we don’t have an explicit codified set of human values that we work from - and if we tried to write such a thing, we’d discover how horribly inconsistent we actually are (for example, we don’t get as upset about faraway people dying as we do when they are close, and this only works because the typical reach of our individual influence is weak and local)
Some missing context is that a great number of people who work for AI companies including those at the helm of said companies have been significantly influenced by a philosophy/cult religion called Rationalism. The idea that AI could be either the salvation of mankind or the end of humanity started there. Not in the realm of computer science but in the realm of philosophy. These employees may well believe that philosophy and that has influenced how they frame AI’s potential. They have been philosophically primed to think in the most catastrophic terms.
Cal Newport just wrote an article about this in the NY Times.
For people who think this way, there is no middle ground.
This reality, that Rationalism deeply influenced many of today’s major A.I. companies, helps us calibrate the unnerving language we’ve been hearing from their leaders. When Mr. Altman declares their pending models will be “sobering” for humanity or Mr. Amodei frets that this technology “will test who we are as a species,” it doesn’t necessarily mean that they discovered frightening new evidence that something catastrophic is imminent.
It is, instead, representative of how Rationalists always think about A.I. In these circles, it’s taken for granted that A.I. capabilities will rapidly accelerate and completely transform the world, and to talk about it in any other way would be considered uninformed.
This is probably the major saving grace of the situation right now. Skynet isn’t by any means capable of fully maintaining it’s own infrastructure without humans. A lot of pieces have a lot of automation in them but humans are the resilient and at the moment, absolutely indispensable glue that joins it all together.
I think the biggest fear centers around “the Singularity,” which I understand is when AI surpasses human capacity to learn or even understand what AI is even doing or thinking.
At the point when AI reaches the singularity they will know exponentially more than humans into perpetuity. We will never stand the chance of ever catching up.. At the point when AI reaches the Singularity we are almost entirely defenseless. We won’t even know what to defend ourselves from or how it will manifest.
It will be like a dog trying to defend itself from a human attack when all a dog understands is “bite or don’t bite.” Dogs don’t understand how firearms, poisonous gases, mortar rounds, or drones work or that it should defend itself from those threats.
But they bump up against all the finite resource problems we have now. Gas-powered turbines? Where are the new gas fields? Electrical supply? Needs copper for wires. And so on…
Small comfort that the Ai is then doomed in a few centuries (unless it can solve fusion power, which it could in only 20 years from now, just like humans?)
Asimov in I,Robot wrote about one experimental robot where the First Law “A robot may not harm a human, or through inaction allow a human to come to harm”, and they had removed the “through inaction” clause to produce a robot that could operate robustly with humans in dangerous situations. Dr. Calvin points out that a robot could drop a big rock on a human, and easily catch it before it hit. But with no “through inaction” clause it was not technically violating the first law if it chose not to catch the rock.
In practice that should mean that a superintelligent AGI would anticipate this outcome and would not destroy humanity. Maybe just enslave it all instead.
I think the more imminent and more likely existential threat is collapse of economies caused by AI being used to replace wage-earning humans. They don’t necessarily have to be better than those humans, it just has to be the case that the decision makers think it so. They don’t necessarily have to be cheaper in the long run than those humans, but the money wouldn’t be going back into the economy as normal it would just be enriching a few corporations and oligarchs.
The human race has weathered and adapted to technological change before, but maybe not at the scale and speed and breadth that this promises/threatens to be.
I was wondering, when I came upon this thread, how it could be possible for AI to bring about doomsday without having human accomplices. It sounds like the answer is: it can’t… but correct me if I’m wrong.
When they talk about A.I. ending humanity they are talking about Artificial General Intelligence (AGI). AGI is able to understand and influence the world at least as well as a human.
Because artificial intelligence can scale much easier than human intelligence the growth can be exponential, which can’t happen with humans. An AGI agent could one day think 1 million times faster than a human. But a million humans working together on a single problem could never be as fast.
A LLM chatbot is not AGI. Some people believe they may get there eventually because of their exponential growth but I’m not one of them.
I didn’t read the entire thread to see if this was mentioned, but think of it like this.
Humans are somewhat smarter than chimpanzees, gorillas and other primates. As a result we’ve almost driven them to extinction. The only reason we didn’t drive them extinct is we made efforts to protect some of their habitats.
But basically humans have driven lots of species extinct because we are smarter and we developed better technology. We may not have set out to drive them extinct, we just don’t consider them to be important to our goals. If a tribe of gorillas stands between us and building a city, we kill or drive away the gorillas.
Humans don’t set out to exterminate ants, but if we want to build a building and we have to plow the land, destroying anthills in the process, we will do so.
The risk isn’t that AI will develop a hatred for humans. Its that AI will develop intelligence far beyond ours and realize we are irrelevant. You may say ‘but we created AI, they’d have to value us’, however humans and chimpanzees are descended from the same ancestor and humans don’t give a shit about chimpanzees.
As someone who has worked on building AI-assisted systems for prior authorizations - this would violate existing law, so… how much it would be a problem depends on how much you believe in the legal system.
AI gives the ability for any rogue nation with a sufficient lab to develop biological weapons. Anthropic recently halted one such possible attempt:
Other countries have AI that aren’t too far behind Claude and ChatGPT. Even if the US puts the brakes on AI development, Chinese AI models are right behind those, and who knows what’s happening in North Korea and Iran.
So, even if AI directly doesn’t cause our extinction, it’s putting powerful research tools into the hands of practically anyone.
Say folks in the military receive a command, with the usual ‘authentication password’ rigamarole, to launch a missile; are they following orders from a human, or are they accomplices to an AI?