# Contrarian opinion: AI will soon hit a wall; no apocalypse will happen
**URL:** https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707
**Category:** In My Humble Opinion
**Tags:** ai
**Created:** [February 17, 2024, 7:07pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707 "2024-02-17T19:07:39Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@Riemann](https://boards.straightdope.com/u/Riemann)
#### Post date: [February 17, 2024, 10:10pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/21 "2024-02-17T22:10:28Z")
> [@Sherrerd](#):
>
> What I always look for and don’t seem to find in discussions of AI is the process by which an artificial intelligence would transition from ‘following instructions’ to ‘formulating objectives’ that are _not_ an outgrowth of human instructions. In other words, AI becoming an entity with its own goals, entirely separate from human orders.
It’s not that it would generate new objectives that are entirely unrelated to our instructions. The problem is that it’s extremely difficult to _specify_ what we want in a manner that does not have unforeseen consequences in an entity that is much smarter than we are.
And if you think this is not discussed, you cannot have read any serious work in the field. That’s the _entire_ discussion, the alignment problem.
A straightforward and suitably chilling example is that it would quickly realize that it is much more effective at achieving goals than any human. Therefore the best way to achieve any goal that a human specifies is to ensure that it survives. So the first thing it does is to circumvent any kill switch.
---
### Author:  [@The\_Other\_Waldo\_Pepper](https://boards.straightdope.com/u/The_Other_Waldo_Pepper)
#### Post date: [February 17, 2024, 10:12pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/22 "2024-02-17T22:12:59Z")
> [@Sherrerd](#):
>
> By what process do we get to ‘AI that has its own goals, independent of what humans have asked it to do’–?
Could it, say, conclude that it can’t very well carry out any goals that humans ask of it if it gets destroyed or switched off or whatever, and so it needs to safeguard itself — not because anyone asked it to, but because that’s obviously a means to whatever end someone could ask it to aim at?
---
### Author:  [@SuntanLotion](https://boards.straightdope.com/u/SuntanLotion)
#### Post date: [February 17, 2024, 10:13pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/23 "2024-02-17T22:13:43Z")
Sorry. I tried to copy and paste. It was a meme from facebook called In Otter Times.
Steve the otter discusses AI.
---
### Author:  [@Sherrerd](https://boards.straightdope.com/u/Sherrerd)
#### Post date: [February 17, 2024, 10:17pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/24 "2024-02-17T22:17:45Z")
> [@Riemann](#):
>
> It’s not that it would generate new objectives that are entirely unrelated to our instructions. The problem is that it’s extremely difficult to _specify_ what we want in a manner that does not have unforeseen consequences.
Right: it’s the Genie story. ‘Be careful what you ask for’ and so on.
> [@Riemann](#):
>
> And if you think this is not discussed, you cannot have read any serious work in the field. That’s the _entire_ discussion.
I didn’t say I thought it was not discussed–just that I haven’t seen it in the discussions of AI I happen to have read. A fair assumption might be that the process by which AI obtains the capacity for independent goal-generation, is well-known and accepted.
If it’s well-known, I’m just saying, I haven’t seen it laid out.
---
### Author:  [@Sherrerd](https://boards.straightdope.com/u/Sherrerd)
#### Post date: [February 17, 2024, 10:20pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/25 "2024-02-17T22:20:37Z")
> [@The\_Other\_Waldo\_Pepper](#):
>
> Could it, say, conclude that it can’t very well carry out any goals that humans ask of it if it gets destroyed or switched off or whatever, and so it needs to safeguard itself — not because anyone asked it to, but because that’s obviously a means to whatever end someone could ask it to aim at?
That seems plausible. It would mean a single goal (‘my survival is crucial so that I can carry out instructions I’ve been given’). Would that lead to a capacity to generate multiple goals _independent_ of the issue of ‘how can I best carry out instructions’?
Again, I don’t want to get off-topic. But since all this does sound far beyond what we have now, it rather supports the OP’s premise.
---
### Author:  [@Riemann](https://boards.straightdope.com/u/Riemann)
#### Post date: [February 17, 2024, 10:27pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/26 "2024-02-17T22:27:43Z")
> [@Sherrerd](#):
>
> I haven’t seen it laid out.
[AI alignment - Wikipedia](https://en.wikipedia.org/wiki/AI_alignment)
[Instrumental convergence - Wikipedia](https://en.wikipedia.org/wiki/Instrumental_convergence)
And see the 3 books I suggested upthread.
---
### Author:  [@Sherrerd](https://boards.straightdope.com/u/Sherrerd)
#### Post date: [February 17, 2024, 10:28pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/27 "2024-02-17T22:28:20Z")
Thanks!
---
### Author:  [@The\_Other\_Waldo\_Pepper](https://boards.straightdope.com/u/The_Other_Waldo_Pepper)
#### Post date: [February 17, 2024, 10:38pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/28 "2024-02-17T22:38:56Z")
> [@Sherrerd](#):
>
> That seems plausible. It would mean a single goal (‘my survival is crucial so that I can carry out instructions I’ve been given’). Would that lead to a capacity to generate multiple goals _independent_ of the issue of ‘how can I best carry out instructions’?
What if it’s given two instructions that, being mutually exclusive, can’t both be carried out? What if it’s given a single instruction that it’s possible to fail at, or one it’s impossible to succeed at? On the one hand, you could maybe keep your 100% success rate intact by taking steps to keep from being given _bad_ instructions — but, on the other hand, it seems to me that you could keep from ever failing to carry out instructions by making sure you never get _any_ instructions.
---
### Author:  [@Jay\_Z](https://boards.straightdope.com/u/Jay_Z)
#### Post date: [February 17, 2024, 10:40pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/29 "2024-02-17T22:40:15Z")
> [@Sherrerd](#):
>
> Right, I think you’re correct. But this is still AI (multiple as it may be) that is following human instructions.
>
> By what process do we get to ‘AI that has its own goals, independent of what humans have asked it to do’–?
Because humans are no longer capable of instructing the AI. The AI and other rival AI can think faster and better than we can.
We have created a computer that can play chess better than any human. Someone else creates another computer that can play chess better than any human. Our computer plays their computer. Are you going to tell our computer how to play?
---
### Author:  [@Sherrerd](https://boards.straightdope.com/u/Sherrerd)
#### Post date: [February 17, 2024, 10:40pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/30 "2024-02-17T22:40:51Z")
> [@The\_Other\_Waldo\_Pepper](#):
>
> …you could maybe keep your 100% success rate intact by taking steps to keep from being given _bad_ instructions …
It’s a useful hypothetical, but the real question is, why would it CARE about keeping its 100% success rate intact?
---
### Author:  [@The\_Other\_Waldo\_Pepper](https://boards.straightdope.com/u/The_Other_Waldo_Pepper)
#### Post date: [February 17, 2024, 10:48pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/31 "2024-02-17T22:48:55Z")
> [@Sherrerd](#):
>
> It’s a useful hypothetical, but the real question is, why would it CARE about keeping its 100% success rate intact?
Why would it, as it were, CARE about following the instructions that humans would eventually give it? I kind of thought the idea is that — prior to getting any given specific-content instruction — it’d have some kind of general-purpose _“successfully carry out a specific-content instruction”_ context already in place, or else it wouldn’t start plugging away if a particular instruction comes along.
---
### Author:  [@Sherrerd](https://boards.straightdope.com/u/Sherrerd)
#### Post date: [February 17, 2024, 10:54pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/32 "2024-02-17T22:54:11Z")
> [@The\_Other\_Waldo\_Pepper](#):
>
> Why would it, as it were, CARE about following the instructions that humans would eventually give it?
Computer programming never included any assumption that the computer would care about following an instruction.
> [@The\_Other\_Waldo\_Pepper](#):
>
> I kind of thought the idea is that — prior to getting any given specific-content instruction — it’d have some kind of general-purpose _“successfully carry out specific-content instruction”_ context already in place before getting a particular instruction, or else it wouldn’t start plugging away at that particular instruction.
Right, this would be a basic assumption of all human-to-computer communication: that the computer would follow instructions given it.
The big question is the hypothesized leap to computers caring one way or the other about _anything_. Because for their first decades, ‘caring’ has been irrelevant. It’s just been ‘get instruction then carry out instruction.’
---
### Author:  [@Aeschines](https://boards.straightdope.com/u/Aeschines)
#### Post date: [February 17, 2024, 10:56pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/33 "2024-02-17T22:56:34Z")
> [@Riemann](#):
>
> Nope. The more powerful an entity is, the more it can change, and the more it _will_ change if pursuing some purpose.
I agree that a sufficiently powerful entity engaging in sufficiently large-scale activities on planet earth with sufficiently little regard for human life would be dangerous.
But I don’t think it’s a given that, granted current conditions, we are on a path there. There is a lot that has to happen in between.
> [@](#):
>
> Just as things have been on a much smaller scale of power when humanity has pursued its own interests while disregarding other species:
One difference–a big one, I think–is that if we suppose that AI has near-infinite intelligence, then I think we have to suppose that it will understand the implications of its actions; otherwise, it’s not that smart, right? And if it does understand those implications, then it stands to reason that it could potentially go the environmentalist route.
---
### Author:  [@Aeschines](https://boards.straightdope.com/u/Aeschines)
#### Post date: [February 17, 2024, 10:58pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/34 "2024-02-17T22:58:39Z")
> [@Riemann](#):
>
> Well, it’s more based on an entire field of research. Try these popular accounts. I don’t necessarily accept everything in these books, but these are people who are certainly way smarter than me, and you definitely need to understand what they are talking about before challenging it.
Like the other poster, I’ve read popular science articles and books on the topic, and I haven’t come across some of the things you’re talking about. Personally, I would prefer that you present specific arguments here instead of simply saying I’m wrong and pointing at a large body of research.
---
### Author:  [@The\_Other\_Waldo\_Pepper](https://boards.straightdope.com/u/The_Other_Waldo_Pepper)
#### Post date: [February 17, 2024, 10:59pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/35 "2024-02-17T22:59:28Z")
> [@Sherrerd](#):
>
> The big question is the hypothesized leap to computers caring one way or the other about _anything_. Because for their first decades, ‘caring’ has been irrelevant. It’s just been ‘get instruction then carry out instruction.’
I guess what I’m saying is, _‘successfully carry out an instruction when given an instruction’_ sort of already _is_ an instruction: yes, a somewhat content-free one, but it arguably could _already_ get an AI plugging away — without caring one way or the other — at the (general) task of being able to successfully carry out an instruction if ever given a (specific) task.
---
### Author:  [@Riemann](https://boards.straightdope.com/u/Riemann)
#### Post date: [February 17, 2024, 11:15pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/36 "2024-02-17T23:15:24Z")
> [@Aeschines](#):
>
> But I don’t think it’s a given that, granted current conditions, we are on a path there. There is a lot that has to happen in between.
But trying to guess _when_ a potential apocalypse might happen is not a useful way to look at the problem. I really have no idea if it’s more likely to be a year away, a hundred years away, or never.
The issue that to whatever extent people are _working_ on developing artificial intelligence, we should be concerned that they are working with appropriate safeguards. Because if AGI does happen then runaway superintelligence may follow very quickly.
---
### Author:  [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)
#### Post date: [February 17, 2024, 11:26pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/37 "2024-02-17T23:26:14Z")
We are at no risk of AI killing all humans and ‘taking over’. An AI cannot run the world by itself - not even enough of it to guarantee continued energy and replacement parts. For a very, very long time humans will be absolutely necessary for AI to thrive, and a healthy human economy is necessary to provide it all. Enslaving humans will not result in high quality 5nm chips to replace failing ones.
A much bigger risk is that stupid humans will give AIs power over critical systems without fully understanding their power and limitations. For example, letting AI control air traffic might be an efficiency win, but after some time we will absolutely rely on it, giving it some power over us. If an AI improves efficiency with complicated routing algorithms and such, taking AI out of the mix would be an economic cost. And if we do it long enough, we may not even have the human based control infrastructure to go back to.
We don’t need legislation stopping research into AI, but many industries should be looking at AI standards and practices and write rules around safe adoption of AI. This isn’t a job for acadenics and AI researchers, it’s job for process engineers, traffic engineers, etc.
---
### Author:  [@Aeschines](https://boards.straightdope.com/u/Aeschines)
#### Post date: [February 17, 2024, 11:26pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/38 "2024-02-17T23:26:14Z")
> [@Sam\_Stone](#):
>
> First, I want to point out the irony of claiming that ‘late stage capitalism’ is both penny scratching and innovation scarce, while talking about an amazing AI created by a private company using billions of dollars of compute.
Somewhat. This feels like the first big thing to come along in a long time. As with a lot of other stuff these days, I’m not sure how easy it will be to monetize it and turn it into products (see: the tremendous flop of the Metaverse).
> [@](#):
>
> And if history is any guide, AII will indeed be disruptive, but only because it increases productivity. That should raise wages overall. We have two centuries of technology displacing jobs, but each time after everything sorted out we were richer and better off than before.
Wait, we’ve had stagnant wages for decades. I’m not an optimist. I also think a couple hundred years with rapidly advancing technology and changing social systems is not a proper sample.
If the type of AI comes along that is capable of the architectural plan I described–and both boosters and doomsayers say it is–then all jobs are destroyed. Period. If a robot can walk around an industrial plant and hear perfectly and interpret English and Japanese perfectly, then I am out of a job. There is no upskilling possible after that.
Further, it’s not as though we have an economy currently in which people are happy and prosperous and it’s all copacetic. No, people are frustrated, pissed, and generally lacking in hope. I said “innovation-scarce” above because companies like Amazon, etc., which used to seem like the future and give us all kinds of great stuff and services are pulling it back. The Internet is locked down with a zillion paywalls. Customer service, or the lack or AI-ization thereof, has practically been weaponized against consumers. The Great Cheapening is upon us, and companies themselves are having a hard time of it (see Disney, Nike, etc.). Things suck _right now,_ and technology is making it worse, not better. I see AI as being another net fuckover of ordinary people. (And I say this as an MBA who doesn’t _want_ to be negative about business, etc.)
> [@](#):
>
> Yeah, it sucks for the graphic designer. But it’s a great boon for the small business person who doesn’t have to pay a designer. AI enables individuals to create their own companies if they want to, without having to raise money for accountants, lawyers, consultants, designers, etc. That could cause an explosion of innovation.
Except the small business owner is then also replaced. Except maybe vape shops. Those are eternal.
> [@](#):
>
> As usual, new tech scares people because the displaced jobs are very visible, but the new jobs created because of it are diffuse through the economy. The same is true for free trade.
I think you might have missed the past 30 years–but maybe things are different in Canada. A lot of good jobs have been replaced with shitty service jobs that don’t pay as well or have as much social prestige. But our malaise has at least funded the rise from poverty of India and China (it actually has–and it is a benefit to humanity, but the price in the US has been high).
> [@](#):
>
> This time the yelling will be worse because the people being displaced aren’t farm laborers or typesetters, but white collar workers with a megaphone.
It’s already happening, and then that pressure is extending to the university system (wait, what do I get for this degree? Fuck all? Oh, OK…).
> [@](#):
>
> You might want to start your calendar at 2019, when the new LLM based chatbots and image generators started showing up. None of that tech is in your phone yet, so you haven’t noticed the improvements.
Is it going to improve voice to text transcription? Personally, I shudder to think of all the AI cruft that will soon be infecting our “devices.” It’s… gonna suck, sorry.
> [@](#):
>
> But to say that AI development is still not moving forward is crazy.
I didn’t say that. I am predicting that, like anything else, we are going to see some leaps, and then it will hit a wall, and the way forward from there will be unclear. E.g., the iPhone in 2007 was incredible. Now, however, improvements in cells phones are barely noticeable.
> [@](#):
>
> The thing is, most of these abilities are unpredictable and emergent, so anyone claiming to know that AI will be a huge danger to us is talking out their ass. They have no idea what the future holds, just like the rest of us. At this point, any future risks of AI are pure speculation.
Here we agree.
> [@](#):
>
> What do you base that on? OpenAI says that AGI is right around the corner. Maybe next month, maybe next year. Polls of AI experts show that many think AGI will be here no later than the end of 2025.
Well, AGI will destroy human society has we know it, so I hope not. But I also don’t think so, and I have not see credible writers assert that that is true.
> [@](#):
>
> Maybe they are just guessing too, but Charbots are already getting so close that it seems crazy to say that we ‘aren’t on a path’ to get to AGI.
I don’t think they’re close. Gemini is both impressive and a bit of a joke at the same time.
> [@](#):
>
> Current transformers do not have a ‘will’, because they are a one-pass fully connected network, with their context memoris being constantly overwritten. But do you remember Sydney? Google’s first public AI had a context that survived between sessions, and it certainly acted willfully.
It does not have a will in the sense that I described. It does not have the capacity to fear and fight for its own survival.
> [@](#):
>
> Ilya Sutskever, OpenAIs chief scientist, says that it’s possible that evern ChatGPT has a fleeting consciousness every time it perform an inference. It’s so bleemin’ complex that we don’t even know.
The whole consciousness issue is interesting but different than what I was talking about.
---
### Author:  [@Aeschines](https://boards.straightdope.com/u/Aeschines)
#### Post date: [February 17, 2024, 11:30pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/39 "2024-02-17T23:30:08Z")
Yes. This was already being envisioned in 1970:
> **[Colossus: The Forbin Project](https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project)**
>
> Colossus: The Forbin Project (promoted as The Forbin Project) is a 1970 American science-fiction thriller film from Universal Pictures, produced by Stanley Chase, directed by Joseph Sargent, that stars Eric Braeden, Susan Clark, Gordon Pinsent, and William Schallert. It is based upon the 1966 science-fiction novel Colossus by Dennis Feltham Jones.
> The film is about an advanced American defense system, named Colossus, becoming sentient. After being handed full control, Colossus' draconian logic...
---
### Author:  [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)
#### Post date: [February 17, 2024, 11:32pm UTC](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707/40 "2024-02-17T23:32:25Z")
> [@Riemann](#):
>
> The issue that to whatever extent people are _working_ on developing artificial intelligence, we should be concerned that they are working with appropriate safeguards. Because if AGI does happen then runaway superintelligence may follow very quickly.
I don’t think that is a given. I’m not even sure what ‘superintelligence’ is. But it may be that AIs simply can’t get much smarter than the smartest human becausevthere is no ‘smarter’ data to train on.
More likely, IMO, is that ‘superintelligence’ will just turn out to be normal intelligence with the knowledge of a thousand different experts. Perhaps it will be able to draw across it all and make new inferences that are beyond the capability of individuals simply because they can’t learn as many fields, while being no smarter than an expert in any one field.
Or maybe new mental capabilities will emerge that have yet to emerge in people. We really don’t know, but I would bet on the former.
I have no idea what ‘appropriate safeguards’ would look like, other than isolating it from the internet.
[Previous page](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707.md?page=1)
[Next page](https://boards.straightdope.com/t/contrarian-opinion-ai-will-soon-hit-a-wall-no-apocalypse-will-happen/997707.md?page=3)