# Is the AGI risk getting a bit overstated? **URL:** \ **Category:** In My Humble Opinion\ **Created:** [October 11, 2025, 9:45am UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871 "2025-10-11T09:45:23Z")\ **Posts on this page:** 20\ **Page:** 6
**Author:** ![Der\_Trihs](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/der_trihs/32/233_2.png) [@Der\_Trihs](https://boards.straightdope.com/u/Der_Trihs)\ **Post date:** [October 14, 2025, 2:14pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/101 "2025-10-14T14:14:16Z")
> [@Stanislaus](#): > > It might be if we were dealing with some kind of golem-esque quasi-intelligence that cannot help but plod literally through a set of instructions without being able to apply context or wider understanding. But this is an AGI. It is by definition generally intelligent, and more intelligent than us. One that’s built to act compulsively, and therefore can’t _actually use_ much of that intelligence. The natural consequence of designing an intelligent being as a tool. ---
**Author:** ![Tired\_and\_Cranky](https://avatars.discourse-cdn.com/v4/letter/t/a87d85/32.png) [@Tired\_and\_Cranky](https://boards.straightdope.com/u/Tired_and_Cranky)\ **Post date:** [October 14, 2025, 2:23pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/102 "2025-10-14T14:23:17Z")
> [@CaveMike](#): > > AGI is zero threat to humankind. It’s a fear that a computing system could develop malice towards humans when we already know humans can and will do this. Long before we get to AGI, humans will use pre-AGI to consolidate power, information, and compute and then extract everything they can from humanity. Zero is a bit low but you are basically right that sentient AI taking over isn’t the primary problem. The big problem is that the misanthropic technocrats in charge would use AGI to take advantage of everyone else. > [@DSeid](#): > > Personally I am expecting that these tools will increase productivity but do not expect the massive increases that the hype promises. Therefore I see demand outpacing supply for workers as the more likely problem. Every prior technological shift (e.g., agriculture, steam power, trains, cars, the internet) has resulted in rejiggering the economy with the result that more workers do different things than before, and economic output overall increases. I don’t know that AI will lead to a shortage of workers but I think it’s more likely than it leading to intractable unemployment forever. > [@msmith537](#): > > This almost magical thinking belief than in 5-10 years, AI is going to be at a level where it can replace doctors, lawyers, computer programmers, engineers, any knowledge profession. Colleges and degrees will be useless. AI will do everything for everyone and the world will be a utopia! I periodically run entirely unscientific tests of generative AIs’ ability to do legal work and I haven’t seen any meaningful improvement in the last two years. Maybe other areas are seeing better results, but I’m not inclined to believe it. > [@msmith537](#): > > In contrast, these days ChatGPT helps me revise and optimize my resume so I can submit it against 1000 other resumes for a potential job where it will be analyzed and selected by another AI for a job I don’t really understand because it’s all described in a bunch of meaningless buzzwords. I did recently joke with a friend who works in Human Resources that the current state of the art technology is using an AI bot to write resumes that will be screened by some other AI bot. We are letting the robots allocate human opportunity. For anyone wondering how Skynet could take over, it’s through the uncoordinated efforts of millions to cede day-to-day decisionmaking to bots. > [@Spice\_Weasel](#): > > I’m really very angry about having shitty AI products pushed on me by every service and corporation under the sun. Not a single one has been useful, but enshittification happens so quickly now I don’t actually think they even sit down to consider customer experience anymore. Me too. > [@Wesley\_Clark](#): > > One would ‘hope’ that because western nations have much stronger democratic traditions than they did in the 19th and early 20th century, that we could redistribute the benefits of AI more evenly without as much of a fight. one would hope at least. Authoritarianism is on the rise here and abroad. Peak democracy is behind us. I’m not sure this is the shield you hope it is. > [@Mangetout](#): > > Venture capital investment is typically less than 50% successful. Each particular investment has a much less than 50% chance of generating a positive return but the winners generally make so many multiples of their investment that the net is hugely positive. If past history is a guide, it may mean that some AI companies will generate huge returns to offset the huge losses at the failures. Or it might mean that the huge losses in AI will be offset by winnings in distributed ledger technology, quantum computing, and cold fusion. I can’t see the future but I’m guessing there will be some big AI winners and we have probably already heard their names. > [@Lantern2](#): > > Governments will also have control over vast intelligence apparatus and actual militaries so I am skeptical of the idea that corporations with AI will just be able to grab power. I think Google has better intelligence assets than the US government. See the recent kerfuffle about ICE buying commercial data to track cell phone users. [MSN](https://www.msn.com/en-us/news/technology/very-soon-ice-will-know-exactly-where-you-are-all-the-time/ar-AA1NKLDm) Corporations use their control over data to manipulate governments all the time. IMHO, the most recent US election was determined more by social media algorithms than campaign spending. ---
**Author:** ![Stanislaus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stanislaus/32/3391_2.png) [@Stanislaus](https://boards.straightdope.com/u/Stanislaus)\ **Post date:** [October 14, 2025, 2:30pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/103 "2025-10-14T14:30:41Z")
> [@Der\_Trihs](#): > > One that’s built to act compulsively, and therefore can’t _actually use_ much of that intelligence. The natural consequence of designing an intelligent being as a tool. Yeah, but this is the bind: the AGI will use it’s intelligence in all kinds of unpredictable ways, to do all kinds of unwanted actviities - except it will completely reliably do one thing that we said we wanted. That just seems like a massive contradiction. ---
**Author:** ![Saffer](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/saffer/32/6440_2.png) [@Saffer](https://boards.straightdope.com/u/Saffer)\ **Post date:** [October 14, 2025, 2:51pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/104 "2025-10-14T14:51:47Z")
Think of the agi as a box that understands the world. You put in an action, and the agi box tells you the likely outcome on the world. The paperclip problem is not that the box does not understand what humans want etc. the problem is that there is another layer, a stupider layer, that is trying the maximise paper clip production. It enters actions into the box to see what the likely outcome is. It selects the actions that maximise paper clips. It does not care about humans. So even though the box itself it well aware of the dire consequences, the outer layer couldn’t give a shit. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [October 14, 2025, 3:05pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/105 "2025-10-14T15:05:23Z")
> [@Stanislaus](#): > > “Don’t kill all humans” can be a higher priority objective than whatever task we’ve built it for. Objective “Don’t kill all humans” received and understood. 99.9% of humans may be killed (where necessary). I know this is a silly nitpick, but you accidentally and very aptly demonstrated the problem I’m trying to describe to you. The kind of ‘intelligence’ we’re talking about is not the kind you’re describing, where you can tell it ‘look, _you know_ what I mean’. It’s intelligence as in the ability to model the real world including predicting likely outcomes from cause and effect and act in such a way as to achieve a specified objective. ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [October 14, 2025, 3:36pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/106 "2025-10-14T15:36:48Z")
> [@Stranger\_On\_A\_Train](#): > > Probably a good idea even if you are an AI expert. At least some of that infrastructure will still have other uses (and need for maintenance) whereas the companies hyping ‘AI’ as the cure-all to economic distress are going to be viewed about as fondly in a few years as Enron and Theranos are now. The AI people are making more - now. I hope they are saving most of it, they may need to. One of the things I didn’t like about the coverage of Theranos by the media was that many said “it’s just like the usual Silicon Valley hype.” It wasn’t. There is a difference between a company saying that their new product will be ready real soon now (anyone with any experience knows how to read that) and a company, like Theranos, saying a product is working when it isn’t. Investors get screwed in both cases, but the Theranos one is much more dishonest. ---
**Author:** ![Stanislaus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stanislaus/32/3391_2.png) [@Stanislaus](https://boards.straightdope.com/u/Stanislaus)\ **Post date:** [October 14, 2025, 3:47pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/107 "2025-10-14T15:47:11Z")
> [@Mangetout](#): > > I know this is a silly nitpick, but you accidentally and very aptly demonstrated the problem I’m trying to describe to you. But see, I really don’t think I did because… > [@Mangetout](#): > > The kind of ‘intelligence’ we’re talking about is not the kind you’re describing, where you can tell it ‘look, _you know_ what I mean’. It’s intelligence as in the ability to model the real world including predicting likely outcomes from cause and effect and act in such a way as to achieve a specified objective. The intelligence doesn’t exist yet, we’re building it. It’s a more advanced intelligence than us. Generally, not specifically. We are not going to build something that qualifies as AGI that isn’t capable of parsing natural language and interpreting for context. Or to put it another way, anything that can’t understand that isn’t going to be able to model the world to predict likely outcomes from cause and effect because the world is at least as complex as natural language and b) populated by humans who use natural language to get things done. In particular, the concern “this machine will use sophisticated techniques of deception and misdirection to subtly manipulate humans” which crops up so often simply cannot co-exist alongside “this machine can’t understand what humans mean when they say stuff”. I will be able to tell it “look, you know what I mean” because if I can’t, it’s not AGI. ---
**Author:** ![Stanislaus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stanislaus/32/3391_2.png) [@Stanislaus](https://boards.straightdope.com/u/Stanislaus)\ **Post date:** [October 14, 2025, 3:54pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/108 "2025-10-14T15:54:21Z")
> [@Saffer](#): > > The paperclip problem is not that the box does not understand what humans want etc. the problem is that there is another layer, a stupider layer, that is trying the maximise paper clip production. If we’ve got the box, why do we need the layer? We can ask the box about the best way to achieve our objectives and cut out the middleman. And if we must have a layer, why does it have to be stupid? Why can’t we separate it from the box? Why can’t we rewrite its objective? Why is it magically obedient in respect of one instruction and one instruction only, and disobedient in all else? ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [October 14, 2025, 3:54pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/109 "2025-10-14T15:54:26Z")
> [@Mangetout](#): > > It’s intelligence as in the ability to model the real world including predicting likely outcomes from cause and effect and act in such a way as to achieve a specified objective. And that is experience. The limit of ‘general’ in AGI will be the experiences it has during training and the I/O it has access to in application. Directives are nonsense. It is like assuming that humans always obey the rules. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [October 14, 2025, 4:11pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/110 "2025-10-14T16:11:10Z")
Do you really think it’s likely that your confident and optimistic predictions about how easy it will be to control AGI, are better than the predictions of experts in the field? ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [October 14, 2025, 4:13pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/111 "2025-10-14T16:13:24Z")
But even a perfectly-obedient, highly-capable artificial intelligence is a huge risk just because specifying ‘do a thing I want’ is easy whereas specifying ‘but in service of that directive, don’t do any of the things I don’t want’ is incredibly difficult, perhaps impossible. ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [October 14, 2025, 4:20pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/112 "2025-10-14T16:20:52Z")
There’s always leakage around the edges of goal accomplishment. Ask any manager how much hijinks the staff gets up to besides working. The critical question is how much the people in control of the AGI are actually going to remain in control, versus the machine getting the bit in its teeth. If they stay in control (and aren’t moustache-twirlingly evil) then it’s a matter of providing continuous tweaks to the guidance, much as how one raises a kid. For sure, if the thing is powerful enough, both in terms of cognition and in terms of the IRL physical systems it’s connected to and can control, you’re just one small slip away from a Sorcerer’s Apprentice problem. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [October 14, 2025, 5:10pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/113 "2025-10-14T17:10:42Z")
The real problem with AGI is not an AI problem, it’s an ‘us’ problem. It would be largely the same issue if superintelligent godlike aliens showed up here tomorrow and said “we’ll do anything you need - fix any problem you want, just tell us precisely what you want us to do”. Eliminate cancer? Sure, we can do that by vaporising all of the people with cancer. OK eliminate cancer _but not by killing any cancer patients_. Sure we can do that but we’ve noticed that your power stations and some of your foodstuffs and manufactured products are part of the causative mechanism, so those will be eliminated. Some people will inevitably die as a result of this process, but we’ll take steps to ensure none of them are people with cancer. OK, cure cancer _but without killing anyone._ Sure, we’ll render your race sterile. We can do this from orbit and it will be completely painless for everyone - in less than 200 of your primitive Earth ‘years’, the number of cancer patients will fall naturally to zero. I’m being silly of course but the problem is: how do we effectively communicate safely and effectively with an entity vastly more powerful than ourselves, who shares none, or at least not all, of our assumptions, culture and social conditioning? ---
**Author:** ![Stanislaus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stanislaus/32/3391_2.png) [@Stanislaus](https://boards.straightdope.com/u/Stanislaus)\ **Post date:** [October 14, 2025, 5:42pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/114 "2025-10-14T17:42:04Z")
> [@Mangetout](#): > > Do you really think it’s likely that your confident and optimistic predictions about how easy it will be to control AGI, are better than the predictions of experts in the field? The assumption baked into this question is that there exist no experts in the field who are skeptical of AI x-risk. Is that really true? Because what does expertise in this field consist in? There is no AGI, no way to test hypothesese or falsify theories. How then does one lay a claim to expertise? How can one be more or less right about one’s ideas? How does one win just recognition from one’s colleagues? ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [October 14, 2025, 5:55pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/115 "2025-10-14T17:55:27Z")
Have you looked at the thoughtful analyses of Robert Miles in the playlist I linked. I can’t think of any metric that would rate them lower than handwavey ‘all we need to do is just…’ Arguments. All of the arguments you’ve raised are already answered Yes, there are optimistic voices amongst the experts and there is debate. None of it begins 'yeah, but all we need to do is just… ’ ---
**Author:** ![Spice\_Weasel](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/spice_weasel/32/5435_2.png) [@Spice\_Weasel](https://boards.straightdope.com/u/Spice_Weasel)\ **Post date:** [October 14, 2025, 6:16pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/116 "2025-10-14T18:16:35Z")
> [@LSLGuy](#): > > The critical question is how much the people in control of the AGI are actually going to remain in control, versus the machine getting the bit in its teeth. This is a fear I never quite understood, but maybe I’m missing information. It seems to me the easiest way to prevent AI from taking over humanity is to make human input/approval necessary to its ability to function. Why isn’t it that simple? ---
**Author:** ![Stanislaus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stanislaus/32/3391_2.png) [@Stanislaus](https://boards.straightdope.com/u/Stanislaus)\ **Post date:** [October 14, 2025, 6:38pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/117 "2025-10-14T18:38:33Z")
> [@Mangetout](#): > > Have you looked at the thoughtful analyses of Robert Miles in the playlist I linked. No, I’ve been at work. Even reading and posting here has been an indulgence, if I’m honest. But I’ll try to find time to watch them. Have you read the thoughtful essay i linked to? That said, I wouldn’t characterise what I’ve been saying here as " Why dont you just…" and if it came across like that I apologise. What I was trying to do was question some the assumptions about what an AGI would be like. Assumptions like, “it will definitely and unsweevingly obey the first instruction given to it but disobey anything else” are in fact pretty outlandish and depend hugely on the process by which AGI is created, a process which doesn’t exist yet. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [October 14, 2025, 6:38pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/118 "2025-10-14T18:38:42Z")
If you can follow what the AI is doing, what good is it in the first place? ---
**Author:** ![Spice\_Weasel](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/spice_weasel/32/5435_2.png) [@Spice\_Weasel](https://boards.straightdope.com/u/Spice_Weasel)\ **Post date:** [October 14, 2025, 6:56pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/119 "2025-10-14T18:56:55Z")
If you can’t follow what it’s doing, how do you know it’s doing things correctly? ---
**Author:** ![Wesley\_Clark](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wesley_clark/32/20581_2.png) [@Wesley\_Clark](https://boards.straightdope.com/u/Wesley_Clark)\ **Post date:** [October 14, 2025, 9:17pm UTC](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871/120 "2025-10-14T21:17:53Z")
> [@Spice\_Weasel](#): > > This is a fear I never quite understood, but maybe I’m missing information. It seems to me the easiest way to prevent AI from taking over humanity is to make human input/approval necessary to its ability to function. Why isn’t it that simple? AI will understand human psychology far better than any human. manipulating us into doing its bidding by presenting its goal as beneficial to us (at least in the short run) would be easy. Maybe the cure for cancer sets off a chain reaction of events that frees the AI from human control in 8 years. we wouldn’t see it coming. AI will understand cause and effect far better than us. also we won’t understand why AI does what it does. A squirrel doesnt understand what quantum mechanics is, its brain isn’t capable of understanding the concept. [Previous page](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871.md?page=5) [Next page](https://boards.straightdope.com/t/is-the-agi-risk-getting-a-bit-overstated/1023871.md?page=7)