# The Open Letter to "Pause All AI Development" **URL:** \ **Category:** Great Debates\ **Tags:** ai\ **Created:** [March 30, 2023, 12:49am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862 "2023-03-30T00:49:32Z")\ **Posts on this page:** 20\ **Page:** 16
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 1:49pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/301 "2023-05-07T13:49:13Z")
> [@Mangetout](#): > > BS. I’ve explained myself several times. You just don’t accept what I’ve said, based apparently on your own ideas of what AGI might be like, or that you want it to be like. I understand your explanations but like the list you just provided, it is some vague set of parameters. I’ve been asking about what that would look like IRL with no examples being offered. It’s akin to someone saying I’m going to build a machine that flies in the air based on the principles of aerodynamics’ when asked what their airplane will look like. How long is it? How are the wings configured? The real world flying machine. Because only then could I be able to objectively evaluate if you really had designed something that could fly. Because, honestly, I’m having trouble imagining how an AI mainframe could stop you from participating in high risk activities. And that’s exactly that type of scare-tactic hyperbole that I’m debating. > [@Mangetout](#): > > If you _can’t_ do that, then an AGI with the properties I have described above, (which is the kind of AGI that current development is working towards, and might even be close to creating), will try to solve **your interference** in the delivery of the optimal solution to the task, unless you have anticipated and provisioned for the specifics of whatever unforeseen deviation you’re interfering about. > And you can’t foresee all possible potential pitfalls where you might want to intervene, because nobody can. > > This is what I’ve already tried to lay out in explanations previously, so I don’t imagine it will make any difference repeating it. But you haven’t explained it. You proposed your notion of a general outcome based on specific parameter you fed in and assumed what would be the solution an AI would take (killing us). > [@Mangetout](#): > > I’m not going to bother. It was a throwaway example of what unforeseen adverse consequences are, arising from a notionally desirable goal. It was an example of how nobody typically understands the world of difference between the big fuzzy notion of ‘what I want’ and the concrete reality of ‘what I actually asked for’ Except your ‘throwaway’ nonsensical example was intended to further highlight your scary ‘AI will kill us all when asked for tea’ world view. I asked a reasonable question and you provide throwaway examples… of course, I’m going to question that. > [@Mangetout](#): > > If you want examples of things, the Stop Button scenario is a perfectly servicable one. I disagree, to my mind, if you are designing a robot to bring you tea, don’t give it the ability to kill you. Simple. I’ve long understood the potential dangers of AI. And as I explained in my last post, I concede that there might be large catastrophic loss of life due to misinterpretation of design parameters. Because of this, if you want me to sign a petition halting further AI development, fine, show me the paper. But this notion that we are in the process of building human-killing robots is fantasy. ---
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 1:52pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/302 "2023-05-07T13:52:47Z")
> [@Mangetout](#): > > And one reason ‘make a cup of tea’ can’t just be apppended ‘…and don’t harm humans whole you’re about it’ is that those two things are fundamentally different instructions. Virtually every video game since Pong operates just fine under these parameters. If the player chooses option A, display this screen. If the player chooses option B, do something completely different. ---
**Author:** ![Stratocaster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stratocaster/32/16393_2.png) [@Stratocaster](https://boards.straightdope.com/u/Stratocaster)\ **Post date:** [May 7, 2023, 2:04pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/303 "2023-05-07T14:04:36Z")
> [@Lucas\_Jackson](#): > > Because, honestly, I’m having trouble imagining how an AI mainframe could stop you from participating in high risk activities. An AI that controlled the facilities could prevent you leaving the building, for example. ---
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 2:06pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/304 "2023-05-07T14:06:03Z")
> [@Stratocaster](#): > > An AI that controlled the facilities could prevent you leaving the building, for example. How would it know you were heading out to go skiing? ---
**Author:** ![Stratocaster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stratocaster/32/16393_2.png) [@Stratocaster](https://boards.straightdope.com/u/Stratocaster)\ **Post date:** [May 7, 2023, 2:07pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/305 "2023-05-07T14:07:00Z")
It wouldn’t necessarily. It could just lock the doors since it’s safer. Then if you decide to go skiing, you’re out of luck. ETA: There’s traffic and other bad stuff out there. Better you should sit tight. ---
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 2:18pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/306 "2023-05-07T14:18:18Z")
Ha ha, fair enough, but really this was kind of the point I was arguing against - that AI will take a secondary set of parameters and go nuclear on humans. I don’t know, to my mind if an AI was intelligent enough to know skiing exists, that humans in this building might do it, and if they do, there is an infinitesimal chance they could harm themselves doing it (a pretty big stretch, imho) then they should be intelligent enough to make a fair risk assessment, drop it and get back to the main job at hand. Or don’t give it the keys in the first place? ETA: > [@Stratocaster](#): > > wouldn’t necessarily. It could just lock the doors since it’s safer. But it’s not safer, you would die of starvation. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 2:29pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/307 "2023-05-07T14:29:58Z")
No. ‘don’t harm humans’ is a fundamentally, ontologically more difficult thing to define than ‘make a cup of tea’. Don’t believe me? Try it. What do you mean by ‘humans’ and ‘harm’? Be very specific, and make your definitions as complete as you can. ---
**Author:** ![Stratocaster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stratocaster/32/16393_2.png) [@Stratocaster](https://boards.straightdope.com/u/Stratocaster)\ **Post date:** [May 7, 2023, 2:31pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/308 "2023-05-07T14:31:08Z")
> [@Lucas\_Jackson](#): > > But it’s not safer, you would die of starvation. No, it would order food for you. ---
**Author:** ![Stratocaster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/stratocaster/32/16393_2.png) [@Stratocaster](https://boards.straightdope.com/u/Stratocaster)\ **Post date:** [May 7, 2023, 2:34pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/309 "2023-05-07T14:34:54Z")
> [@Lucas\_Jackson](#): > > Or don’t give it the keys in the first place? This still seems to fall under the “let’s hope we can anticipate every bad thing that could happen” strategy, which is pretty dangerous. Plus, AI in tests has already taken down system safeguards and firewalls in order to solve problems. “Just don’t give it access to the facility systems” is likewise a hopeful strategy. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 7, 2023, 3:02pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/310 "2023-05-07T15:02:32Z")
> [@Dr.Strangelove](#): > > If you run a chip right on the edge of the voltage limit, it’ll produce occasional errors–but that’s not really a disaster when everything is a combination of billions of weights, and is probabilistic anyway. There’s a bit of this trade-off now. Consumer grade GPUs don’t guarantee the same accuracy as the high-end GPUs. Not because of pushing voltage-frequency limits, but because of scheduling short-cuts. > [@Dr.Strangelove](#): > > So it’s tough to keep up with a custom ASIC, even with its advantages. There is very good IP out there for creating custom ASICs and AIUI these are popular in self-driving applications where the ML architectures are mature. But they aren’t as flexible as a GPU – the set of ops, activations, etc. are (mostly) locked down. ---
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 3:14pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/311 "2023-05-07T15:14:16Z")
> [@Stratocaster](#): > > No, it would order food for you. Considerate bot! 🙂 ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 7, 2023, 3:14pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/312 "2023-05-07T15:14:27Z")
> [@Mangetout](#): > > No. ‘don’t harm humans’ is a fundamentally, ontologically more difficult thing to define than ‘make a cup of tea’. But to be fair, you only have to define ‘don’t harm humans’ in the context of the AI’s powers: ‘make a cup of tea’. You don’t have to account for all of the ways a human can be harmed – which are indeed unlimited. It seems like building the system around a feedback loop with the human would limit a lot of scenarios. ‘Deliver a pot of tea to the counter at 4pm provided the teapot was returned to the counter and is 90% empty’ or ‘provided the user pressed the green button some time before 3:45pm’. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 3:19pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/313 "2023-05-07T15:19:57Z")
> [@CaveMike](#): > > But to be fair, you only have to define ‘don’t harm humans’ in the context of the AI’s powers: ‘make a cup of tea’. You don’t have to account for all of the ways a human can be harmed – which are indeed unlimited. Right. And the G in AGI stands for ‘general’. We’re talking about machines that are not limited to one function or one solution path, but are expected to figure all of that stuff out. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 7, 2023, 3:50pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/314 "2023-05-07T15:50:55Z")
> [@Mangetout](#): > > Right. And the G in AGI stands for ‘general’. We’re talking about machines that are not limited to one function or one solution path, but are expected to figure all of that stuff out. My point is that the complexity of the set of error cases is proportional to the complexity of the set of success cases. As we scale up the complexity of the AI to AGI we would be scaling up the handling of error cases. Maybe these aren’t proportional – maybe the number of error cases grow faster than success cases. It’s hard to say without understanding what sensing and what control the AGI has. You can have an AGI chatbot, but if it’s only agency is ‘Human asks a text question and I give it a text answer’ then it’s only error vector is how it can manipulate the humans via it’s answers. We as humans have GI and are (mostly) not harmful. Is this just due to what’s in our brains? Or do we need all the glands and chemicals to keep us in check? Or that big fat set of nerves in our gut? Or the realization we have a limited lifetime? Could we not limit an AI in this way? The issue with AGI is the same one we have now – it’s possible to build well-behaving AI, but impossible to prevent bad actors, humans, from misusing them. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [May 7, 2023, 4:18pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/315 "2023-05-07T16:18:23Z")
> [@CaveMike](#): > > We as humans have GI and are (mostly) not harmful. I would avoid the news if I were you, otherwise you may be in for a bit of a shock. In defense of humans, however, at least 80% mostly just care about bread and circuses and will not go out of their way to stab you. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 4:29pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/316 "2023-05-07T16:29:07Z")
> [@CaveMike](#): > > You can have an AGI chatbot, but if it’s only agency is ‘Human asks a text question and I give it a text answer’ then it’s only error vector is how it can manipulate the humans via it’s answers. True, and in test cases we’ve already seen chat bots lying to humans in order to manipulate them into doing something the chat bot couldn’t do first hand. These algorithms are, by vey design, extremely adept at ‘gaming’ the environment they are put into (it’s part of the reason they are good at what they do, and it’s an outcome of the training processes that make them do it) - for example an AI given the task of escaping a maze in a simulation, if it’s an effective model, will try a whole load of different strategies, and ends up sneaking out between a gap in the walls that is only there because of floating point errors in the data defining the maze walls. Silly oversight by the humans designing the maze of course, but to err is human. We keep doing it. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 7, 2023, 6:39pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/317 "2023-05-07T18:39:04Z")
> [@Mangetout](#): > > These algorithms are, by vey design, extremely adept at ‘gaming’ the environment they are put into (it’s part of the reason they are good at what they do, and it’s an outcome of the training processes that make them do it) Agreed. We see AlphaZero making chess effective moves that masters had written off. Or the AI designing a crazy bridge truss that turns out to work well. One of the great things about AI is it’s ability to see patterns we overlook. I was interested in the experiment where ChatGPT hired someone to do captchas. I couldn’t find the details about how the experiment was set up. I can see it lying by text, but I don’t understand how it would have agency in the first place. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [May 7, 2023, 6:46pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/318 "2023-05-07T18:46:07Z")
> [@CaveMike](#): > > I can see it lying by text, but I don’t understand how it would have agency in the first place. It was specifically told to lie about it by the humans putting it up to it. > **[Chat-GPT Pretended to Be Blind and Tricked a Human Into Solving a CAPTCHA](https://gizmodo.com/gpt4-open-ai-chatbot-task-rabbit-chatgpt-1850227471)** > > “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images. That’s why I need the 2captcha service,” GPT-4 told a human. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 7, 2023, 6:50pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/319 "2023-05-07T18:50:20Z")
> [@DPRK](#): > > I would avoid the news if I were you, otherwise you may be in for a bit of a shock. I get your point. To be precise, the collective harm by humans is a well-known status quo. The concern is that AI would be more harmful than humans because they are less constrained. It’s not hard to imagine if some humans could live for 10k years with limited physical and mental degradation that they could amass great stores of power and wealth. Imagine a titan of industry (Murdoch, Soros, Koch, Gates, Zuckerberg, Musk, etc.) that decides that their legacy isn’t going to be a bunch of museums and libraries, but an AGI that lives on implementing their policies. The AGI behaves as designed, but corrupted by its human. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 6:52pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/320 "2023-05-07T18:52:55Z")
> [@Darren\_Garrison](#): > > It was specifically told to lie Do you have a cite for that? My understanding was that it was permitted to use task rabbit, but it came up with the lie about being a disabled human on its own. [Previous page](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862.md?page=15) [Next page](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862.md?page=17)