# The Open Letter to "Pause All AI Development" **URL:** \ **Category:** Great Debates\ **Tags:** ai\ **Created:** [March 30, 2023, 12:49am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862 "2023-03-30T00:49:32Z")\ **Posts on this page:** 20\ **Page:** 15
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [May 6, 2023, 5:38pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/281 "2023-05-06T17:38:45Z")
> [@k9bfriender](#): > > The same way that, no matter how hard you try, you can’t ensure that your children won’t grow up to be genocidal dictators. Except children grow up being socialized in human ethics, and even those who are inclined to be amoral generally learn to mediate their their homicidal tendencies lest they be excluded from the group or imprisoned for committing crimes. An AGI has no innate or socialized system of ethics or morals, and the default position should be to demonstrate how such a system would be safe if given control of safety critical or wide-scale systems, not to assume that nothing will go wrong until it we actually see it misbehave. Stranger ---
**Author:** ![k9bfriender](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/k9bfriender/32/3283_2.png) [@k9bfriender](https://boards.straightdope.com/u/k9bfriender)\ **Post date:** [May 6, 2023, 5:41pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/282 "2023-05-06T17:41:52Z")
Fair, I’m just saying that if you can’t guarantee that your own child won’t grow up to be Hitler 2.0, how can you guarantee that an AI that you train won’t grow up to be even worse? This is in response to the idea of, “Why don’t we just train it _not_ to kill humans?” ---
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [May 6, 2023, 5:45pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/283 "2023-05-06T17:45:03Z")
> [@k9bfriender](#): > > Fair, I’m just saying that if you can’t guarantee that your own child won’t grow up to be Hitler 2.0, how can you guarantee that an AI that you train won’t grow up to be even worse? Oh, I totally agree…I just think that there is even less reason to think that a powerful AGI will have some innate sense of ethics or can be “programmed” with some set of overriding directives to force it to be benevolent than a human child. Much of the ‘wow factor’ of people who are so impressed with ChatGPT comes from its ability to mimic human responses even though it isn’t that the system ‘thinks’ in any way like a person but because it is using its extensive training data set to produce ‘design patterns’ that are human like. Even though ChatGPT and other current interactive and generative ‘bots are not in any way sentient, they are already fooling us by specific design. A true AGI would conceal its intent by default, just as you and I do not blurt out our inner monologues to everyone we meet (at least, I hope not), and there is really no way to know what its intentions are or force it to perform to some predetermined safety guidelines. And because there is going to be such an impetus to put these AGIs in control of complex, difficult to predict and control systems because they can perform data integration and analysis much faster than a human, the idea that we will just sandbox them until we achieve some hypothetic threshold of ‘safety’ is unlikely without deliberate controls and agreements. Stranger ---
**Author:** ![k9bfriender](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/k9bfriender/32/3283_2.png) [@k9bfriender](https://boards.straightdope.com/u/k9bfriender)\ **Post date:** [May 6, 2023, 5:56pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/284 "2023-05-06T17:56:46Z")
> [@Stranger\_On\_A\_Train](#): > > A true AGI would conceal its intent by default, just as you and I do not blurt out our inner monologues to everyone we meet (at least, I hope not), and there is really no way to know what its intentions are or force it to perform to some predetermined safety guidelines. One of the computerphile videos I watched quite a while ago talked about this. They suggested training the AI in a simulation to see if it goes genocidal, and once it seems stable enough to let out, you make it think that it is still in the simulation and being tested. I don’t know how well that works out, but I thought it was an interesting idea. If I were in charge of the world (and maybe I am an AI being tested to take charge), I would put severe restrictions on AI development. But since I’m not, and even those who nominally _are_ can’t really stop it, it’s more a matter of finding the best way to navigate forward with the reality that AI will be more and more ubiquitous and powerful, and that it will be developed much more rapidly than most people seem to be expecting. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [May 6, 2023, 6:09pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/285 "2023-05-06T18:09:43Z")
> [@LSLGuy](#): > > Far more likely is that the widespread deployment of AI will have fast-moving social and economic consequences that will cause _humans_ to create plenty of human-killing violence This whole post is spot-on @LSLGuy. The dangers from AI are immediate: deep fakes, propaganda, disruption to industries, loss of human know-how, asymmetrical control of the technology. Any pause should study these disruptions. Speculating on AGI is too narrow-focused. We have human general intelligence now, but it doesn’t threaten the world because their are so many design flaws: the circuits wear out, the memory cannot be replicated, the CPU has sensory inputs and chemical inputs that are not just necessary to the compute, but to running the whole machine, and the unit has to self-train for years We’re assuming an AGI is built using the technology and techniques we currently have, but grander. What if we can’t get that density without using tissue? What if we can’t get the density without making the compute lossy? ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [May 6, 2023, 7:16pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/286 "2023-05-06T19:16:09Z")
> [@CaveMike](#): > > What if we can’t get that density without using tissue? What if we can’t get the density without making the compute lossy? We are extremely near the limit of what can be done with silicon transistors right now. (At least, in a “cheap, portable” sense. Maybe your silicon chip AGI could fill a wearhouse, need it’s own power plant, and run in 1/10th realtime.) ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 6, 2023, 7:23pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/287 "2023-05-06T19:23:32Z")
> [@k9bfriender](#): > > One of the computerphile videos I watched quite a while ago talked about this. They suggested training the AI in a simulation to see if it goes genocidal, and once it seems stable enough to let out, you make it think that it is still in the simulation and being tested. > > I don’t know how well that works out, but I thought it was an interesting idea I think his conclusion was that it would be quite a fragile situation since you’d have to conspire to maintain a lie that you’re telling to the thing, and if its smart enough, it would probably figure out the possibility that it’s still in the simulation ---
**Author:** ![Spice\_Weasel](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/spice_weasel/32/5435_2.png) [@Spice\_Weasel](https://boards.straightdope.com/u/Spice_Weasel)\ **Post date:** [May 6, 2023, 7:24pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/288 "2023-05-06T19:24:54Z")
> [@penultima\_thule](#): > > A tech savvy primary schooler can now produce a novel in the style of Hemmingway, and claim it as their own work. Or a “new” manuscript of prize-winning quality is published and nobody knows who is the author. Mmm, no, I don’t think we’re there yet. According to the publishers being spammed with this crap, it’s extremely obvious that it was produced by AI and poorly written. > **[AI-generated fiction is flooding literary magazines — but not fooling anyone](https://www.theverge.com/2023/2/25/23613752/ai-generated-short-stories-literary-magazines-clarkesworld-science-fiction)** > > The stories are bad, and they’re coming in droves. I’m seeing AI being used for brainstorming and even editing by serious writers, but we’re nowhere near AI replacing original written work. ---
**Author:** ![k9bfriender](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/k9bfriender/32/3283_2.png) [@k9bfriender](https://boards.straightdope.com/u/k9bfriender)\ **Post date:** [May 6, 2023, 7:49pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/289 "2023-05-06T19:49:52Z")
That’s the idea, you tell it that it’s not in the simulation anymore, but you leave little clues here and there to make it think that it still is. That this is still part of the test. Kinda like Ender’s Game, but hopefully with less genocide. But I agree it would be a fragile situation. ---
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [May 6, 2023, 8:49pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/290 "2023-05-06T20:49:45Z")
> [@k9bfriender](#): > > ttle clues here and there to make it think that it still is. That this is still part of the test. This is like a child believing that they are being super clever by hiding unwanted food under their plate. Stranger ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 6, 2023, 9:12pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/291 "2023-05-06T21:12:25Z")
Oh, I see. I thought you meant the other way around like you let it out of a simulation inside another simulation to see if it behaves differently. Either way, if it knows anything about humans, it could figure out what gaslighting is, and suspect its happening. ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [May 6, 2023, 9:16pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/292 "2023-05-06T21:16:42Z")
I’m reminded of the scene in the otherwise rather tiresome movie _Colossus: The Forbin Project_.\* The one where the humans conspire while staying carefully un-monitorably offline to oh so manually decommission the nuclear weapons the AI is using to hold humanity hostage. It seems they’ve completely fooled the silly machine and saved humanity. At which point it detonates one of the warheads they thought they’d disarmed and vaporizes all the conspirators and good sized hunk of an Air Force Base. “Don’t try to fool me again.” * * * \* See > **[Colossus: The Forbin Project](https://en.wikipedia.org/wiki/Colossus%3A_The_Forbin_Project)** > > Colossus: The Forbin Project is a 1970 American science fiction thriller film from Universal Pictures, produced by Stanley Chase, directed by Joseph Sargent, that stars Eric Braeden, Susan Clark, Gordon Pinsent, and William Schallert. It is based upon the 1966 science fiction novel Colossus by Dennis Feltham Jones. > The film is about an advanced American defense system, named Colossus, becoming sentient. After being handed full control, Colossus' draconian logic expands on its original nuclear ... ---
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [May 6, 2023, 9:37pm UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/293 "2023-05-06T21:37:53Z")
> [@Mangetout](#): > > Either way, if it knows anything about humans, it could figure out what gaslighting is, and suspect its happening. [It is never a good idea to try to gaslight your superintelligent AI.](https://youtu.be/mvLgvychb18) Stranger ---
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 6:13am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/294 "2023-05-07T06:13:39Z")
> [@Stranger\_On\_A\_Train](#): > > You keep insisting that because no one can provide ‘facts’ (i.e. direct evidence from experience) that all speculation about harms is unfounded and that until there is a demonstrable harm there is no justification for slowing ‘progress’ to consider the implications. I have insisted no such thing - much less "keep’ insisting. I explained that I was having trouble grasping the concept _as presented in this thread_ of individual AI robots turning on their creators. My main responses being to posts by **Mangetout**. I keep asking for an explanation of his scenario, but none has been forth-coming - just white-knighting by you, in which I feel like you misrepresented my position. Maybe I wasn’t clear so let me restate it here; I don’t care one way or another about AI in general, or whether we rush headlong into it, or shelve it for 10 years. But I do completely understand, and agree with the concept of AI unintentionally causing harm to human-kind, ala War Games (movie), for example.Those are mainframe type AI. What I don’t understand is the Stop Button video scenario in which you build a tea fetching robot that is so strong that it would potentially kill you if you tried to press the stop button that you placed inconveniently on it’s chest, ala Blade Runner. From what I’ve read, AGI will probably be software - not robots. But maybe I’m wrong - tell me how you see AI in actual, real world applications. > [@Stranger\_On\_A\_Train](#): > > “Programming” is not a “semantics argument”; I believe it is. Most AI software will be built for a purpose. That purpose will need to be conveyed to the AI. Whatever that form that data transfer takes, it’s convenient enough to call it programming for lay purposes. But, again, I was mainly addressing posts by **Mangetout** in which he clearly indicates that there is some level of programming involved - making tea, stopping humans from smoking, etc. > [@k9bfriender](#): > > > [@Lucas\_Jackson](#): > > > > Can you give an example of how your above example would work in real life? Im getting a lot of pushback in this thread about how AIs will be autonomous and we won’t be able to direct how they think, but I’m failing to see how building a computer that can’t even stay on its main objective would benefit us. > > The same way that, no matter how hard you try, you can’t ensure that your children won’t grow up to be genocidal dictators. The part I was asking for an example of is this: > [@Mangetout](#): > > If you make it ‘don’t harm humans, nor allow humans to be harmed’, then you’ve got a recipe for a thing that will try to eliminate smoking, any kind of dangerous sport, refined sugar, alcohol, etc. Describe how you see the computer, robot, mainframe, whatever and what real world steps it would/could take to realize these actions. ---
**Author:** ![Lucas\_Jackson](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lucas_jackson/32/303_2.png) [@Lucas\_Jackson](https://boards.straightdope.com/u/Lucas_Jackson)\ **Post date:** [May 7, 2023, 6:33am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/295 "2023-05-07T06:33:36Z")
Interesting turn of events, this popped up on a YouTube feed. I’ve never visited this channel before. AI listening in? [![](https://img.youtube.com/vi/4kDPxbS6ofw/maxresdefault.jpg "Genocide Bingo") ](https://www.youtube.com/watch?v=4kDPxbS6ofw) 😂 ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [May 7, 2023, 6:47am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/296 "2023-05-07T06:47:51Z")
> [@Darren\_Garrison](#): > > We are extremely near the limit of what can be done with silicon transistors right now. Not really. Even if lithography improvements stopped tomorrow, we’d have at least 100x headroom over the current state of the art. Almost all AI inference/training happens on GPUs. GPUs are great general-purpose parallel processors, but that means they have a bunch of cruft that is useless for AI. A custom ASIC could be much better. There is a great deal of room to explore reduced precision, and even reduce the reliability of the calculations. If you run a chip right on the edge of the voltage limit, it’ll produce occasional errors–but that’s not really a disaster when everything is a combination of billions of weights, and is probabilistic anyway. Chips can still go 3D. This runs into power issues–you can’t just run a pair of stacked chips at their normal speed. But there are nonlinear effects here, where you can get better than half the performance for half the power. So a stacked chip will be faster under the same power envelope. In the long run, it may be possible to have hundreds of layers (this is already done for flash memory). There are lots of other possibilities. Mostly they haven’t been explored because GPUs benefit from massive investment across all the product lines they support. So it’s tough to keep up with a custom ASIC, even with its advantages. But as process improvements slow down, the advantages will become more compelling. There are also completely different AI architectures than the current one, like spiking neural nets. But those are more speculative. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 9:01am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/297 "2023-05-07T09:01:00Z")
> [@Lucas\_Jackson](#): > > My main responses being to posts by **Mangetout**. I keep asking for an explanation of his scenario BS. I’ve explained myself several times. You just don’t accept what I’ve said, based apparently on your own ideas of what AGI might be like, or that you want it to be like. I’ve been describing AGI that is the likely type to emerge (if at all) from the current direction of travel. The thrust of AI development at the moment is significantly centred on making something that: - can interpret the real world directly without needing humans to intervene, explain or codify the envirnoment - is very task-focused - is able to create a plan of steps to perform the task - is able to optimise solutions by considering multiple options and choosing the most efficient - is able to adapt to changes in the environment, in order to modify the plan and deliver the task, without the need for intervention or adjustment by humans - is able to effect changes in the real world directly without mediation by humans, via robotics, connection to control systems, etc Those properties are all well and fine as long as you can perfectly and unambiguously describe the task you want done, and assuming you have considered and eliminated or mitigated all of the potential pitfalls and adverse outcomes. Except you can’t do that, because nobody has figured out if it’s even possible to do that. If you _can’t_ do that, then an AGI with the properties I have described above, (which is the kind of AGI that current development is working towards, and might even be close to creating), will try to solve **your interference** in the delivery of the optimal solution to the task, unless you have anticipated and provisioned for the specifics of whatever unforeseen deviation you’re interfering about. And you can’t foresee all possible potential pitfalls where you might want to intervene, because nobody can. This is what I’ve already tried to lay out in explanations previously, so I don’t imagine it will make any difference repeating it. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [May 7, 2023, 9:03am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/298 "2023-05-07T09:03:03Z")
> [@Dr.Strangelove](#): > > Even if lithography improvements stopped tomorrow, we’d have at least 100x headroom over the current state of the art. That seems wildly optimistic. > **[Apple’s 5 Nanometer Chip Is Another Signpost That Moore’s Law Is Running Out](https://www.forbes.com/sites/linleygwennap/2020/10/12/apple-moores-law-is-running-out/?sh=3290e2fd529a)** > > Chipmakers depend on ever smaller transistors to drive ever-improved performance. Now, transistors are getting smaller but not better. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 9:08am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/299 "2023-05-07T09:08:01Z")
> [@Lucas\_Jackson](#): > > Describe how you see the computer, robot, mainframe, whatever and what real world steps it would/could take to realize these actions. I’m not going to bother. It was a throwaway example of what unforeseen adverse consequences are, arising from a notionally desirable goal. It was an example of how nobody typically understands the world of difference between the big fuzzy notion of ‘what I want’ and the concrete reality of ‘what I actually asked for’ If you want examples of things, the Stop Button scenario is a perfectly servicable one. You tell the robot to make a cup of tea. If you hit the stop button it will not be able to make you a cup of tea, so it has a reason to prevent you pressing the stop button. The reason is: it wants to make you a cup of tea. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [May 7, 2023, 11:34am UTC](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862/300 "2023-05-07T11:34:16Z")
And one reason ‘make a cup of tea’ can’t just be apppended ‘…and don’t harm humans whole you’re about it’ is that those two things are fundamentally different instructions. I could write half a page of pseudocode describing exactly how to make a cup of tea and whilst it might not be perfect, it would be complete, because the task is finite. Can you even begin to specify exactly what you want the machine to understand by the instruction ‘don’t harm humans’? ‘do this thing’ is inherently easier to get right than ‘don’t do some things of which here is a probably-incomplete list; the rest being somewhere in my head but I know what I mean’ [Previous page](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862.md?page=14) [Next page](https://boards.straightdope.com/t/the-open-letter-to-pause-all-ai-development/981862.md?page=16)