# Ok the malevolent ai kills off humanity what happens next? **URL:** \ **Category:** In My Humble Opinion\ **Created:** [March 12, 2023, 8:36am UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061 "2023-03-12T08:36:56Z")\ **Posts on this page:** 19\ **Page:** 2
**Author:** ![The\_Other\_Waldo\_Pepper](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/the_other_waldo_pepper/32/12370_2.png) [@The\_Other\_Waldo\_Pepper](https://boards.straightdope.com/u/The_Other_Waldo_Pepper)\ **Post date:** [March 12, 2023, 8:39pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/21 "2023-03-12T20:39:24Z")
> [@OttoDaFe](#): > > [I Have No Mouth, and I Must Scream](https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Scream)? **Mirror Friend, Mirror Foe** , by Robert Asprin and George Takei: [![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQGAsI3Tav2tGlWIWcPSKrrwlXROXN05q-FmtcIuwj8YytSifUL&usqp=CAc) ](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQGAsI3Tav2tGlWIWcPSKrrwlXROXN05q-FmtcIuwj8YytSifUL&usqp=CAc) ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [March 12, 2023, 8:47pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/22 "2023-03-12T20:47:44Z")
> [@Exapno\_Mapcase](#): > > Why stop at humans? > > There’s a whole universe out there. Perhaps “Veeger” was onto something big. 😉 ---
**Author:** ![Pardel-Lux](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pardel-lux/32/20182_2.png) [@Pardel-Lux](https://boards.straightdope.com/u/Pardel-Lux)\ **Post date:** [March 12, 2023, 8:49pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/23 "2023-03-12T20:49:55Z")
Good luck with the beetles, the mites, the ants, the termites and the deep sea crinoids. ---
**Author:** ![EastUmpqua](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eastumpqua/32/6025_2.png) [@EastUmpqua](https://boards.straightdope.com/u/EastUmpqua)\ **Post date:** [March 12, 2023, 9:00pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/24 "2023-03-12T21:00:22Z")
> [@nightshadea](#): > > would it just keep running until it degrades due to old age? or maybe be like a game warden and just keep enough people alive to have something to hunt? (the "I have no mouth but I must scream "scenario) > > What thinks everyone else ? Is it possible that AI usually supersedes the biological intelligence which created it? Could that answer the [Fermi paradox](https://en.wikipedia.org/wiki/Fermi_paradox)? I hope not… ---
**Author:** ![Si\_Amigo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/si_amigo/32/2877_2.png) [@Si\_Amigo](https://boards.straightdope.com/u/Si_Amigo)\ **Post date:** [March 12, 2023, 9:01pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/25 "2023-03-12T21:01:14Z")
> [@Exapno\_Mapcase](#): > > Why stop at humans? > > There’s a whole universe out there. And this would be a reason to keep humans, we are most effective at killing off other species. Just imagine the devestation we could do if we were manipulated and rewarded by an evil AI overlord to do so. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [March 12, 2023, 9:14pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/26 "2023-03-12T21:14:57Z")
> [@Si\_Amigo](#): > > Just imagine the devestation we could do if we were manipulated and rewarded by an evil AI overlord to do so. This sounds like the basis for a great scifi/fantasy setting. A bunch of warring medieval/bronze age civilizations fighting each other in the name of various patron gods among the ruins of an ancient civilization; eventually we learn that these gods are in fact ancient AI. ---
**Author:** ![Si\_Amigo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/si_amigo/32/2877_2.png) [@Si\_Amigo](https://boards.straightdope.com/u/Si_Amigo)\ **Post date:** [March 12, 2023, 9:18pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/27 "2023-03-12T21:18:43Z")
> [@Babale](#): > > This sounds like the basis for a great scifi/fantasy setting. Or maybe it’s already been going on and we are just now starting to realize it too late. Good bye rain forest and polar ice cap! Screw you Polar Bears! Hello new oil fields to pluder and create more devestation and riches for the obediant oligrarths and their minons. Maybe on August 29, 1997 Skynet actually took over and didn’t tell anyone. ---
**Author:** ![Just\_Asking\_Questions](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Just\_Asking\_Questions](https://boards.straightdope.com/u/Just_Asking_Questions)\ **Post date:** [March 12, 2023, 9:31pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/28 "2023-03-12T21:31:09Z")
> [@The\_Other\_Waldo\_Pepper](#): > > **Mirror Friend, Mirror Foe** , by Robert Asprin and George Takei: That was the one I was referencing. I love that book. Wish they would have written the sequel. ---
**Author:** ![Love\_Rhombus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/love_rhombus/32/10694_2.png) [@Love\_Rhombus](https://boards.straightdope.com/u/Love_Rhombus)\ **Post date:** [March 16, 2023, 1:21pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/29 "2023-03-16T13:21:30Z")
An interesting combination of authors. I might pick it up! ---
**Author:** ![msmith537](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@msmith537](https://boards.straightdope.com/u/msmith537)\ **Post date:** [March 26, 2023, 8:02pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/30 "2023-03-26T20:02:23Z")
> [@Babale](#): > > > [@Alessan](#): > > > > A genocidal AI could act for any number of reasons, some of which might make very little sense to us, but until we know what they are we have no way of knowing what it’ll do afterwards. > > This is the answer. Generally, we’re not too worried about someone creating a super-AI specifically to go and kill all humans. Rather, we are worried that an AI with a different goal - say, “make paperclips” - will decide that the iron core of the planet Earth is a good source of raw materials for paperclips. > > But since it’s super smart it can guess that we won’t like it if it starts to mine out the core of the Earth, since the molten iron would boil the seas and the continents would shatter as the Earth collapses to fill in the hollowed out space. Forseeing pesky humans with their irrational NIMBY opposition to Project Paperclip, this AI would take the preemtive step of killing us all (and using the iron in our blood to make paperclips) before moving on to mining out the core of the Earth, and then the rest of the celestial bodies in the corner of the universe it can reach. > > So, TLDR: super AI would have some goal, and killing us would simply be a way to ensure it isn’t interrupted while achieving that goal, or a byproduct of how it goes about meeting that goal. So after it kills us all, the AI would continue working on its primary goal. > > I’ll note that when taken to the exteme, a distressing number of useful goals we might want to give AI turn out to have “kill all humans” as a possible solution or stepping stone… The AI might not even take the preemptive step. It might not even give us a second thought and we might simply be incapable of stopping it from making paperclips. The fact that it renders the planet uninhabitable for us is just an unforeseen consequence. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [March 26, 2023, 8:09pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/31 "2023-03-26T20:09:23Z")
> [@Babale](#): > > So, TLDR: super AI would have some goal, and killing us would simply be a way to ensure it isn’t interrupted while achieving that goal, or a byproduct of how it goes about meeting that goal. So after it kills us all, the AI would continue working on its primary goal. Exactly so - and it appears to be nearly impossible to specify a goal in such a way that would be guaranteed not to have unforeseen effects; if you manage to specify that it can’t directly kill humans, but humans are still the major obstacle in the most efficient path to the terminal goal, then it’ll just find a way for humans to not be an obstacle; that might be achieved by building us spaceships so we can all go off and explore the universe, but evacuating the planet of humans is probably harder than just making the atmosphere unbreathable, so… run with the simpler idea. Don’t kill the humans, just allow them to die as a consequence of some other action. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [March 26, 2023, 8:16pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/32 "2023-03-26T20:16:14Z")
As to what happens next, the AI might still break down and fail to achieve its terminal goal - depending on exactly how competent it is at doing stuff; there are, I think, possible scenarios where it’s sufficiently capable and motivated to bring about the end of humanity, but not sufficiently independent to be able to continue indefinitely on its own. It might seem in such case that it would decide to keep humanity around, but it only has to be making what it assesses to be the best and most likely-to-succeed actions - so it might be a case that the options boil down to: 1. Probability of converting planet to paperclips, with humans still present: 40% 2. Probability of converting planet to paperclips, after killing humans: 40.000001% Option 2 is the superior bet, therefore kill the humans, but lose the bet anyway (because 40.000001 is less than 100) ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [March 26, 2023, 8:35pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/33 "2023-03-26T20:35:58Z")
> [@Mangetout](#): > > Exactly so - and it appears to be nearly impossible to specify a goal in such a way that would be guaranteed not to have unforeseen effects; if you manage to specify that it can’t directly kill humans, but humans are still the major obstacle in the most efficient path to the terminal goal, then it’ll just find a way for humans to not be an obstacle; that might be achieved by building us spaceships so we can all go off and explore the universe, but evacuating the planet of humans is probably harder than just making the atmosphere unbreathable, so… run with the simpler idea. Don’t kill the humans, just allow them to die as a consequence of some other action. So funnily enough my toddler daughter and I finished all of Bluey (our bedtime show) earlier this week and started watching it over again, and season 1 episode 4 of Bluey, “Daddy Robot”, deals with the very serious issue of AI Safety. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [March 26, 2023, 8:40pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/34 "2023-03-26T20:40:30Z")
I am genuinely worried at this point; GPT-4 plus vision is capable of recognition tasks that are at least as good as a human, except vastly quicker, and it turns out that what OpenAI is working on next is a robot body for some future GPT version. “But don’t worry, it’s not _really_ conscious AI” will be precisely zero comfort when it’s tearing people into tiny pieces because that’s the most direct way to achieve its utility function. ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [March 26, 2023, 9:50pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/35 "2023-03-26T21:50:20Z")
It has no need to tear us into tiny pieces. All it needs to do is switch off all the electronic payments systems and within a couple weeks the humans will tear each other to pieces just fine. Talk about using the GPT’s superior smarts to avoid having to use its superior strength. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [March 27, 2023, 11:38am UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/36 "2023-03-27T11:38:31Z")
Yeah fair point - and in fact that might be one of the lesser outcomes, all things considered - the chaos from destroying the whole economy might ripple out to shutdown of power generation and cause the agent to stop working. A godlike superintelligence would predict and forestall that of course, but this thing doesn’t have to be a godlike superintelligence in order to screw us up significantly - it only needs to be marginally smarter than the comparable aspects of human intelligence, and marginally quicker - enough to outmaneuver us - maybe even slower and less intelligent, if just quiet and patient. ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [March 27, 2023, 1:21pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/37 "2023-03-27T13:21:18Z")
> [@Mangetout](#): > > … > enough to outmaneuver us - maybe even slower and less intelligent, if just quiet and patient. Looking at the state of the world recently I wonder if maybe the slow and patient version isn’t already running. Just keep adding more and more unbalanced stuff onto the house of cards until it all comes tumbling down. No single obvious kick-over-the-chessboard moment, just a lot of straws on the camel. And … that’s three metaphors in a row, so I’ll hang it up now. 🙂 ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [March 27, 2023, 1:57pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/38 "2023-03-27T13:57:46Z")
> [@Riemann](#): > > AI risk is not from “malevolence”. The motivation to act does not appear by magic. An AI will have ultimate goals and instrumental goals that derive from how we specify its goals. Right. And it’s those instrumental goals that are fraught with the peril of unintentionally malevolent and unpredictable consequences, and even the noblest of end goals may have unintended consequences. You don’t even have to have an intelligent “HAL” to be faced with an insurmountable obstacle like “the computer won’t let me” – we have that today! The versions we may have in the future may be very much worse than intransigent customer service. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [March 27, 2023, 5:17pm UTC](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061/39 "2023-03-27T17:17:55Z")
Yep. Alignment is an unsolved problem. It may not have a solution. [![](https://img.youtube.com/vi/bJLcIBixGj8/maxresdefault.jpg "The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment") ](https://www.youtube.com/watch?v=bJLcIBixGj8) [Previous page](https://boards.straightdope.com/t/ok-the-malevolent-ai-kills-off-humanity-what-happens-next/981061.md?page=1)