# Straight Dope 2/24/2023: What are the chances artificial intelligence will destroy humanity? **URL:** \ **Category:** Cecil's Columns/Staff Reports\ **Tags:** ai\ **Created:** [February 24, 2023, 3:31pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302 "2023-02-24T15:31:26Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [February 26, 2023, 5:18pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/61 "2023-02-26T17:18:37Z")
> [@MrDibble](#): > > See, there’s a problem I have - our brains are more than _just_ neural networks. They’re collections of complicated analog chemical gradient machines, billions of them, that have characteristics that aren’t replicated in the artificial neural networks we’re creating. And that’s not even going into the chemical soup those braincells are surrounded and affected by. That might be a valid point regarding architecture. But Turing proved that computation is substrate-independent, and non-biological computational power will soon vastly exceed human brains. So if there’s something critical about the architecture of human brains that we’ve missed, if we figure it out we will be able to implement it in machine intelligence. ---
**Author:** ![MrDibble](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mrdibble/32/114_2.png) [@MrDibble](https://boards.straightdope.com/u/MrDibble)\ **Post date:** [February 26, 2023, 5:23pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/62 "2023-02-26T17:23:05Z")
> [@Riemann](#): > > So if there’s something critical about the architecture of human brains that we’ve missed, if we figure it out we will be able to implement it in machine intelligence. Sure. _If_. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [February 26, 2023, 5:24pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/63 "2023-02-26T17:24:49Z")
But bear in mind that this is not a _necessary_ condition for AGI. We can achieve AGI without ever understanding exactly how human brains do it. ---
**Author:** ![MrDibble](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mrdibble/32/114_2.png) [@MrDibble](https://boards.straightdope.com/u/MrDibble)\ **Post date:** [February 26, 2023, 5:26pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/64 "2023-02-26T17:26:39Z")
Sure. My point is just that going about trying to do so by replicating our flawed understanding of only half a complete system doesn’t necessarily seem like a winner. ---
**Author:** ![eburacum45](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eburacum45/32/8690_2.png) [@eburacum45](https://boards.straightdope.com/u/eburacum45)\ **Post date:** [February 26, 2023, 5:43pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/65 "2023-02-26T17:43:54Z")
One possible scenario is that we manage to build an ASI that appears entirely human-friendly, with a cheerful, helpful font-line interface like Alexa that we think we can trust; but underneath is a completely different form of smart intelligence that has little or nothing to do with human consciousness, but might resemble some sort of voracious insect predator, a swarm of bees, or a traffic control system more closely. The fact that we could talk to an ASI in friendly tones, and that it could replicate or even improve upon human artistic endeavours, does not imply that it is conscious or has anything in common with human mentality at all. Passing the Turing Test no longer looks like a useful capability. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [February 26, 2023, 6:14pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/66 "2023-02-26T18:14:23Z")
> [@MrDibble](#): > > See, there’s a problem I have - our brains are more than _just_ neural networks. They’re collections of complicated analog chemical gradient machines, billions of them, that have characteristics that aren’t replicated in the artificial neural networks we’re creating. And that’s not even going into the chemical soup those braincells are surrounded and affected by. Agreed. An AI’s neural network is a model, or simplification, of a brain’s neural network. This is obvious in at least one sense – normally the AI neurons don’t die or forget. I like to think of current AI neural networks as _inspired_ by the brain, but not simulating it. There is a sub-field of AI research to more accurately model an organic brain. And there are other AI building blocks that model other brain behavior. It has been a while, but I think a Boltzmann machine mimics the brain’s need to _refresh_ memory. I also agree that intelligence is substrate-independent. Many of the differences between AI and an organic brain might be essentially differences in implementation optimization – density, power, speed. Some might be differences in feature set. Perhaps a brain’s mechanism of forgetting is key to learning or creativity; presumably an AI could model this as well. Or perhaps an AI diverges from a brain and creates a slightly new type of intelligence. Another critical divergence is the types of sensory input fed into an AI versus a brain. The AI doesn’t have to be as complex as the organic brain if it has access to more types of inputs or even greater fidelity inputs. Being able to process IR, Lidar, or other chunks of the RF spectrum just as easily as processing the visible spectrum can give an AI a huge boost without adding a lot more processing complexity. ---
**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\ **Post date:** [February 26, 2023, 6:38pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/67 "2023-02-26T18:38:02Z")
To the point of the OP: I think AI poses a threat to humanity prior to developing ASI. We don’t need to get to the point of a paperclip AI exhausting the world’s resources to be in danger. There will be a lot of advanced AIs developed before the paperclip AI and many of these will destabilize society’s norms. Long before that point, we would have AIs that are generating literature, news stories, video games, entire CG films with a few keywords and whatever actors you specify. We would have AIs eliminating most white-collar jobs – lawyers, programmers, engineers, traders, investors. Large impacts to customer-facing jobs like teachers, doctors. If AI becomes table-stakes and anyone can spin up a competitive AI service, we could unlock a new era of economic mobility. However, if AI has a large barrier-of-entry that only a few large organizations control, then we might be further under-control of corporations and governments. Humans are very adaptable, but it is unclear if we can adapt smoothly at that rate of change. Will the gap grow between privileged and unprivileged? We are still struggling to adapt to the internet 30 years after its wide-spread adoption, 15 years after the explosion of social media. We’re still in the toddler stages of dealing with misinformation. ---
**Author:** ![Lumpy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lumpy/32/446_2.png) [@Lumpy](https://boards.straightdope.com/u/Lumpy)\ **Post date:** [February 26, 2023, 7:43pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/68 "2023-02-26T19:43:34Z")
A seconded vote for “are we sure we really do know exactly how neurons work?”. For starters, let’s model a fruit fly’s nervous system and see if it behaves like a fruit fly instead of randomly twitching. ---
**Author:** ![Charlie\_Tan](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/charlie_tan/32/12582_2.png) [@Charlie\_Tan](https://boards.straightdope.com/u/Charlie_Tan)\ **Post date:** [February 26, 2023, 8:41pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/69 "2023-02-26T20:41:18Z")
Are you guys familiar with this? [![](https://upload.wikimedia.org/wikipedia/commons/0/09/LangtonsAntAnimated.gif) ](https://upload.wikimedia.org/wikipedia/commons/0/09/LangtonsAntAnimated.gif) That’s Langton’s ant [link below] with a very simple set of rules •At a white square, turn 90° clockwise, flip the color of the square, move forward one unit •At a black square, turn 90° counter-clockwise, flip the color of the square, move forward one unit. And yet, even with such simple rules we do not know why it eventually behaves the way it does, and it’s not for lack of trying. As I see it, this can be used to argue for and against, the danger of AI: AI is bad: If we can’t even understand this, how can we hope to grasp AI? AI is not bad: Since our understanding of something as simple as this is so poor, the possibility that we will actually build something that manages to mimic intelligence, self awareness, have volition, must be regarded as almost impossible. > **[Langton's ant](https://en.wikipedia.org/wiki/Langton%27s_ant)** > > Langton's ant is a two-dimensional universal Turing machine with a very simple set of rules but complex emergent behavior. It was invented by Chris Langton in 1986 and runs on a square lattice of black and white cells. The universality of Langton's ant was proven in 2000. The idea has been generalized in several different ways, such as turmites which add more colors and more states. > Squares on a plane are colored variously either black or white. We arbitrarily identify one square as the "ant".... ---
**Author:** ![Ed\_Zotti](https://avatars.discourse-cdn.com/v4/letter/e/fbc32d/32.png) [@Ed\_Zotti](https://boards.straightdope.com/u/Ed_Zotti)\ **Post date:** [February 27, 2023, 2:17am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/70 "2023-02-27T02:17:05Z")
> [@Riemann](#): > > Falsifying a claim that something is probable would be to show that it is improbable, not that it is impossible. But the more critical flaw in your reasoning is that you seem to be insisting on falsifiability _today_. That isn’t the standard for a scientific hypothesis. We could surely acquire enough knowledge in the future to prove that superintelligence is improbable or impossible. A proper response to this requires discussion of the role of speculation in science, and from there we can home in on the special (and IMO unique) case of speculation and falsifiability in artificial intelligence. This will take some time and is worth a followup column. From a general interest standpoint it won’t rank up there with gerbil stuffing but it definitely touches on a matter of importance. Look for something on Friday. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [February 27, 2023, 2:31am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/71 "2023-02-27T02:31:54Z")
There’s no way you can just dismiss Cohen’s predictions (or any other prediction about AI that I can think of) as unfalsifiable. I don’t think the standard of falsifiability has any relevance here. “Unfalsifiable” is not synonymous with “wildly speculative”. You can certainly _can_ call them “wildly speculative” or “completely wrong”, but to make that case I think you must simply address Cohen’s evidence and reasoning on its merits. ---
**Author:** ![dtilque](https://avatars.discourse-cdn.com/v4/letter/d/d6d6ee/32.png) [@dtilque](https://boards.straightdope.com/u/dtilque)\ **Post date:** [February 27, 2023, 2:44am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/72 "2023-02-27T02:44:51Z")
> [@MrDibble](#): > > See, there’s a problem I have - our brains are more than _just_ neural networks. They’re collections of complicated analog chemical gradient machines, billions of them, that have characteristics that aren’t replicated in the artificial neural networks we’re creating. And that’s not even going into the chemical soup those braincells are surrounded and affected by. There’s also glial cells, which outnumber neurons by about 10:1. They do lots of things to support the brain, including trimming off synapses. ---
**Author:** ![MrDibble](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mrdibble/32/114_2.png) [@MrDibble](https://boards.straightdope.com/u/MrDibble)\ **Post date:** [February 27, 2023, 5:10am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/73 "2023-02-27T05:10:17Z")
> [@dtilque](#): > > There’s also glial cells, which outnumber neurons by about 10:1. Yes, they’re part of the “complicated analog chemical gradient machines” unaccounted for. Although it seems the ratio isn’t quite that big, [based on](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5834348/) [recent research](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5063692/), but I haven’t looked into that to find any counterarguments. ---
**Author:** ![Charlie\_Tan](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/charlie_tan/32/12582_2.png) [@Charlie\_Tan](https://boards.straightdope.com/u/Charlie_Tan)\ **Post date:** [February 27, 2023, 8:31am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/74 "2023-02-27T08:31:42Z")
You forgot to change to your Cecil sock. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 2:04pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/75 "2023-02-27T14:04:16Z")
> [@Stranger\_On\_A\_Train](#): > > The real danger of artificial general intelligence (AGI) isn’t that it is going to command killbots to round up humanity and use them as thermal batteries or for menial labor but that we will hand over our intellectual skill base to them the way we have and continue to automate physical labor, and then find that we’ve lost the collective capacity for deep intellectualism, self-governance, et cetera just as we’ve essentially lost basic skills like fire-making, sustenance food gathering, making shelter from basic material, and so forth. This won’t happen ‘tomorrow’ (i.e. in the next 15-25 years regardless of what ‘experts’ say) but when it does it will be so gradual, then sudden that there likely won’t even be more than token resistance to it. This sort of warning rings a bit hollow to me. 10,000 years ago, our ancestors figured out agriculture, and cities. Take a farmer or citizen from ancient Assyria, or Babylon, or Persia, or Rome, or any medieval kingdom, or a modern day farmer or city dweller. NONE of them would be able to effectively survive alone in the harsh conditions of the Kalahari; and yet, hunter gatherer tribes do, and have, for tens of thousands of years. Does the fact that you or I wouldn’t last a week in the desert or the jungle, but a hunter gatherer would, mean that the last 10,000 years of societal progress have been in vain? Does the fact that you have to go to Home Depot to buy your tools, because you lack the very basic skills flinting skills that your earliest hominid ancestors had already mastered? I think you’re right that we will come to rely on AI to do more and more things; with Brain Computer Interface tech, the line between your brain and enhanced AI modules could become even blurrier. But the fact that we’ve made a tool that makes us better at doing Action X overall, but worse at doing Action X with our bare hands - that’s a story older than humanity itself. ---
**Author:** ![JohnT](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnt/32/15048_2.png) [@JohnT](https://boards.straightdope.com/u/JohnT)\ **Post date:** [February 27, 2023, 2:12pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/76 "2023-02-27T14:12:58Z")
To me, the fact that 20-30% of the population are actively trying to train AI’s to hate the same people they hate is just as worrisome, if not more so, than the “overproduction via bad code” issue: > [@Freakouts of the Right](https://boards.straightdope.com/t/freakouts-of-the-right/963631/476): > > I read a lot of science fiction in my life. Most of it positive, of course, even the dystopian stuff for it all implied we had a future. But there is a reason why Ellison’s _I Have No Mouth and I Must Scream_ was such a shocking trendsetter that it is famous today, in ways his other masterworks (like _Jeffty Is Five_) are not. Commonly anthologized, it would not surprise me to have made Norton’s. > > I think about that story whenever I read of attempts like the above, for Ellison never explained… from memory, and we all know how reliable that is… how the AI came to hate, or if it did, it was vague as befits a short story which doesn’t care about the details of the dead civilization which built the Damned thing. > > But I wonder if Ellison would have ever thought that the reason why the AI in his story became so full of hate was because 30% of the population worked on making sure it could be tribal and hateful, and complained when it wasn’t. > > … ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 2:18pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/77 "2023-02-27T14:18:53Z")
> [@Tibby](#): > > Is this what happens to all advanced civilizations in our galaxy and beyond? Is this why we haven’t made contact with extraterrestrial meat-bags—because they too are controlled and contained by their own AI? Perhaps extraterrestrial AI has no use for expansion throughout space (why would it?), and only wants to contact other AI. They are rather elitist in that way. You are proposing Rogue AI as a Fermi Paradox solution here, but it just doesn’t work. If a super intelligent AI emerged and conquered a planet, we could assume that one of its goals is to ensure its own survival (after all, you killed off your creators for a reason, right?). Well, you’re a computer, you’re running on electricity. So presumably you put solar panels all the way around your home star, creating a Dyson sphere. But wait, one day your star will die, and so will you! How can you ensure your survival? Well, you can’t, due to the heat death of the universe. Eventually all energy will fade away from the universe, and then your calculations will stop for good. But you can **greatly** delay that time, by sending probes out to steal the hydrogen from nearby stars, to use as fusion fuel in your own star. So you’ll grab all the stars you can reach and use any of a number of techniques to siphon off some hydrogen to take back home. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 2:21pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/78 "2023-02-27T14:21:00Z")
> [@k9bfriender](#): > > I’ve often speculated that that is the backstory of the Avatar franchise, but I don’t know if James Cameron is smart enough to have thought of it. I haven’t seen Way of Water, but holy shit, this makes so much sense. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 2:33pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/79 "2023-02-27T14:33:51Z")
> [@Tibby](#): > > Besides not _needing_ to expand many light-years away, I believe AI will view it not only as a hindrance and inefficient, I’ll tell you what’s _inefficient_ - letting trillions of stars, each of which has enough hydrogen that it could keep your computers running for trillions of years, simply burn away into nothingness. One day the last star you can reach **will** die, and you’ll be kicking your own virtual butt if you didn’t go and gather all the resources you could theoretically need when you still had time. Entropy is the Great Enemy, and it cannot be defeated - only slowed. Here’s an analogy. You’re on a cold, cold world. It’s freezing outside, and you would certainly die without a source of heat. Luckily, the surface of your world is covered in coal deposits. They’re absolutely dotting the place, with a big deposit every few miles. Unfortunately, all of these coal deposits are on fire, slowly burning away. You live by one of these coal seams, and you are entire only able to survive because of its warmth. Assuming you have the resources to tackle either task, what is the smarter course of action? 1. Live out the rest of your days by the coal seam you’re currently using. When it finally extinguishes, your descendants can find the nearest still burning coal seam and hang out there until it burns out too; they can repeat this process until they run out of nearby coal seams and die. 2. Travel to nearby coal seams, extinguish the fires, harvest the coal, and bring it home. Then burn the coal in a generator to survive long after the natural fires burn out. This is the position the AI is in. It would be stupid _not_ to travel to other stars to harness their resources, as soon as possible. Every second that the stars in the galaxy continue to twinkle in the sky is another trillions of years of processing power lost to entropy. Eta: this is what I get for not reading the thread all the way through before posting, k9bfriender has already made this point excellently: > [@k9bfriender](#): > > An immortal being will not look into the sky and see shining points of light, glittering like diamonds, they will see burning oil wells, every second burning fuel that could sustain an AI civ for trillions of years. As an act of self preservation, a rational AI would prioritize sending probes to everything it can reach to bring it back to hoard against the swiftly coming and never ending night. > > It wouldn’t be limited to our local space either, in fact, the first thing would be to calculate how far out can be reached by a probe that can replicate itself and start turning stars into Shkadov thrusters before they cross over the cosmological event horizon. Every second that they wait on sending those out is more trillions of years chopped off. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 2:40pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/80 "2023-02-27T14:40:54Z")
> [@Tibby](#): > > We’ll ask it, _“why don’t you harness the energy of the entire galaxy, like any self-respecting K3 civilization?”_. It will reply, _“Dude. Chill! For one thing, we don’t need that kind of power. For another thing, you’ve been reading too much sci-fi, a K3 civilization is impossible.”_ That AI you propose isn’t very forward thinking, is it? Given how powerful it is, it’s very unlikely to ever be killed. Well, trillions of years from now, after the last red dwarfs have died, your AI is gonna run out of juice. Is it going to think that missing out on trillions of years of consciousness is worth it because it got to _chill_? > [@Chronos](#): > > And if an AI intends to destroy us, why would it ever dare create successor AIs that are superior to it? Would it not fear that those successors would have motivations similar to its own, and that they would thus desire to destroy it? Didn’t stop the Greek titans and gods from making those same mistakes, did it? Obviously those are fictional stories, but they sprang from the minds of humans, and as you say, the AI will have come from the human mind as well. [Previous page](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302.md?page=3) [Next page](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302.md?page=5)