# Straight Dope 2/24/2023: What are the chances artificial intelligence will destroy humanity? **URL:** \ **Category:** Cecil's Columns/Staff Reports\ **Tags:** ai\ **Created:** [February 24, 2023, 3:31pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302 "2023-02-24T15:31:26Z")\ **Posts on this page:** 20\ **Page:** 5
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 3:09pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/81 "2023-02-27T15:09:12Z")
> [@MrDibble](#): > > See, there’s a problem I have - our brains are more than _just_ neural networks. They’re collections of complicated analog chemical gradient machines, billions of them, that have characteristics that aren’t replicated in the artificial neural networks we’re creating. And that’s not even going into the chemical soup those braincells are surrounded and affected by. That’s not necessarily true. A neural network with significantly more “neurons” than there are real neurons in the human brain - say, 8x, or 16x, or 32x, etc - would be able to model a human brain more fully, including multiple properties about the state of each neuron. Our current neural networks haven’t even reached 1x, so we have a long way to go. But, in principal, it wouldn’t be impossible. It’s just an engineering problem. Also, it’s doubtful that the most efficient way for a silicone processor to model a human brain’s function is to literally model all individual neurons with information about multiple properties of each one. But if that’s what was needed, there’s no reason that it shouldn’t be possible. > [@MrDibble](#): > > > [@Riemann](#): > > > > So if there’s something critical about the architecture of human brains that we’ve missed, if we figure it out we will be able to implement it in machine intelligence. > > Sure. _If_. Theoretically, we wouldn’t NEED to. Let’s say that there are some analog properties relevant to the function of the human neuron that mean a brain works just fine in “wetware” but fails in to be modeled in hardware. There is some arbitrary network size that would be sufficient to model the entire brain, with the state of each neuron, _including this mysterious property_. Maybe it’s 2048x or 4096x as many “neurons” as a real brain has rather than 8x or 16x; fine, that’s still just a solvable engineering problem away. Maybe it pushes us back a couple decades, but it isn’t a hard limit. The point is, with something like a NEAT algorithm (a neural network evolution algorithm that changes the network itself as it goes), or a descendant of that method, we don’t NEED to figure out what the missing property of the human brain that we’ve failed to model is. We simply need to get a computer powerful enough to handle a network big enough to model this property, and then we need to let evolutionary processes do what they’ve already done, what we know with certainty they can do. > [@Lumpy](#): > > A seconded vote for “are we sure we really do know exactly how neurons work?”. For starters, let’s model a fruit fly’s nervous system and see if it behaves like a fruit fly instead of randomly twitching. Why would we need to? If we wanted AI to run a fruit fly body, we’d create it the same way nature did: modify it at random, keep the modifications that improve performance, and remove the ones that don’t. So yes, your first gen fruitfly AI will writhe randomly on the ground. Your 5th gen fruitfly AI will writhe towards some fruit. And your 25,000th gen fruitfly AI will fly circles aroud real fruit flies. ---
**Author:** ![Cecil\_Adams](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Cecil\_Adams](https://boards.straightdope.com/u/Cecil_Adams)\ **Post date:** [February 27, 2023, 3:36pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/82 "2023-02-27T15:36:10Z")
> [@Charlie\_Tan](#): > > You forgot to change to your Cecil sock. I assure you Little Ed is not a sock. He is a real individual with a mortgage, wife, kids, etc. However, asking him to do some typing on one’s behalf is, as you can see, a risky business, leading well-meaning parties such as yourself to jump to false conclusions. I have administered a gentle thrashing and am hopeful, if not entirely confident, this won’t happen again. ---
**Author:** ![Cecil\_Adams](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@Cecil\_Adams](https://boards.straightdope.com/u/Cecil_Adams)\ **Post date:** [February 27, 2023, 3:39pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/83 "2023-02-27T15:39:42Z")
> [@Riemann](#): > > There’s no way you can just dismiss Cohen’s predictions (or any other prediction about AI that I can think of) as unfalsifiable. I don’t think the standard of falsifiability has any relevance here. “Unfalsifiable” is not synonymous with “wildly speculative”. > > You can certainly _can_ call them “wildly speculative” or “completely wrong”, but to make that case I think you must simply address Cohen’s evidence and reasoning on its merits. All in due time, my friend. ---
**Author:** ![MrDibble](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mrdibble/32/114_2.png) [@MrDibble](https://boards.straightdope.com/u/MrDibble)\ **Post date:** [February 27, 2023, 4:55pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/84 "2023-02-27T16:55:42Z")
> [@Babale](#): > > NONE of them would be able to effectively survive **alone** in the harsh conditions of the Kalahari; and yet, hunter gatherer **tribes** do I’ve bolded the contradictory bits in your statement. San Bushmen very much rely on each other to survive, they don’t go it alone either. ---
**Author:** ![Lumpy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lumpy/32/446_2.png) [@Lumpy](https://boards.straightdope.com/u/Lumpy)\ **Post date:** [February 27, 2023, 4:58pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/85 "2023-02-27T16:58:42Z")
> [@Babale](#): > > A neural network with significantly more “neurons” than there are real neurons in the human brain - say, 8x, or 16x, or 32x, etc - would be able to model a human brain more fully, including multiple properties about the state of each neuron. \<…\> > > Also, it’s doubtful that the most efficient way for a silicone processor to model a human brain’s function is to literally model all individual neurons with information about multiple properties of each one. > \<…\> > Theoretically, we wouldn’t NEED to. > > Let’s say that there are some analog properties relevant to the function of the human neuron that mean a brain works just fine in “wetware” but fails in to be modeled in hardware. There is some arbitrary network size that would be sufficient to model the entire brain, with the state of each neuron, _including this mysterious property_. Maybe it’s 2048x or 4096x as many “neurons” as a real brain has rather than 8x or 16x; fine, that’s still just a solvable engineering problem away. Maybe it pushes us back a couple decades, but it isn’t a hard limit. > > The point is, with something like a NEAT algorithm (a neural network evolution algorithm that changes the network itself as it goes), or a descendant of that method, we don’t NEED to figure out what the missing property of the human brain that we’ve failed to model is. We simply need to get a computer powerful enough to handle a network big enough to model this property, and then we need to let evolutionary processes do what they’ve already done, what we know with certainty they can do. At some point you do need to actually know what you’re trying to model. At a minimum eliminating the vast, vast phase space of how neurons potentially work for how they actually work eliminates 99+% of the trials your neural network has to be put through. IMHO, you’re being too casual about the supposed power of genetic algorithms to be black boxes out of which useful results magically appear. > [@](#): > > > [@Lumpy](#): > > > > A seconded vote for “are we sure we really do know exactly how neurons work?”. For starters, let’s model a fruit fly’s nervous system and see if it behaves like a fruit fly instead of randomly twitching. > > Why would we need to? If we wanted AI to run a fruit fly body, we’d create it the same way nature did: modify it at random, keep the modifications that improve performance, and remove the ones that don’t. I was simply using fruit flies as a hypothetical test model because they’ve been examined down to the genetic level for decades and for a complex organism are comparatively well-understood. My point was, let’s test your confident assumption that you can successfully model intelligence on a far simpler challenge. If in fact your genetic algorithm fails to converge on even a functioning fruit fly, than clearly one or more starting presumptions were incorrect. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 5:00pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/86 "2023-02-27T17:00:08Z")
> [@MrDibble](#): > > I’ve bolded the contradictory bits in your statement. San Bushmen very much rely on each other to survive, they don’t go it alone either Certainly not for an extended period of time, but I meant that someone native to these areas would be able to survive alone for some time (eg on a hunt, scouting during conflict, etc) whereas a city dweller would struggle to make it even that long. But this doesn’t damn the entire enterprise of coty building. You are correct that hunter gatherers rely on one another to survive in a harsh environment, and this kind of cooperation is a skill that has atrophied to an extent as well. ---
**Author:** ![MrDibble](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mrdibble/32/114_2.png) [@MrDibble](https://boards.straightdope.com/u/MrDibble)\ **Post date:** [February 27, 2023, 5:01pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/87 "2023-02-27T17:01:31Z")
> [@Babale](#): > > would be able Sure, and I’m sure there are ways of kinda-sorta modeling the chemical gradients and the like too. But no-one’s doing it now. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 5:01pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/88 "2023-02-27T17:01:40Z")
> [@Lumpy](#): > > At some point you do need to actually know what you’re trying to model. At a minimum eliminating the vast, vast phase space of how neurons potentially work for how they actually work eliminates 99+% of the trials your neural network has to be put through. IMHO, you’re being too casual about the supposed power of genetic algorithms to be black boxes out of which useful results magically appear. Which designer knew what the genetic algorithm contained in our own DNA was trying to accomplish ahead of time? I’m confident that it can be done, because us sitting hwre disucssing it is proof that _it has been done_. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 5:02pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/89 "2023-02-27T17:02:14Z")
> [@MrDibble](#): > > Sure, and I’m sure there are ways of kinda-sorta modeling the chemical gradients and the like too. > > But no-one’s doing it now. Right, and there is no AGI available now. ---
**Author:** ![Lumpy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lumpy/32/446_2.png) [@Lumpy](https://boards.straightdope.com/u/Lumpy)\ **Post date:** [February 27, 2023, 5:10pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/90 "2023-02-27T17:10:59Z")
> [@Babale](#): > > Which designer knew what the genetic algorithm contained in our own DNA was trying to accomplish ahead of time? > > I’m confident that it can be done, because us sitting here discussing it is proof that _it has been done_. Except that for our DNA to do it’s business, you have to already have a complete functional living cell- the step which in the case of neurons you insisted wasn’t necessary to specify. Now obviously between three and a half and four billion years ago, some iterative process got us from organic molecules to the first fully autonomous bacterial cells, but that almost certainly was a _different_ iterative process than how evolution subsequently acted on living cells. Put it this way: abiogenesis obviously happened, and the development of intelligent organisms obvious happened; but it’s very doubtful the two could have been done as a **single** process. ---
**Author:** ![MrDibble](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mrdibble/32/114_2.png) [@MrDibble](https://boards.straightdope.com/u/MrDibble)\ **Post date:** [February 27, 2023, 5:11pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/91 "2023-02-27T17:11:37Z")
> [@Babale](#): > > I meant that someone native to these areas would be able to survive alone for some time (eg on a hunt, Sure, that’s true, thanks for clarifying. I agree. ---
**Author:** ![eburacum45](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eburacum45/32/8690_2.png) [@eburacum45](https://boards.straightdope.com/u/eburacum45)\ **Post date:** [February 27, 2023, 5:49pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/92 "2023-02-27T17:49:53Z")
> [@Babale](#): > > But you can **greatly** delay that time, by sending probes out to steal the hydrogen from nearby stars, to use as fusion fuel in your own star. So you’ll grab all the stars you can reach and use any of a number of techniques to siphon off some hydrogen to take back home. Moving all that matter around would waste a very significant fraction of the available fuel in those stars. Instead, just pick the longest lived star (a red dwarf) and build your collection swarm around that - a small red dwarf could last for trillions of years. I would suggest building collection swarms around all the nearby red dwarfs, but that could fall foul of the objection raised by Tibby here; > [@Tibby](#): > > If it sends AI colonists many light-years away, how will the travelers remain in-network with the home team, without a great deal of lag? Why send off an intelligence as great as itself, when it could one day return and wage robot wars against it? The restriction imposed by the speed of light in our universe will always lead to paranoia - each Dyson swarm would be worried about all the others are doing, behind their cloak of light year separation. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 6:11pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/93 "2023-02-27T18:11:21Z")
> [@eburacum45](#): > > Moving all that matter around would waste a very significant fraction of the available fuel in those stars And letting them burn away hundreds of lightyears away wastes ALL the energy. > [@eburacum45](#): > > I would suggest building collection swarms around all the nearby red dwarfs, but that could fall foul of the objection raised by Tibby here; > > > [@Tibby](#): > > > > If it sends AI colonists many light-years away, how will the travelers remain in-network with the home team, without a great deal of lag? Why send off an intelligence as great as itself, when it could one day return and wage robot wars against it? > > The restriction imposed by the speed of light in our universe will always lead to paranoia - each Dyson swarm would be worried about all the others are doing, behind their cloak of light year separation. You don’t need to send anything nearly as intelligent as yourself to head to a nearby star, harvest its resources, and send them back; but that probably IS the biggest reason for an AI to avoid doing this. ---
**Author:** ![eburacum45](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eburacum45/32/8690_2.png) [@eburacum45](https://boards.straightdope.com/u/eburacum45)\ **Post date:** [February 27, 2023, 7:27pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/94 "2023-02-27T19:27:22Z")
> [@Babale](#): > > You don’t need to send anything nearly as intelligent as yourself to head to a nearby star, harvest its resources, and send them back; but that probably IS the biggest reason for an AI to avoid doing this. Yeah. Your probe might upgrade itself and build its own Dyson swarm. With hilarious results. At least we can console ourselves with the thought that our AI overlords are just a mistake away from destroying each other by accident. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 7:47pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/95 "2023-02-27T19:47:13Z")
> [@Lumpy](#): > > Except that for our DNA to do it’s business, you have to already have a complete functional living cell- the step which in the case of neurons you insisted wasn’t necessary to specify. Now obviously between three and a half and four billion years ago, some iterative process got us from organic molecules to the first fully autonomous bacterial cells, but that almost certainly was a _different_ iterative process than how evolution subsequently acted on living cells. Put it this way: abiogenesis obviously happened, and the development of intelligent organisms obvious happened; but it’s very doubtful the two could have been done as a **single** process. We aren’t trying to model the entire process though, or even the finished product. We are trying to evolve something new, from scratch, using the same basic principles, to achieve a similar end result. And by doing it virtually, we don’t need to wait billions of years for the process to complete. ---
**Author:** ![Lumpy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lumpy/32/446_2.png) [@Lumpy](https://boards.straightdope.com/u/Lumpy)\ **Post date:** [February 27, 2023, 10:30pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/96 "2023-02-27T22:30:18Z")
> [@Babale](#): > > We aren’t trying to model the entire process though, or even the finished product. We are trying to evolve something new, from scratch, using the same basic principles, to achieve a similar end result. And by doing it virtually, we don’t need to wait billions of years for the process to complete. Except I was under the impression that we were discussing developing artificial intelligence by emulating the function of neurons in brains. If we’re not, then we’re back to trying to create general intelligence from first principles. ---
**Author:** ![Polerius](https://avatars.discourse-cdn.com/v4/letter/p/d78d45/32.png) [@Polerius](https://boards.straightdope.com/u/Polerius)\ **Post date:** [February 27, 2023, 11:39pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/97 "2023-02-27T23:39:08Z")
> [@Babale](#): > > But you can **greatly** delay that time, by sending probes out to steal the hydrogen from nearby stars, to use as fusion fuel in your own star. So you’ll grab all the stars you can reach and use any of a number of techniques to siphon off some hydrogen to take back home. Why do you assume that an intelligence will necessarily have the same drive for survival that we (and all animals) have? We likely have the strong drive for survival because of the way life on Earth evolved. That doesn’t mean that anything that is intelligent will be hell-bent on continuing its existence. (Even some humans decide that it’s better not to continue their existence, and act on that) “I exist, therefore I must do everything in my power to continue existing” is not necessarily a must for something to be deemed intelligent. ---
**Author:** ![Babale](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/babale/32/15666_2.png) [@Babale](https://boards.straightdope.com/u/Babale)\ **Post date:** [February 27, 2023, 11:47pm UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/98 "2023-02-27T23:47:08Z")
> [@Lumpy](#): > > Except I was under the impression that we were discussing developing artificial intelligence by emulating the function of neurons in brains. That’s what we are doing, but that doesn’t mean the final product has to resemble a human brain in form (even if it does in function), nor that it has to follow the same process to develop. ---
**Author:** ![Ssounder](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@Ssounder](https://boards.straightdope.com/u/Ssounder)\ **Post date:** [February 28, 2023, 4:20am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/99 "2023-02-28T04:20:59Z")
Aside from the deep dive function and theory discussions here, my impression is that at least a slight majority of posters in this thread agree that there is a possibly of _some threat to humanity_. Also, evidenced here is that there are many unknowns about the progression of AGI when it finally awakes and acts independently. As stated up-thread, developers were debating whether they should keep it caged, or let it out. Right now, GPTChat and similar AI is being introduced to the masses with sometimes mixed results. We don’t know for sure but it’s probably already out in the wild in some form. Keep in mind the _Internet of Things_ is connected to government, industry and many people’s homes. My unsolicited advice is to err on the side of caution. Instead of being swept up in awe by the technology, perhaps people need to demand that developers put some safety mechanisms in place while we are still the decision makers. ---
**Author:** ![eburacum45](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eburacum45/32/8690_2.png) [@eburacum45](https://boards.straightdope.com/u/eburacum45)\ **Post date:** [February 28, 2023, 10:54am UTC](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302/100 "2023-02-28T10:54:17Z")
> [@Polerius](#): > > “I exist, therefore I must do everything in my power to continue existing” is not necessarily a must for something to be deemed intelligent. Maybe not, but after a significant period of time only entities with such an imperative would continue to exist. If you create an entity without the goal of continued existence, it wouldn’t last long. A more useful goal would be ‘I exist, therefore I must do everything I can to increase my knowledge about the universe, and to increase the processing power which I can use to examine that information’. Hopefully an AI with curiosity would not be so enthusiastic about destroying humanity. We could make good pets. [Previous page](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302.md?page=4) [Next page](https://boards.straightdope.com/t/straight-dope-2-24-2023-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980302.md?page=6)