# Straight Dope 3/3/23: Followup - What are the chances artificial intelligence will destroy humanity? **URL:** \ **Category:** Cecil's Columns/Staff Reports\ **Tags:** ai\ **Created:** [March 3, 2023, 5:55pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665 "2023-03-03T17:55:42Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![Sam\_Stone](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)\ **Post date:** [March 4, 2023, 10:44pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/61 "2023-03-04T22:44:52Z")
> [@Crane](#): > > Computing is a process that is able to simulate human activity. The amazing thing is how human it appears to be. However we know exactly what the computer is doing. The folks who made it have told us in detail. It is computing. No, we don’t know what the computer is doing, and the folks who made ChatGPT and other LLMs explicitly admit they don’t know how it is doing what it does. This has been explained to you multiple times in multiple threads. Saying that ‘we know what it’s doing’ because we understand microprocessors is like saying that we understand consciousness because we understand electrochemistry. The hardware of the computer or the ‘wetware’ of the brain are just the substrates on which neural networks are built. It’s the neural nets themselves that are evolved/trained and not programmed, and which are hellishly complex and mostly inscrutable. There are efforts going on to use small neural nets and lots of instrumentation to figure out how they work, and we’re making a little progress in discovering circuits, ‘induction heads’ and other evolved structures. We’ve also found structures that look pretyy much like structures evolved in the human brain. Today, large language models are black boxes. No one knows how they do what they do. ---
**Author:** ![Dr\_Paprika](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr_paprika/32/3042_2.png) [@Dr\_Paprika](https://boards.straightdope.com/u/Dr_Paprika)\ **Post date:** [March 4, 2023, 10:44pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/62 "2023-03-04T22:44:52Z")
If _everything_ is computation, the brain is a subset of that. I don’t agree, but that doesn’t matter. It’s a sideshow to Cecil’s column. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 4, 2023, 10:54pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/63 "2023-03-04T22:54:16Z")
> [@Dr\_Paprika](#): > > If _everything_ is computation, the brain is a subset of that. I don’t know what you mean. Nobody has claimed that “everything” is computation. > [@Dr\_Paprika](#): > > I don’t agree And others have said here that they don’t agree, but nobody has any suggestion for what the brain is doing _other_ than computation. There is not only no evidence that the brain is doing something other than computation, nobody has given any coherent hypothesis for what they mean by “something else”. > [@Dr\_Paprika](#): > > It’s a sideshow to Cecil’s column. Well, it’s not a sideshow when people use it as the basis for a belief that AGI (and therefore ASI) won’t happen. ---
**Author:** ![thorny\_locust](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thorny_locust/32/431_2.png) [@thorny\_locust](https://boards.straightdope.com/u/thorny_locust)\ **Post date:** [March 4, 2023, 11:05pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/64 "2023-03-04T23:05:45Z")
> [@Riemann](#): > > Sure, but I don’t see how that implies that the brain is doing something other than computation. I’m not. I’m saying that part of what’s going on in the human as a whole isn’t computation. > [@Riemann](#): > > But experiencing emotions clearly also happens in your brain. What do you suggest is happening in your brain _other_ than computation? I’m not saying that emotions don’t _also_ happen in the brain, or that the brain is doing something other than computation. I’m saying that emotions don’t happen _entirely_ in the brain. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 4, 2023, 11:11pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/65 "2023-03-04T23:11:52Z")
> [@thorny\_locust](#): > > I’m saying that emotions don’t happen _entirely_ in the brain. There are many _correlates_ of emotional states outside the brain. But it is curious to suggest that the _experience_ of emotion happens anywhere other than the brain. If I burn my finger, surely you don’t think that my finger is experiencing the burning. ---
**Author:** ![thorny\_locust](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thorny_locust/32/431_2.png) [@thorny\_locust](https://boards.straightdope.com/u/thorny_locust)\ **Post date:** [March 4, 2023, 11:20pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/66 "2023-03-04T23:20:28Z")
I think your body’s endocrine response to that is happening all the way through your body; and that, without it, your brain’s calculation that you should move your finger away from the stove would be an entirely different thing. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 4, 2023, 11:23pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/67 "2023-03-04T23:23:10Z")
Consider the nerve connections between your finger and your brain. Do you think it would be impossible (in principle) to hook the brain up to an artificial source of nerve impules that would make your brain think that you have a finger and that it is hot? ---
**Author:** ![thorny\_locust](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thorny_locust/32/431_2.png) [@thorny\_locust](https://boards.straightdope.com/u/thorny_locust)\ **Post date:** [March 4, 2023, 11:27pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/68 "2023-03-04T23:27:33Z")
It is certainly possible for your brain to think that your finger’s in pain when you don’t even have a finger. But the emotional reaction to the sensation of “pain” involves a lot more of the body than just the finger and the brain. ---
**Author:** ![Crane](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/crane/32/3495_2.png) [@Crane](https://boards.straightdope.com/u/Crane)\ **Post date:** [March 4, 2023, 11:29pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/69 "2023-03-04T23:29:29Z")
> [@Sam\_Stone](#): > > No, we don’t know what the computer is doing Actually we do. It’s explained very clearly [here](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/). Being surprised by results is a positive. It means “Wow” not “Ooooo spooky”. There is no mystery about how neural nets work. Of course there is research on better understanding their fine points. Nothing spooky about that. Have you read the link? ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 4, 2023, 11:31pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/70 "2023-03-04T23:31:45Z")
> [@thorny\_locust](#): > > But the emotional reaction to the sensation of “pain” involves a lot more of the body than just the finger and the brain. If you’re simply going to redefine “emotion” to include all the physical _correlates_ of emotion, then sure. The question is, do you really think the _experience_ of emotion is happening somewhere other your brain? But in any event, if you’re going to argue that human intelligence is partly located outside of the brain, that doesn’t really change anything. If your want to say that the endocrine system is part of human intelligence, what is _that_ doing other than information processing - i.e. computation? Remember that the basis for this discussion is the claim that there is something that humans do that a machine cannot (in principle) do. What is your endocrine system doing that a machine could not do? ---
**Author:** ![Tibby](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tibby/32/17030_2.png) [@Tibby](https://boards.straightdope.com/u/Tibby)\ **Post date:** [March 4, 2023, 11:31pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/71 "2023-03-04T23:31:45Z")
> [@thorny\_locust](#): > > I don’'t mean, is your brain doing computation in response to the symptoms of the emotions. I mean, is your body doing computation when it releases hormones, changes bloodflow, changes heart rate, etc.? Those processes (and others) are controlled by the autonomic nervous system (further divided into the sympathetic and parasympathetic systems…turtles all the way down) which is a component of the peripheral nervous system, PNS (nerves and the ganglia), which is basically the conduit between the central nervous system, CNS (brain and spinal cord) and the body parts/organs. No consciousness is involved with the PNS. It functions…automatically/autonomously. The CNS is the domain of consciousness. The Merck Manual explains the PNS clearly and easily understood: [Overview of the Autonomic Nervous System - Brain, Spinal Cord, and Nerve Disorders - Merck Manuals Consumer Version](https://www.merckmanuals.com/home/brain,-spinal-cord,-and-nerve-disorders/autonomic-nervous-system-disorders/overview-of-the-autonomic-nervous-system#:~:text=The%20autonomic%20nervous%20system%20Autonomic%20nervous%20system%20is%20the%20part,%2C%20salivary%2C%20and%20digestive%20glands). ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 4, 2023, 11:52pm UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/72 "2023-03-04T23:52:46Z")
Not to put words into your mouth, but I suspect that your underlying reasoning is - we don’t understand the basis of consciousness and subjective experience, therefore a machine intelligence cannot do what a human can do? The fact that we don’t understand consciousness certainly introduces uncertainty, but I don’t think it warrants that conclusion. There is just no hypothesis (and certainly no evidence) that biological intelligences do anything other than computation. The endocrine system certainly isn’t remotely an _explanation_ of subjective experience. We have no explanation whatsoever. Consciousness just seems to be an emergent property in the evolution of biological intelligence, and there is no evdience to imply that it could not emerge in machine intelligence. Furthermore, the fact that an articificial intelligence might not be conscious does not place any constraints on its competence. Perhaps the most frightening prospect is that we might develop superintelligence that supercedes us because of its vastly greater competence - but it is not conscious, and there is nobody left to _experience_ the universe. ---
**Author:** ![thorny\_locust](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thorny_locust/32/431_2.png) [@thorny\_locust](https://boards.straightdope.com/u/thorny_locust)\ **Post date:** [March 5, 2023, 12:01am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/73 "2023-03-05T00:01:26Z")
> [@Tibby](#): > > No consciousness is involved with the PNS. It functions…automatically/autonomously. The CNS is the domain of consciousness. Yeah. And they’re all tangled up with each other. > [@Riemann](#): > > Not to put words into your mouth, but I suspect that your underlying reasoning is - we don’t understand the basis of consciousness and subjective experience, therefore a machine intelligence cannot do what a human can do? > > The fact that we don’t understand consciousness certainly introduces uncertainty, but I don’t think it warrants that conclusion. There is just no hypothesis (and certainly no evidence) that biological intelligences do anything other than computation. The endocrine system certainly isn’t remotely an _explanation_ of subjective experience. We have no explanation whatsoever.Consciousness just seems to be an emergent property in the evolution of biological intelligence, and there is no evdience to imply that it could not emerge in machine intelligence. Really isn’t any evidence either way, as far as I can tell. But I was talking about the emergence of emotion, not only of consciousness. > [@Riemann](#): > > Furthermore, the fact that an articificial intelligence might not be conscious does not place any constraints on its competence. Isn’t that where the idea of the AI trying to turn the whole universe into paper clips comes from? Hyper competence; and no compassion for anybody destroyed in the process. I’m not at all sure you couldn’t get that along with consciousness. But it isn’t only our consciousness that makes us human. (Not that all humans care who or what gets destroyed in the process of trying to get something done, either.) ---
**Author:** ![Tibby](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tibby/32/17030_2.png) [@Tibby](https://boards.straightdope.com/u/Tibby)\ **Post date:** [March 5, 2023, 12:11am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/74 "2023-03-05T00:11:30Z")
> [@thorny\_locust](#): > > Yeah. And they’re all tangled up with each other. Yeah. Likewise you could say guitar pics are tangled up with musicians, but I wouldn’t attribute credit to them when Bob Dylan wrote _Tangled Up in Blue._ ---
**Author:** ![k9bfriender](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/k9bfriender/32/3283_2.png) [@k9bfriender](https://boards.straightdope.com/u/k9bfriender)\ **Post date:** [March 5, 2023, 1:31am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/75 "2023-03-05T01:31:30Z")
> [@Sam\_Stone](#): > > There are efforts going on to use small neural nets and lots of instrumentation to figure out how they work, and we’re making a little progress in discovering circuits, ‘induction heads’ and other evolved structures. We’ve also found structures that look pretyy much like structures evolved in the human brain. > > Today, large language models are black boxes. No one knows how they do what they do. A big difference is that you _can_ get a read out on what every neuron is doing, what it connections and weights are, and you can reset and repeat stimuli. It’s a lot harder to get that information out of a brain. At this point, NNs are “simple” enough that we could theoretically analyze them and understand how they work. But it would be an exceptionally difficult task, only getting more difficult exponentially for every neuron added. I don’t think that current implementations of AI are nearly complex enough for self-awareness to emerge, but I don’t think that the limiting factor is that it’s made of silicon vs carbon. How big and complex it needs to be is hard to say, but I think it will be lower than some may think. Our brains aren’t necessarily all that efficient, and we don’t need all of it to be self aware, so it’s entirely possible that a computer can become self aware with a fraction of the complexity of a human brain. ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 5, 2023, 1:51am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/76 "2023-03-05T01:51:12Z")
> [@thorny\_locust](#): > > But I was talking about the emergence of emotion, not only of consciousness. Emotion arose as evolution’s way of implementing sophisticated behavior in intelligent animals. If (say) a goal is to obtain food, then a whole lot of environment-dependent decisions and skills and sub-goals are required. Searching, catching, etc. - it would be incredibly difficult to hard code all of this. So “hunger” is evolution’s way of specifying the ultimate goal of finding food, while leaving all of the details of exactly how we do it to our general intelligence that’s already equipped with many skills and the ability to understand and reason and react to the specifics of the environment. Without consciousness, an emotion is really just the same kind of thing as a reward function. The subjective experience of an emotion is what a reward function feels like to a conscious intelligence. > **[Emotions = Reward Functions — LessWrong](https://www.lesswrong.com/posts/3ccE9WvXLacGkFEoi/emotions-reward-functions)** > > ... Or more specifically, a post on how, and why, to encode emotions to find out more about goals, rationality and safe alignment in general. • >   … ---
**Author:** ![Sam\_Stone](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)\ **Post date:** [March 5, 2023, 2:05am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/77 "2023-03-05T02:05:25Z")
> [@k9bfriender](#): > > A big difference is that you _can_ get a read out on what every neuron is doing, what it connections and weights are, and you can reset and repeat stimuli. It’s a lot harder to get that information out of a brain. True. And that’s exactly what they are doing with ‘trivial’ neural nets, like the one I mentioned earlier - 50,000 parameters in a single layer. That little NN managed to teach itself trig identities and fast fourier transforms to solve addition problems. It was extremely difficult figuring that out. Now imagine 175 billion parameters arranged in 96 layers… The process for figuring out what’s going on in a neural net is called “Mechanistic Interpretability.” Here are some good links to information about the efforts to understand neural nets: [A Mechanistic Interpretability Analysis of Grokking](https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking) [Progress Measures for ‘Grokking’ via Mechanistic Interpretability](https://arxiv.org/pdf/2301.05217.pdf) [Zoom in: An Intoduction to Circuits](https://distill.pub/2020/circuits/zoom-in/) [A Mathematical Framework for Transformer Circuits](https://transformer-circuits.pub/2021/framework/index.html#induction-heads) [In-Context Learning and Induction Heads](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html) ---
**Author:** ![Sam\_Stone](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)\ **Post date:** [March 5, 2023, 2:28am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/78 "2023-03-05T02:28:33Z")
> [@Crane](#): > > Actually we do. It’s explained very clearly [here](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/). I provided that link originally, I believe. And Wolfram describes a lot about the basics of training, some of what we have discovered about structure, etc. But he acknowledges repeatedly that we _don’t_ know much of what’s going on inside ChatGPT when it’s answering questions, For example: > [@](#): > > What determines this structure [weights in the network]? Ultimately it’s presumably some “neural net encoding” of features of human language. But as of now, what those features might be is quite unknown. In effect, we’re “opening up the brain of ChatGPT” (or at least GPT-2) and discovering, yes, it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language. And… > [@](#): > > It has to be emphasized again that (at least so far as we know) there’s no “ultimate theoretical reason” why anything like this should work. And in fact, as we’ll discuss, I think we have to view this as a—potentially surprising—scientific discovery: that somehow in a neural net like ChatGPT’s it’s possible to capture the essence of what human brains manage to do in generating language. There is more along those lines in the paper. ---
**Author:** ![Dr\_Paprika](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr_paprika/32/3042_2.png) [@Dr\_Paprika](https://boards.straightdope.com/u/Dr_Paprika)\ **Post date:** [March 5, 2023, 4:47am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/79 "2023-03-05T04:47:08Z")
Apparently, _everything is_ a computer… just define things in abstractly enough. > **[Yes, the brain is a computer…](https://medium.com/the-spike/yes-the-brain-is-a-computer-11f630cad736)** > > No, it’s not a metaphor ---
**Author:** ![Riemann](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/riemann/32/3133_2.png) [@Riemann](https://boards.straightdope.com/u/Riemann)\ **Post date:** [March 5, 2023, 5:41am UTC](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665/80 "2023-03-05T05:41:15Z")
> [@Dr\_Paprika](#): > > Apparently, _everything is_ a computer… just define things in abstractly enough. Yes, there is a loose sense in which everything is a computer - everything is implementing algorithms. But the article also explains the narrow meaning for computer: a universal Turing machine (aka Turing complete). This is a computer that can implement _any_ algorithm, and can therefore simulate any other computer. Then we have: (1) The brain does not just fit the loose “everything” definition of computer, it almost certainly fits the narrow definition - it is probably Turing complete. (2) There is no suggestion that the brain is doing “something else”, it is a computer that is either Turing complete or it is not. (3) Even current programming languages are Turing complete - your cellphone is a universal Turing machine (ignoring memory limitations). So a machine intelligence can (theoretically) simulate the human brain. The brain is a computer, and computation is substrate-independent. [Previous page](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665.md?page=3) [Next page](https://boards.straightdope.com/t/straight-dope-3-3-23-followup-what-are-the-chances-artificial-intelligence-will-destroy-humanity/980665.md?page=5)