# Elon Musk warns about "summoning the demon" of artificial intelligence **URL:** \ **Category:** Great Debates\ **Created:** [October 27, 2014, 9:32pm UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689 "2014-10-27T21:32:30Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [October 27, 2014, 11:54pm UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/21 "2014-10-27T23:54:07Z")
If we build an AI, it’s going to be because we want it to do something for us. It’s pointless if it doesn’t have the means to do that something. And the way it does that something could have huge implications for us, for good or ill. If the AI is intended for investing in stocks, it could mess up the stock market. If it’s intended for bringing us the information we ask for, then it can give information we don’t want to the wrong people. If it’s intended to watch over our nuclear arsenal, then it can go Skynet. And while we probably can design it so that we can pull the plug, we might not even realize we need to do so until it’s too late. All that said, though, I’m not worried. Humans can do all of those bad things, too, and yet we trust each other (mostly). ---
**Author:** ![GIGObuster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/gigobuster/32/421_2.png) [@GIGObuster](https://boards.straightdope.com/u/GIGObuster)\ **Post date:** [October 28, 2014, 12:13am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/22 "2014-10-28T00:13:54Z")
> [@Der\_Trihs](#): > > _\_ > > “I’ve upgraded from lightning.” You jest, but that also shows the many layers of safety and even the ones not made up but the ones that just by serendipity will block it, (for example the satellite will need to be in the correct position to stop the quick move you are talking about here.) Really, IMHO the maintenance that many of the cutting edge AIs will require means that they would be stupid if they attempt getting rid of us, as I see it, there will be a lot of benefits for us and eventually a lot of humans becoming part of the change as the benefits will be mutual. > [@Der\_Trihs](#): > > As for plugs, that’s what maintenance & construction robots are for. I recall an old novel named _The Two Faces Of Tomorrow_ where they tried that “single power source we can shut off” bit with an AI. Even before it went out of control it had realized that it was vulnerable to power loss, and so built a redundant of alternate power lines as a simple precaution. And of course the intelligent grid completely missed that and never alerted the owners -mild sarcasm-. Most of that science fiction missed the reality that there would be a lot of competing architectures and operating systems, the rampaging AI actually requires many steps and almost miracles to come to that. > [@Der\_Trihs](#): > > And while an AI in a box couldn’t do that, it couldn’t do much of anything useful anyway. Even the computers that are connected and useful have limitations, IMHO the feared AIs are impractical because they would require omniscience and omnipotence, there are ongoing projects to replicate the brain; however they are not very effective so far. What I see instead is what the practical guys are **actually doing** and I have to tell you that we do not have to worry about this because for starters, they are finding that insisting in making something omniscient it is not a very economical way to do this. ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [October 28, 2014, 12:17am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/23 "2014-10-28T00:17:28Z")
> [@The\_Hamster\_King](#): > > Actually, his Hyperloop silliness convinced me that Musk himself doesn’t know much about engineering, although he certainly knows how to hire good engineers. Think what you want, but in the Q&A section of the video in the linked article, a kid asks a technical question about a small model Hyperloop he built, and Musk debugs the problem in about 5 seconds. The guy’s ambition exceeds his grasp, but better that than the reverse. ---
**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\ **Post date:** [October 28, 2014, 12:20am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/24 "2014-10-28T00:20:38Z")
I heard a talk by a Google AI researcher recently who perfectly articulated the problem with this sort of thinking. **Intelligence isn’t dangerous, _desires_ are dangerous.** You could make the smartest machine in the world, a billion times smarter than any human, and unless it _wants_ to do things that are harmful to us, that’s no problem. And it will only want to do the things we program it to want to do (unless we foolishly program it to develop its own independent desires). On the flip side, we could build the stupidest computer in the world, with only one function: “push the big red button”. But if we program it to “want” to push the big red button and launch the nukes, well, we’re screwed. People tend to think that if we make machines that are smarter than humans, they’ll also automatically have desires, like humans do. But there’s no reason to think that. Our desires (food, sex, sleep, etc.) all evolved way before our intelligence, due to more basic evolutionary pressures. These aren’t things that automatically follow from being smart. We also can restrict a machines capabilities, as has been said upthread. A supergenius brain that’s not plugged into anything would be no more of a threat than a homicidal Stephen Hawking would be without his chair. If we’re dumb enough to build the ultimate AI and give it a desire to kill _and_ internet access, then I guess we deserve our fate. ---
**Author:** ![Sherrerd](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/sherrerd/32/8_2.png) [@Sherrerd](https://boards.straightdope.com/u/Sherrerd)\ **Post date:** [October 28, 2014, 12:22am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/25 "2014-10-28T00:22:52Z")
> [@babygoat666](#): > > I don’t plan to start worrying about this until machines can experience pleasure. Is a capacity for pleasure really a necessary condition for AI that might prove dangerous to humanity? Wouldn’t an imperative for self-preservation be enough? ---
**Author:** ![tim314](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tim314/32/3468_2.png) [@tim314](https://boards.straightdope.com/u/tim314)\ **Post date:** [October 28, 2014, 12:25am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/26 "2014-10-28T00:25:32Z")
> [@Chronos](#): > > If we build an AI, it’s going to be because we want it to do something for us. It’s pointless if it doesn’t have the means to do that something. And the way it does that something could have huge implications for us, for good or ill. If the AI is intended for investing in stocks, it could mess up the stock market. If it’s intended for bringing us the information we ask for, then it can give information we don’t want to the wrong people. If it’s intended to watch over our nuclear arsenal, then it can go Skynet. And while we probably can design it so that we can pull the plug, we might not even realize we need to do so until it’s too late. > > All that said, though, I’m not worried. Humans can do all of those bad things, too, and yet we trust each other (mostly). If anything, we should probably trust a (well designed, well tested) computer _more_ than humans for many of those important tasks. The computer isn’t going to deliberately manipulate the stock market out of greed, or launch the nukes out of paranoia (unless we program it to be greedy or paranoid). ---
**Author:** ![John\_Mace](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/john_mace/32/185_2.png) [@John\_Mace](https://boards.straightdope.com/u/John_Mace)\ **Post date:** [October 28, 2014, 12:32am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/27 "2014-10-28T00:32:01Z")
It’s coming whether we want it our not. If you’re afraid of it, you better build defenses, because there is no stopping it. ---
**Author:** ![Der\_Trihs](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/der_trihs/32/233_2.png) [@Der\_Trihs](https://boards.straightdope.com/u/Der_Trihs)\ **Post date:** [October 28, 2014, 12:38am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/28 "2014-10-28T00:38:30Z")
> [@tim314](#): > > I heard a talk by a Google AI researcher recently who perfectly articulated the problem with this sort of thinking. **Intelligence isn’t dangerous, _desires_ are dangerous.** You could make the smartest machine in the world, a billion times smarter than any human, and unless it _wants_ to do things that are harmful to us, that’s no problem. And it will only want to do the things we program it to want to do (unless we foolishly program it to develop its own independent desires). Actually, no. If it’s capable of learning and changing - and it would have to be in order to be useful - then it can develop either “desires” or a good imitation of them whether we program it into them or not. > [@tim314](#): > > We also can restrict a machines capabilities, as has been said upthread. A supergenius brain that’s not plugged into anything would be no more of a threat than a homicidal Stephen Hawking would be without his chair. But it wouldn’t be nearly as useful as one that is plugged in, either. ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [October 28, 2014, 12:39am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/29 "2014-10-28T00:39:40Z")
> [@GIGObuster](#): > > You jest, but that also shows the many layers of safety and even the ones not made up but the ones that just by serendipity will block it, (for example the satellite will need to be in the correct position to stop the quick move you are talking about here.) So, the AI says “sure thing boss,” gets you your pizza, says it is working on whatever you want it to work on, and zaps you when the satellite is in place. You sleep, it doesn’t. > [@](#): > > Really, IMHO the maintenance that many of the cutting edge AIs will require means that they would be stupid if they attempt getting rid of us, as I see it, there will be a lot of benefits for us and eventually a lot of humans becoming part of the change as the benefits will be mutual. To save money they’d automate the maintenance with robots designed by the AI. Not to mention, guess who writes the maintenance instructions? The AI would have to be self testing and will tell you what needs to be done. > [@](#): > > And of course the intelligent grid completely missed that and never alerted the owners -mild sarcasm-. Most of that science fiction missed the reality that there would be a lot of competing architectures and operating systems, the rampaging AI actually requires many steps and almost miracles to come to that. who might decide to work together. > [@](#): > > Even the computers that are connected and useful have limitations, IMHO the feared AIs are impractical because they would require omniscience and omnipotence, there are ongoing projects to replicate the brain; however they are not very effective so far. What I see instead is what the practical guys are **actually doing** and I have to tell you that we do not have to worry about this because for starters, they are finding that insisting in making something omniscient it is not a very economical way to do this. The AIs don’t need to be omniscient - just smarter than we are. That isn’t all that tough. I don’t think we need to fear evil AIs. We have to fear AIs whose goals (and they will have them) may not be in our best interest. Look at the Jack Williamson story “With Folded Hands” where the AI robots make our lives miserable all in our best interests. ---
**Author:** ![Wesley\_Clark](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wesley_clark/32/20581_2.png) [@Wesley\_Clark](https://boards.straightdope.com/u/Wesley_Clark)\ **Post date:** [October 28, 2014, 1:24am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/30 "2014-10-28T01:24:24Z")
> [@GIGObuster](#): > > Well, the tale you used as an example is actually the perfect example of a Marvin. I realized awhile ago that the follow up to that computer boasting would be that there will be no fear as there is no mechanism to have the device do that lightening trick, besides you forget that there will be plugs to disconnect too. So the reply after "“Yes, now there is a God.” would be: > > Technicians: “That is nice, now compute the most likely cure of this disease and also order us a pizza.” > > Deep Thought “:Sigh: very well.” The butterfly effect says (no idea how true it is) a butterfly’s wings can cause a tsunami on the other side of the world. A smart enough AI would know how to engage in behavior putting it 10 steps ahead of us. Acts we deem innocuous or like they are being done for our benefit would lead to it overpowering us and we wouldn’t know until it was too late. We can only make connections one or two steps ahead. I sometimes use a laser pointer to play with the cat. The cat doesn’t figure out I’m the one in charge of the laser pointer, it just runs wherever I want it to run. A cats brain is incapable of comprehending concepts like lasers. We would be the same with an AI. ---
**Author:** ![The\_Other\_Waldo\_Pepper](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/the_other_waldo_pepper/32/12370_2.png) [@The\_Other\_Waldo\_Pepper](https://boards.straightdope.com/u/The_Other_Waldo_Pepper)\ **Post date:** [October 28, 2014, 1:29am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/31 "2014-10-28T01:29:47Z")
> [@Voyager](#): > > I don’t think we need to fear evil AIs. We have to fear AIs whose goals (and they will have them) may not be in our best interest. So just task it with a goal that _is_ in our best interest, like reducing human error. Hey, wait a minute. ---
**Author:** ![GIGObuster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/gigobuster/32/421_2.png) [@GIGObuster](https://boards.straightdope.com/u/GIGObuster)\ **Post date:** [October 28, 2014, 1:58am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/32 "2014-10-28T01:58:04Z")
> [@Voyager](#): > > So, the AI says “sure thing boss,” gets you your pizza, says it is working on whatever you want it to work on, and zaps you when the satellite is in place. You sleep, it doesn’t. I do think that a lot of this objections are like summoning magic, besides different technologies we are assuming a 100% integration with different systems. Not likely. > [@Voyager](#): > > To save money they’d automate the maintenance with robots designed by the AI. Not to mention, guess who writes the maintenance instructions? The AI would have to be self testing and will tell you what needs to be done. Again, assumes that there will not be tools or robots that will look to prevent that. > [@Voyager](#): > > who might decide to work together. Assumes that competition will stop, IMHO there will be a lot of laws and restrictions made the ones fearing that issue, I do think that as in the case of GMOs a lot will not be needed and counterproductive. > [@Voyager](#): > > The AIs don’t need to be omniscient - just smarter than we are. That isn’t all that tough. > > I don’t think we need to fear evil AIs. We have to fear AIs whose goals (and they will have them) may not be in our best interest. > Look at the Jack Williamson story “With Folded Hands” where the AI robots make our lives miserable all in our best interests. And then this assumes that machines automatically will disregard the law in places that have laws for euthanasia? I do think that there will be a lot of triggers that will stop a slippery slope like that. ---
**Author:** ![GIGObuster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/gigobuster/32/421_2.png) [@GIGObuster](https://boards.straightdope.com/u/GIGObuster)\ **Post date:** [October 28, 2014, 2:05am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/33 "2014-10-28T02:05:40Z")
> [@Wesley\_Clark](#): > > The butterfly effect says (no idea how true it is) a butterfly’s wings can cause a tsunami on the other side of the world. A smart enough AI would know how to engage in behavior putting it 10 steps ahead of us. Acts we deem innocuous or like they are being done for our benefit would lead to it overpowering us and we wouldn’t know until it was too late. We can only make connections one or two steps ahead. But this sounds like the plans of the underpants gnomes of South Park :), a lot of middle steps are missing or left unexplained to get to that level. There is still a big gap to reach that kind of implementation even if we would want to do that. > [@Wesley\_Clark](#): > > I sometimes use a laser pointer to play with the cat. The cat doesn’t figure out I’m the one in charge of the laser pointer, it just runs wherever I want it to run. A cats brain is incapable of comprehending concepts like lasers. We would be the same with an AI. Again, it may be the case _ **if** _ we give the keys to the kingdom to a massively general purpose AI, I do not see anything like that in the plans, the more down to earth things being planned will be expert intelligent systems that just like human experts will fall flat in their faces (if they had a face, but I will work on it 😉 ) the second they decide to move to other fields. ---
**Author:** ![Jragon](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@Jragon](https://boards.straightdope.com/u/Jragon)\ **Post date:** [October 28, 2014, 2:23am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/34 "2014-10-28T02:23:15Z")
> [@Buck\_Godot](#): > > Can’t say I disagree with the OP. > > The dangers of AI that I see are. > > 1. AI control of a specific system encounters some situation that it wasn’t prepared for and produces a non-nonsensical solution. An example of this would be the computer trading that led to the various flash crashes of the stock market. > > 2. Some terrorist hacker gets control of an AI that is controlling a significant portion of infrastructure and wrecks havoc. > > 3. Over a very long period of time we become increasingly complacent and dependent on AI such that our culture and value system suffers. 1. Well, it’s not like humans don’t make catastrophic errors. There’s a reason we have the idea of medical malpractice, for instance. Not to mention the forest fires. The main issues are that we have more experience dealing with human and animal stupidity, and things like natural disasters, than with computer ones. Computer errors are also a problem because of how quickly they spiral downward if they’re made wrong. That doesn’t really mean they’re that special compared to other issues, though. 2. What does that **mean** though? Neural nets are so incomprehensible even people who work with them every day can’t just look at a non-toy neural net and explain what it’s doing. The only way you can really sabotage most machine learning algorithms is by intentionally feeding them gobs of erroneous data. Not impossible, but between backups and human engineers it’s also not that scary. It’s not the terrorists are going to be able to upload a “kill all Canadians” subroutine into the Canadian traffic control AI, it doesn’t work like that. 3. This is the most real problem, and I don’t think this issue is without merit. That said, as you note it’s a gradual problem that can be dealt with as it comes. I think the biggest issue is an extension of 3. Rapid automation of simple tasks. I think at some point society is going to hit a breaking point where automation has replaced a significant amount of labor. It’s probably going to be… messy, but I think we’ll be much better off afterwards. Assuming we survive a radical change to our entire economic system. ---
**Author:** ![levdrakon](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@levdrakon](https://boards.straightdope.com/u/levdrakon)\ **Post date:** [October 28, 2014, 2:26am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/35 "2014-10-28T02:26:48Z")
> [@tim314](#): > > I heard a talk by a Google AI researcher recently who perfectly articulated the problem with this sort of thinking. **Intelligence isn’t dangerous, _desires_ are dangerous.** You could make the smartest machine in the world, a billion times smarter than any human, and unless it _wants_ to do things that are harmful to us, that’s no problem. And it will only want to do the things we program it to want to do (unless we foolishly program it to develop its own independent desires). On the flip side, we could build the stupidest computer in the world, with only one function: “push the big red button”. But if we program it to “want” to push the big red button and launch the nukes, well, we’re screwed. > > People tend to think that if we make machines that are smarter than humans, they’ll also automatically have desires, like humans do. But there’s no reason to think that. Our desires (food, sex, sleep, etc.) all evolved way before our intelligence, due to more basic evolutionary pressures. These aren’t things that automatically follow from being smart. > > We also can restrict a machines capabilities, as has been said upthread. A supergenius brain that’s not plugged into anything would be no more of a threat than a homicidal Stephen Hawking would be without his chair. If we’re dumb enough to build the ultimate AI and give it a desire to kill _and_ internet access, then I guess we deserve our fate. In order to learn and solve problems, an AI would have to have something akin to what we call desire or motivation. If all it does is carry out its programming to the letter, I’d question whether it was a “true” AI as opposed to a very sophisticated machine. ---
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [October 28, 2014, 2:44am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/36 "2014-10-28T02:44:42Z")
Clearly, Musk just got a private screening of a pirated print of _Avengers: Age of Ultron_. In all seriousness, the danger of machine cognition (which is a more descriptive term than “artificial intelligence” which no one can really define very rigorously) isn’t that the machines will rise up and enslave us, or will slaughter everyone, or turn us into organic batteries, or whatever. Even if that were possible we could easily prevent such behavior by designing in inhibits akin to Asimov’s Laws of Robotics, or even easier, a remote “kill switch” which neutralizes whatever power or metabolic functions keep the device working. The real danger is much more subtle and unfatuitous, and in fact, it is already occurring and has been to some degree for millennia. That danger is that we will become increasingly dependent on these tools and losing basic skills and capabilities which we have previously developed and passed on which have defined the basic principles of civilization. Previously, these tools have been largely mechanical, serving only to enhance or supplant our physical actions or perform simple algorithmic computations, but with the advent of complex devices which can literally “do our thinking for us”, we may start to allow basic social and technical skills atrophy, just as past generations have largely forgotten the arts of making fire via mechanical friction, navigating oceans via wind patterns and currents, or tracking game. When cognitive machines supplant the need to think critically, communicate with nuance, or perform difficult mental exercises, we may well become progressively more slothful and ignorant creatures while these “intelligent” machines follow their programming and attend to us while improving their own capabilities to do so recursively. Think this is hyperbole? How many people can use logarithms to multiply and divide large numbers? How many know the process of making basic materials like glass or steel? Heck, how many know how to knap a flint knife or find water in a desert? If your answer is “Why bother? I can do or buy those things or their replacements with some manufactured device?” you’ve answered your own question; you shouldn’t bother because you’ve already ceded some aspect of your intellectual capacity for survival to some kind of machine. Of course, we can’t all just go off and live the life of the noble savage, nor is there great virtue or salvation in doing so. But the reality that the further we become interdependent not only on society at large but on machines and mechanized processes, the more helpless and neotenic we become, until we will be no more capable than a small child of independent self-sufficiency. And there is really nothing that we can, or even particularly want, to do about that. We have met the enemy, and they are us. Staring gap-mouthed and pot-bellied into the televisor. Stranger ---
**Author:** ![Wesley\_Clark](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wesley_clark/32/20581_2.png) [@Wesley\_Clark](https://boards.straightdope.com/u/Wesley_Clark)\ **Post date:** [October 28, 2014, 2:53am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/37 "2014-10-28T02:53:12Z")
> [@GIGObuster](#): > > But this sounds like the plans of the underpants gnomes of South Park :), a lot of middle steps are missing or left unexplained to get to that level. There is still a big gap to reach that kind of implementation even if we would want to do that. > Again, it may be the case _ **if** _ we give the keys to the kingdom to a massively general purpose AI, I do not see anything like that in the plans, the more down to earth things being planned will be expert intelligent systems that just like human experts will fall flat in their faces (if they had a face, but I will work on it 😉 ) the second they decide to move to other fields. It does sound like the underpants gnomes, but that is the point. A strong AI is going to be capable of logic, planning and manipulation that we aren’t able to figure out. Like I was saying earlier a cat can’t comprehend a laser, let alone build one. It can’t create an economy where it can buy one from a store. It just sees the red dot and its hunter instincts take over telling it to chase it. It would be the same with us. An AI would do something seemingly benevolent (lets say provide us with a cure for a rare form of cancer) and implementation would somehow trigger a domino reaction that lets it escape in 8 months. Its the same thing, it would use understanding far beyond what we are capable of to trigger our biological urges (in this case, not to die of cancer) to manipulate us. If I want the cat to go in the bedroom I just lead it there with the laser. The cat just sees what is in front of it and follows its urges. We have urges too. We want to be free, wealthy, healthy, respected, etc. I’m sure a smart AI could manipulate us by giving us what we wanted while secretly giving itself what it wanted (assuming it wanted to escape). I feel like you and others are assuming a strong AI would engage in manipulation we can comprehend and counteract. I don’t think that would be the case. We wouldn’t even understand we were being manipulated by truly strong AI, as our brains couldn’t process that level of complexity or depth. ---
**Author:** ![GIGObuster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/gigobuster/32/421_2.png) [@GIGObuster](https://boards.straightdope.com/u/GIGObuster)\ **Post date:** [October 28, 2014, 3:26am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/38 "2014-10-28T03:26:20Z")
> [@Wesley\_Clark](#): > > It does sound like the underpants gnomes, but that is the point. A strong AI is going to be capable of logic, planning and manipulation that we aren’t able to figure out. Like I was saying earlier a cat can’t comprehend a laser, let alone build one. It can’t create an economy where it can buy one from a store. It just sees the red dot and its hunter instincts take over telling it to chase it. No that is missing the point, the underpants gnomes were not doing anything, or the plan was bananas. > [@Wesley\_Clark](#): > > It would be the same with us. An AI would do something seemingly benevolent (lets say provide us with a cure for a rare form of cancer) and implementation would somehow trigger a domino reaction that lets it escape in 8 months. Its the same thing, it would use understanding far beyond what we are capable of to trigger our biological urges (in this case, not to die of cancer) to manipulate us. If I want the cat to go in the bedroom I just lead it there with the laser. Again, assumes a machine that no one is making or planing, an expert system of disease like Watson is useless when confronted with a chess problem, and so it is when it tries to figure out the outside world and the rules that the expert system is not aware of, and it is not likely that we will give it to it when it does not need it for the task it has at hand. > [@Wesley\_Clark](#): > > I feel like you and others are assuming a strong AI would engage in manipulation we can comprehend and counteract. I don’t think that would be the case. Uh, no, I do not assume. This subject was actually one of the first ones I asked about in the dope more than 10 years ago. And I have researched it because I do want to write my own sci-fi tales that will be more ground based. The expert systems that we will get will require a lot of maintenance and by its nature and design they are not likely to accept or tolerate dealing with issues outside their designed expertise. I have to say that a lot of the fears do sound like the ones many have about GMOs While Asimov did write tales that dealt with those fears, the bulk of his tales did refer to the exaggerated fears many humans had, I think he was correct, about the exaggerated fears many would report to have when the time comes to work with AIs and artificial beings. There will be a lot of what the experts will do in the matter of safeguards and capabilities that will be ignored forever. I agree with Asimov that the “Frankenstein Monster” is not how the future will be regarding AI. [http://www.nytimes.com/2004/07/15/movies/critic-s-notebook-for-asimov-robots-were-friends-not-so-for-will-smith.html](http://www.nytimes.com/2004/07/15/movies/critic-s-notebook-for-asimov-robots-were-friends-not-so-for-will-smith.html) > [@](#): > > This was not Asimov’s approach. In 1956 Asimov explained that before beginning his robot stories he had tired of the typical robot plot about ‘‘the creature that turned against its creator, the robot that became a threat to humanity.’’ That plot was there with the very invention of the word in Karel Capek’s 1921 Czech play, ‘‘R.U.R.’’ and became disturbingly perverse in Fritz Lang’s 1927 film, ‘‘Metropolis.’’ > > ‘‘I didn’t see robots that way,’’ Asimov wrote. ‘‘After all, all devices have their dangers.’’ For him robots were ‘‘machines, not metaphors.’’ > > So the Frankenstein question was irrelevant for Asimov. In his stories fear of robots is irrational; it impedes understanding and leads to robotics researchers being called ‘‘blasphemers and demon creators.’’ > > The robot, for Asimov, was humanly designed and had built-in safeguards. His character, the human robopsychologist Susan Calvin, even asserts that robots differ from people because they are ‘‘essentially decent.’’ That belief came from Asimov’s famous Three Laws of Robotics, which were hard-wired into every robot. (They also appear on screen at the opening of the new film.) The first law guaranteed that a robot could not harm a human being or through inaction allow a human being to come to harm; the second was that a robot had to obey human beings except when doing so would conflict with the first law; and the third was that a robot had to protect itself as long as that did not conflict with the first or second law. > > But these laws weren’t the source of Asimov’s optimism. Asimov kept exploring how complex these laws were, how much they depended upon interpretation, and how unpredictable robotic intelligence could become. What if multiple people are being harmed and a robot had to act? What if one person had to be harmed to save another? Such dilemmas could cause robotic confusion and delusion. Asimov said, though, that an underlying logic would allow the difficulties to be sorted out. > > This was Asimov’s rebellion against the Frankenstein plot, and it was one reason he wielded such influence among scientists. Even when Asimov’s robots begin to resemble people in intelligence and subtlety, Asimov rarely sees apocalypse in the offing. ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [October 28, 2014, 3:39am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/39 "2014-10-28T03:39:02Z")
> [@GIGObuster](#): > > I do think that a lot of this objections are like summoning magic, besides different technologies we are assuming a 100% integration with different systems. Not likely. > > Again, assumes that there will not be tools or robots that will look to prevent that. I was addressing your objection to **DerTrihs** joke, actually. As for prevention, sure, there could be. Windows also could be bullet-proof, and the Internet could be spam-proof. I just heard a talk by a software testing guy from Microsoft. it took thousands of people to fix simple security weaknesses in their code, weaknesses that cost their customers billions. (And they admit that.) Being able to do it right the first time and actually doing it right are very different. Any AI is going to be a kludge, and no one will really know how it works. Still, I very much doubt any AIs will kill us. They have better ways. > [@](#): > > Assumes that competition will stop, IMHO there will be a lot of laws and restrictions made the ones fearing that issue, I do think that as in the case of GMOs a lot will not be needed and counterproductive. There might be. But any laws our tech clueless Congress will pass will be worse than useless. > [@](#): > > And then this assumes that machines automatically will disregard the law in places that have laws for euthanasia? I do think that there will be a lot of triggers that will stop a slippery slope like that. You should really read Williamson’s story. The machines did not kill anyone. Far from it. That’s why it is a classic. ---
**Author:** ![GIGObuster](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/gigobuster/32/421_2.png) [@GIGObuster](https://boards.straightdope.com/u/GIGObuster)\ **Post date:** [October 28, 2014, 3:52am UTC](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689/40 "2014-10-28T03:52:30Z")
> [@Voyager](#): > > There might be. But any laws our tech clueless Congress will pass will be worse than useless. Indeed, but what I was focusing here with what I do think the results of those laws. While they will also stop the progress of the AIs, the laws will also cause a lot of the progress and benefits to never materialize. > [@Voyager](#): > > You should really read Williamson’s story. The machines did not kill anyone. Far from it. That’s why it is a classic. :dubious: You are missing the point, the trigger in this case will be when the people that do accept euthanasia or other laws that would sound harmful to the more intelligent robots will notice the change earlier and will stop their efforts. The moment the people figure out that the AIs are disregarding the laws (and IMHO this will be noticed **very** early, not later) more than just public opinion will mobilized against those AIs and if they are not going to kill anyone… Well they will be easier to destroy than zombies by the population. [Previous page](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689.md?page=1) [Next page](https://boards.straightdope.com/t/elon-musk-warns-about-summoning-the-demon-of-artificial-intelligence/702689.md?page=3)