# Strong AI and the Chinese Room
**URL:** https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658
**Category:** Great Debates
**Created:** [December 5, 2002, 6:32am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658 "2002-12-05T06:32:13Z")
**Posts on this page:** 19
**Page:** 7
### Author:  [@hazel-rah](https://boards.straightdope.com/u/hazel-rah)
#### Post date: [December 11, 2002, 2:38am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/121 "2002-12-11T02:38:59Z")
> [@](#):
>
> Now, imagine a person that has the same means to receive and communicate as the box. What sort of questions could you ask it? I mean, this isn’t even Hellen Keller, for she had touch. Consider, before we get to questioning the box: what could we teach it? —How?
Well, the Chinese Room assumptions kind of cover that aspect of the scenario… the program and tables contain within their functionality all possible Chinese utterances and their proper responses. You can ask it anything, in theory.
As for how the box communicates with the world, it’s not relevant to the point the Chinese Room is making. You can posit any kind of apparatus for communicating with the world… video cameras, microphones and speakers, a fax machine. As long as all the man does is take the input in whatever form it comes in and place it into the program. Then he uses whatever to get the response back outside the box.
According to Searle you can totally internalize the whole thing into the mind of the man and just have him talk to people in Chinese if suits you. He will still understand English and not understand Chinese.
-fh
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 11, 2002, 2:47am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/122 "2002-12-11T02:47:01Z")
> [@](#):
>
> the program and tables contain within their functionality all possible Chinese utterances and their proper responses. You can ask it anything, in theory.
Then I agree with Searle: this case is impossible.
> [@](#):
>
> As for how the box communicates with the world, it’s not relevant to the point the Chinese Room is making.
It is _to me_, when I consider those things relevent to meaning, intention, and so forth. As, I suspect, it is relevent to Searle.
I’d like to read the paper/book in which he puts the argument forth. The CDP lists it as being in, “Minds, Brains and Programs”, just like that, probably indicating a dissertation in a periodical. It references The Rediscovery of the Mind, from 1992, where the argument is fleshed out more.
---
### Author:  [@hazel-rah](https://boards.straightdope.com/u/hazel-rah)
#### Post date: [December 11, 2002, 2:57am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/123 "2002-12-11T02:57:54Z")
Um, try the link in the OP.
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 11, 2002, 2:59am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/124 "2002-12-11T02:59:18Z")
> [@](#):
>
> Then I agree with Searle: this case is impossible.
I think I should clarify. This case is impossible because I agree: natural language is not able to be put into such a rule-system without the various accountriments we, as humans, have, which includes (but is not limited to) our senses and interactive history. However, this only means: we can’t teach boxes Chinese (or English) [without these accompanying elements]; not: these boxes do not have intentional states; not: these boxes _cannot_ ‘understand’ ‘the meaning of sentences’.
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 11, 2002, 3:02am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/125 "2002-12-11T03:02:17Z")
I got the link in the OP. It doesn’t include later amendments.
---
### Author:  [@hazel-rah](https://boards.straightdope.com/u/hazel-rah)
#### Post date: [December 11, 2002, 3:21am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/126 "2002-12-11T03:21:01Z")
But that isn’t what Searle was arguing, that is a criticism of Searle’s Chinese Room scenario.
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 11, 2002, 4:46am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/127 "2002-12-11T04:46:52Z")
Yes, it is a problem with the scenario; _I believe_ Searle creates an impossible situation; impossible situations are not very enlightening to me as a matter of analytical philosophy. But, disregarding my own opinions on meaning… not to sound trite—I mean I’m not a pHD philosopher or anything—but that paper is full of assertions rather than explanations. No one has taken it on, so I will do so, in true “vivisection by vB code” fashion that only philosophical debates can muster. 😃
**Systems Reply**
> [@](#):
>
> So there are really two subsystems in the man; one understands English, the other Chinese, and “it’s just that the two systems have little to do with each other.” But, I want to reply, not only do they have little to do with each other, they are not even remotely alike.
Yeah, neither is the electrical behavior of my neurons and the action of ‘my baking of a cake’. So?
> [@](#):
>
> The subsystem that understands English (assuming we allow ourselves to talk in this jargon of “subsystems” for a moment) knows that the stories are about restaurants and eating hamburgers, he knows that he is being asked questions about restaurants and that he is answering questions as best he can by making various inferences from the content of the story, and so on. But the Chinese system knows none of this.
_How does he know?_ He just asserts it, right off, because he already assumes the only thing that can ‘understand’ is the man in the box. But two things are being questioned: the whole box, and the homunculus. Why o why should we expect their answers to accord except possibly in very degerate cases of meaning (for example, solving equations not written in Chinese or English).
> [@](#):
>
> And it doesn’t meet that argument to postulate subsystems within the man, because the subsystems are no better off than the man was in the first place; they still don’t have anything even remotely like what the English-speaking man (or subsystem) has.
_What_ does he have? An ability to _feel_? But how do we know? Well, we ask him.
> [@](#):
>
> As far as I can tell the only grounds are that in the example I have the same input and output as native Chinese speakers and a program that goes from one to the other.
as far as I can tell, the only grounds for the point of the CR is Searle’s own assertions in the matter by questioning the homunculus. And if Searle is the homunculus, how do _I_ know that _he, in fact, understands English_? Well, I ask him questions.
> [@](#):
>
> The example shows that there could be two “systems,” both of which pass the Turing test, but only one of which understands
Bullshit. I’m sorry, but this is bullshit. He never offers his criteria for understanding in reply to the systems test other than, “I am telling you I understand English” which is what we would expect the CR to say, too (well, about Chinese).
> [@](#):
>
> In short, the systems reply simply begs the question by insisting without argument that the system must understand Chinese.
Now, I don’t like the systems reply, either; I believe it makes a metaphysical assertion that is just as open to skeptical treatment as the problem of other minds; but it is the biggest irony that someone hinging their argument on the report of an unquestionable being, claiming this undermines the Turing Test, is getting worried about the other party making one assertion (which, to tell the truth, is the whole of the systems reply in itself; that is, _passing the Turing Test is the definition of ‘understanding’ or ‘intentional states’ etc_, so of course it is an assertion; viewed in this light, I think there can be no “systems reply” at all, they are simply talking passed each other).
**Robot Reply** [Note: me, with a healthy degree of skepticism and a different perspective on meaning]
> [@](#):
>
> The first thing to notice about the robot reply is that it tacitly concedes that cognition is not solely a matter of formal symbol manipulation, since this reply adds a set of causal relations with the outside world (cf. Fodor 1980).
No, it doesn’t. It concedes that _our language is not meant to function in a vacuum_.
> [@](#):
>
> But the answer to the robot reply is that the addition of such “perceptual” and “motor” capacities adds nothing by way of understanding, in particular, or intentionality, in general, to Schank’s original program…
This may be (re: Schank’s program), but it at least makes me happy; I no longer find the CR impossible when it is a Chinese 'Bot.
> [@](#):
>
> I am the robot’s homunculus, but unlike the traditional homunculus, I don’t know what’s going on.
As Dennett’s “Where am I?” elucidates, a brain doesn’t know it is a brain, either; at least, _not with respect to natural inclinations of expression_, and not with _accustomed use_ of phrases like, “I am in a deep shaft below the Earth’s crust, and my brain is in a vat” or “I am floating in a vat, and my body is in a deep shaft below the Earth’s crust”. (If anyone is not aware of this story, I can give a summary on request… and I have the story here by my side, so no intentional errors of the story’s point will be made—well, assuming you trust me, anyway… do I pass the Turing Test? ;))
Would we expect, for instance, that a human instantly programmed with an ability to speak (but lacking all the attendent _use_ of the words, and the situations they were _taught_ in) would still report the same things and be inclined to _feel_ the same things (that is, feel the same things about these parts of speech and their function)?
> [@](#):
>
> Now in this case I want to say that the robot has no intentional states at all; it is simply moving about as a result of its electrical wiring and its program.
this is so close to my own opinion it is like I am _just_ missing Searle’s point (if I assume he has one), or Searle never follows his own thoughts through.
> [@](#):
>
> And furthermore, by instantiating the program I have no intentional states of the relevant type. All I do is follow formal instructions about manipulating formal symbols.
Of course, “all I do is follow formal instructions”! “All” a brain does is adjust electrical and chemical signals. But, significantly, when questioned in a certain way, it reports understanding.
**Brain Simulator Reply**
> [@](#):
>
> Before countering this reply I want to digress to note that it is an odd reply for any partisan of artificial intelligence (or functionalism, etc.) to make: I thought the whole idea of strong AI is that we don’t need to know how the brain works to know how the mind works.
It is odd only becuase _it is the only repsonse you are willing to consider_. _that_ is odd, yes. 😉
> [@](#):
>
> If we had to know how the brain worked to do AI, we wouldn’t bother with AI.
Yes, and if we could question homunculii, we wouldn’t have this as a thought experiment, but a falsified (or corroborated) theory.
> [@](#):
>
> Now where is the understanding in this system?
Only a dualist will find this question relevent. Non-dualists will find it contains a mistake. Similarly, “Where is the essence of this chair?” is a question only a Platonist can answer. But asking these questions if we are not platonists and dualists is as unfair to the questionee as it is to the questioner should the roles be reversed. And neither Searle (he claims) nor Strong AI proponents (by necessity) are dualists.
> [@](#):
>
> The problem with the brain simulator is that it is simulating the wrong things about the brain. As long as it simulates only the formal structure of the sequence of neuron firings at the synapses, it won’t have simulated what matters about the brain, namely its causal properties, its ability to produce intentional states.
That’s nice. How do you know it doesn’t achieve this? By asking the homunculus if it knows or understands anything about what is happening outside of its perception. And this answer is supposed to demonstrate that there are no intentional states? _If_ I had a homunculus in my brain, I should expect it to answer similarly, for it is in my brain, and only has access to what the brain gives it: electrical and chemical signals (which, I might add, are meaningful _to us, outside of the brain_ in a different way then they are meaningful to _it, inside the brain_; but, this is just how I view meaning, so I’ve only added it parenthetically).
**The Combination Reply**
> [@](#):
>
> Indeed, besides appearance and behavior, the other elements of the combination are really irrelevant. If we could build a robot whose behavior was indistinguishable over a large range from human behavior, we would attribute intentionality to it, pending some reason not to.
Give the man a prize; he understands the Strong AI claim! If something passes a Turing Test, it understands. Full stop. Making the Turing Test harder or more complicated might convince more people. The Strong AI claim is: “this is the definition of ‘understanding’: following an algorithm.” Searle is welcome to combat this definition, though I don’t see how any thought experiment will achieve it.
> [@](#):
>
> According to strong AI, instatitiating a formal program with the right input and output is a sufficient condition of, indeed is constitutive of, intentionality. As Newell (1979) puts it, the essence of the mental is the operation of a physical symbol system. But the attributions of intentionality that we make to the robot in this example have nothing to do with formal programs. They are simply based on the assumption that if the robot looks and behaves sufficiently like us, then we would suppose, until proven otherwise, that it must have mental states like ours that cause and are expressed by its behavior and it must have an inner mechanism capable of producing such mental states.
John, John, John… tsk tsk. You must see the difference between the metaphysical assertion/definition: “This is understanding” and the means by which a person says, “This [or it or he or she] understands.”
> [@](#):
>
> If we knew independently how to account for its behavior without such assumptions we would not attribute intentionality to it, especially if we knew it had a formal program.
Yes, and if natural language operated in a vacuum, the Strong AI claim could indeed be corroborated or falsified (barring the skeptic’s insistence that we know no such thing; that is, accepting the definition). But this is precisely the problem—in my opinion—that faces it.
> [@](#):
>
> Suppose we knew that the robot’s behavior was entirely accounted for by the fact that a man inside it was receiving uninterpreted formal symbols from the robot’s sensory receptors and sending out uninterpreted formal symbols to its motor mechanisms, and the man was doing this symbol manipulation in accordance with a bunch of rules. Furthermore, suppose the man knows none of these facts about the robot, all he knows is which operations to perform on which meaningless symbols. In such a case we would regard the robot as an ingenious mechanical dummy. The hypothesis that the dummy has a mind would now be unwarranted and unnecessary, for there is now no longer any reason to ascribe intentionality to the robot or to the system of which it is a part (except of course for the man’s intentionality in manipulating the symbols).
Yes, we have a tendency to regard pure functionality as mechanical. But from “I feel this is so” does not follow “this is so”, and so Strong AI would have you question the rapidity with which you judge such mechanical beasts.
> [@](#):
>
> The formal symbol manipulations go on, the input and output are correctly matched, but the only real locus of intentionality is the man, and he doesn’t know any of the relevant intentional states; he doesn’t, for example, see what comes into the robot’s eyes, he doesn’t intend to move the robot’s arm, and he doesn’t understand any of the remarks made to or by the robot. Nor, for the reasons stated earlier, does the system of which man and robot are a part.
Neither does a single neuron, or collection of neurons, do any such thing when viewed in the manner you insist: they only manipulate and respond to chemical and electrical signals. Category. Error.
> [@](#):
>
> To see this point, contrast this case with cases in which we find it completely natural to ascribe intentionality to members of certain other primate species such as apes and monkeys and to domestic animals such as dogs. The reasons we find it natural are, roughly, two: We can’t make sense of the animal’s behavior without the ascription of intentionality and we can see that the beasts are made of similar stuff to ourselves—that is an eye, that a nose, this is its skin, and so on.
Yes: we cannot fully explain their behavior, and so we use words whose _formal meaning_ has not been established. Why should this surprise anyone?
**Rejection of behavioristic assessments of intention, etc**
> [@](#):
>
> [erl’s note: what follows in quotes is the argument put forward by a hypothetical (or rather, paraphrased) opponent]“How do you know that other people understand Chinese or anything else? Only by their behavior. Now the computer can pass the behavioral tests as well as they can (in principle), so if you are going to attribute cognition to other people you must in principle also attribute it to computers.”
This is _not_ my argument; however, it resembles it. My argument is that since these are the means at our disposal, given that their use nearly demands the assignment of intelligence etc to things we didn’t normally assign intelligence to, we should reassess our use of the word in a philosophical or psychological (should one consider a naturalistic epistemology) context, and eventually in general. With that caveat, his reply:
> [@](#):
>
> The problem in this discussion is not about how I know that other people have cognitive states, but rather what it is that I am attributing to them when I attribute cognitive states to them. The thrust of the argument is that it couldn’t be just computational processes and their output because the computational processes and their output can exist without the cognitive state.
I don’t feel this has really been demonstrated.
> [@](#):
>
> In “cognitive sciences” one presupposes the reality and knowability of the mental in the same way that in physical sciences one has to presuppose the reality and knowability of physical objects.
That’s right; and in the physical sciences, we do not multiply explantory or descriptive entities unless they are necessary to explain things. In this case, if something passes [what I woudl characterize as] the extreme Turing test (the Chinese 'Bot), then human behavior has been explained, full stop. Just because one is a cognitive scientist does not grant one the ability to put ghosts into _biological_ machines at will. Furthermore, this quote notes what I’ve been trying to point out all along: we already dismissed the problem of other minds, one of the two things I think stand in the way of Strong AI (the other being mentioned earlier with respect to the determination of meaning in a vacuum). Granting these things, the burden of proof is justifiably shifted: why _should_ your homunculus report intentional states about a reality he is not in contact with in a manner we cannot question him in?
I am not going to grant the Many Mansions reply any consideration. At that point the thought experiment goes out of control, IMO. It is, IMO, the “If things were different they wouldn’t be the same” argument. It is a useful thing to read _before_ one reads the rest of the paper, however, as Searle’s “biological” prejudices are given free territory to roam. It seems Searle’s main point is the axiom above (to wit, A4): brains cause minds. This assertion _is no less powerful and question begging_ than the Strong AI camp’s definition of ‘understands’.
This is **erl** , signing off for the night… [and, might I add: whew!]
---
### Author:  [@DSeid](https://boards.straightdope.com/u/DSeid)
#### Post date: [December 11, 2002, 6:01pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/128 "2002-12-11T18:01:48Z")
So erl, let me create a situation. The look-up table for translating Chinese are so vast that a single man in the room can not do it. You have a whole society of men in the room sharing the task according to some organized system. None of them know what the overall goal is. Each of them is just doing their job: I see this symbol I respond thus.
None of these men understand Chinese, by any stretch.
But this society of men can produce an output that translates Chinese. Does the society of men understand Chinese?
If I understand you above, then I believe that you would say yes. Others thoughts?
I would say yes. And I am comfortable substituting neurons or AI components for the men and having the anlysis hold equally.
Being clear to differentiate between your operational definition of “understanding” and “intelligence” and “consciousness”, of course. I do not think that such meets a reasonable defintion of “intelligence” nor of “consciousness”
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 11, 2002, 6:36pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/129 "2002-12-11T18:36:48Z")
> [@](#):
>
> But this society of men can produce an output that translates Chinese. Does the society of men understand Chinese?
Oh, sure! You are correct in your estimate. However, I still remain unsure that, even in Searle’s hypothetical, that the man doesn’t understand Chinese. If all these symbols are Chinese, then I think it could be safe to say the man understands what to do with them; we cannot say that he understands Chinese _like_ a Chinese speaker does, though. But why should this be a reasonable requirement for understanding (other than the inherent bias necessary to answer the question of, “Does it understand”)?
> [@](#):
>
> I do not think that such meets a reasonable defintion of “intelligence” nor of “consciousness”
And in absence of this definition (which we all admit we don’t have, except the Strong AI camp), we are left only with our normal use of the words. We don’t know that understanding is a state one _has_, a state one _is in_, a permanent physical quality or property, etc etc. We only have how the word is used in everyday contexts, and this use strongly implies the necessity of using it to describe things that pass Turing Tests. It is the unfounded, unjustified, metaphysically arbitrary “thing” (that Searle wants to say is _named_ ‘understanding’) that he can’t understand us attributing to robots (and boxes, etc). And most of us (those who aren’t moved by this argument) generally say:
(1) We’re not _attributing_ anything, we’re following consistent use of a word; or,
(2) There is nothing to attribute
(3) Whether or not there is something to attribute, we are using the word until a better method of decision, or a better definition, comes along (which the CR _does not_ offer at all, in either case).
(4) This is what the word means, so we must use it that way.
There are many of us here not motivated by the CR: does anyone not fit into one of those categories? Or who would prefer a different description?
---
### Author:  [@eponymous](https://boards.straightdope.com/u/eponymous)
#### Post date: [December 11, 2002, 10:50pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/130 "2002-12-11T22:50:02Z")
Well, it looks as though **erislover** already provided a nice link that summarizes most of Searle’s positions.
I tend to side with Searle’s objection to the Strong AI position (organizational functionalism), but I, too, am a bit confused as the meaning/definitions that Searle is employing in making his argument.
Something that strikes a cord with me (personally) with Searle’s argument against Strong AI are some of the (to me) problematic conclusions one can draw by supporting the Strong AI position. If all that required for a computer AI to “understand” Chinese if it succesfully manipulates symbols using a sufficiently large table of rules - "instantiating the program(?) - that makes any output from the computer indistinguishable from a human being, then what’s to stop one from making the claim that, for example, a thermostat “understands” that it needs to shut down at a given temperature?
I know the thermostat example is overly simplistic, but at what point do we draw the line at “this entity understands and this entity is just mimicking/simulating understanding?” Can we provide a clear and convincing definition that allows us to make a distinction? Or should a distinction even be made?
Here’s a little thought experiment:
Suppose, for the sake of argument, that a computer AI has been created that passes the Turing Test. By those who favor Strong AI, all this means is that the computer AI successfully exhibited “behavior” in its responses that is indistinguishible from human responses. So we can conclude that the computer AI “undertands”, for example, Chinese, in its responses to questions that are indistinguishable from human responses.
Know let’s suppose that this very same computer AI is sophisitacted enough to answer almost any question, on par with the capacity of the human brain. We pose the question “Does a thermostat “understand” something in the same sense that you (computer AI) understand something?”
If the computer AI answers that yes, the way it (a thermostat) understands something is the same as I (computer AI) understand it. That is, the thermostat is exhibiting behavior (it shuts down at a given temperature, thus it “understands” that the temperature is getting too high). The program or procedure a thermostat is instantiating to shut down when the temperature gets too high is and extremely limited one, but by the behavior it exhibits, one can surmise that it exhibits a level of understanding that is the same kind (though not of the same degree), as that of my (computer) AI understanding.
If one accepts the first case, then at what point would the computer AI decide that an entity doesn’t understand something? Would “instantiating some type of program” (however rudimentary) be required for the computer AI to conclude that an entity “understands” something? What justification could it give (or be programmed into the AI) in deciding whether or not a rock or water “understands” something?
If the computer AI answers no, that the thermostat doesn’t “understand” in the same sense that I (computer AI) understand something, then wouldn’t it be invoking a similar kind of objection a la Seale? That is, the way in which the computer AI understands something (instantiating a program) is somehow different from the way a thermostat “understands” something.
If one accepts the second case, then Searle’s position is vindicated. That is, the computer AI’s understanding is somehow qualitatively different from a thermostat’s understanding. And, therefore, a human being’s understanding is qualitatively different than that of a computer AI (because we can make the claim that it is). After all, if the Strong AI thesis is correct, then all we as humans are doing is just “instantiating our programs” just as a computer AI would. If part of our “program” is to reject the notion that a computer AI “understands” in the same way that humans understand, then we’re right back where we started.
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 11, 2002, 11:26pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/131 "2002-12-11T23:26:09Z")
**eponymous**
> [@](#):
>
> then what’s to stop one from making the claim that, for example, a thermostat “understands” that it needs to shut down at a given temperature?
Well, it sounds silly, but why can’t we say it? A thermostat’s IQ would be, well, pretty much 0. We wouldn’t expect to be able to meaningfully question it.
There are other examples, too, that Penrose has mentioned, questioning the manifestation of intelligent algorithms… that is, how do they need to exist in order to be intelligent? And various answers present themselves here, which can cause us to attribute intelligence to things like books (for instance).
> [@](#):
>
> If the computer AI answers no, that the thermostat doesn’t “understand” in the same sense that I (computer AI) understand something, then wouldn’t it be invoking a similar kind of objection a la Seale?
🙂 Sure! What, we don’t expect it to make mistakes? 😉 That is funny, though.
> [@](#):
>
> If one accepts the second case, then Searle’s position is vindicated. That is, the computer AI’s understanding is somehow qualitatively different from a thermostat’s understanding.
Why? Is intelligence a linear scale, for example? Is it quantifiable? If it is, then it it quantitatively different, I’ll not object. If it is qualitatively different, why do we say this based on the responses of the object in question? —especially if we are dubious about the claim of the computer understanding in the first place!
---
### Author:  [@Scupper](https://boards.straightdope.com/u/Scupper)
#### Post date: [December 11, 2002, 11:38pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/132 "2002-12-11T23:38:59Z")
> [@](#):
>
> \*Originally posted by partly\_warmer \*
> \*\*In practice, I would expect environmental, safety, and religious groups to stop artificial life that couldn’t be distinguished from real life. I simplified the argument by pointing out that one person alone, even, is capable of acting.
>
> …
>
> As the problems with it are discovered, research will be stopped. \*\*
I agree with you that it should be stopped. It will not be stopped, however. If it is _possible_, then somebody will eventually do it, mad scientist or no.
I would offer a friendly challenge to anyone to name a technology so hazardous or morally unacceptable that no one anywhere will ever take it up again. Nuclear weapons? Biological weapons? Surely we’d stop _somewhere_?
The monkeys do not stop.
The monkeys cannot stop.
That is what makes them monkeys.
That will eventually make them cease to be monkeys.
---
### Author:  [@Shalmanese](https://boards.straightdope.com/u/Shalmanese)
#### Post date: [December 12, 2002, 1:44pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/133 "2002-12-12T13:44:04Z")
On Searle:
Instead of a lone person, imagine if you were in a massive office where each person is constrained to their cubicle. You can send and recieve messages to adjacent cubicles via a slot but you never meet anybody else. In front of you, is a code book which tells you what you are supposed to do when you recieve a message.
Now, this example is almost exactly analogous to the previous one and you would be hard pressed to make a case stating that this model would obtain any more “understanding” that the classical model.
However, while the classical model is based on a computer algorithm, this model is based on the human brain. How can we attribute understanding to humans if we do not attribute them to neurons?
---
### Author:  [@Ramanujan](https://boards.straightdope.com/u/Ramanujan)
#### Post date: [December 12, 2002, 3:41pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/134 "2002-12-12T15:41:12Z")
> [@](#):
>
> I know the thermostat example is overly simplistic, but at what point do we draw the line at “this entity understands and this entity is just mimicking/simulating understanding?” Can we provide a clear and convincing definition that allows us to make a distinction? Or should a distinction even be made?
if we ever get a good definition of “understands” from someone who is arguing either way, perhaps we’ll have an answer for you.
> [@](#):
>
> Know let’s suppose that this very same computer AI is sophisitacted enough to answer almost any question, on par with the capacity of the human brain. We pose the question “Does a thermostat “understand” something in the same sense that you (computer AI) understand something?”
is a human sophisticated enough to answer that question? it seems as though we’re arguing over it now. it seems that if the computer could answer such a question truthfully, it would be a failure of the turing test.
whether or not a thermostat understands depends very heavily on how we define “understands”. if understanding is something organisms with brains do, then no, thermostats don’t understand. but i think you’d be hard-pressed to come up with a more abstract definition of “understands” that limits it to organisms with brains.
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 14, 2002, 1:46am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/135 "2002-12-14T01:46:31Z")
> [@](#):
>
> it seems that if the computer could answer such a question truthfully, it would be a failure of the turing test.
Why?
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 14, 2002, 3:41am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/136 "2002-12-14T03:41:38Z")
G.E.M. Anscombe has an interesting argument that intention is not ‘a something’ that accompanies actions (not a mental state, for example), and does so without appealing to Wittgenstein’s private language argument (though it would apply there, as well). Roughly, this would cause infinite regress, as one would have to intentionally look for intentions in order to grasp the ‘sensation’ or event, etc.
---
### Author:  [@Ramanujan](https://boards.straightdope.com/u/Ramanujan)
#### Post date: [December 14, 2002, 5:44am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/137 "2002-12-14T05:44:31Z")
> [@](#):
>
> \*Originally posted by erislover \*
> \*\*Why? \*\*
if a computer can answer a question that no human can, we can tell that it’s not human.
---
### Author:  [@Ramanujan](https://boards.straightdope.com/u/Ramanujan)
#### Post date: [December 14, 2002, 6:04am UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/138 "2002-12-14T06:04:21Z")
> [@](#):
>
> \*Originally posted by erislover \*
> \*\*G.E.M. Anscombe has an interesting argument that intention is not ‘a something’ that accompanies actions (not a mental state, for example), and does so without appealing to Wittgenstein’s private language argument (though it would apply there, as well). Roughly, this would cause infinite regress, as one would have to intentionally look for intentions in order to grasp the ‘sensation’ or event, etc. \*\*
when i was younger (say, 17 or so), before i had heard the physical arguments against free will, i decided that free will was impossible anyway. the way i reasoned this was something akin to mr. anscombe. when i make a decision, i evaluate some value i have. somehow i have to arrive at that value. suppose i have another value that allowed me to arrive at that one. at some point, though, every decision i ever made was influenced solely by environment or genetics, which are both outside my intentionality. so, for those who believe in free will, where does it come from? what is intentional, and what caused that intentional being to be that way?
to be honest, i never really believed that the intentionality argument held any water, since i fail to understand where my own intentionality comes from.
---
### Author:  [@erislover](https://boards.straightdope.com/u/erislover)
#### Post date: [December 14, 2002, 5:04pm UTC](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658/139 "2002-12-14T17:04:43Z")
But if the AI responded in that way, it would just be a Searle-based AI 😉 You see what I mean? All factual answers do not lie in our language, simply waiting to be constructed. AIs could make cognitive mistakes just like the rest of us if all they had to work with was our language.
I, too, have a hard time finding an anchor for free will.
Final note, Gertrude Anscombe was a female (and, surprisingly or not, a Catholic). She died in 2001, I believe, and most of her writings were in philosophical journals rather than complete philosophy books. The one book of hers I’ve been able to find, Intention, is an excellent continuation of Wittgenstein’s method of philosophy, but without [understanding Wittgenstein](http://www.roangelo.net/logwitt/), it might be quite intractable. His method of analysis was, I think, wholly unique, and—for me—quite convincing.
[Previous page](https://boards.straightdope.com/t/strong-ai-and-the-chinese-room/140658.md?page=6)