I agree. I think maybe you misparsed what I was saying.
I’m not convinced by Strong AI, yet I don’t claim to know it is wrong either. I was actually arguing against the POV that Strong AI is a fact, or that if you are a Physicalist, you must believe that Strong AI is trivially true.
What I’ve always thought was fun is that the Chinese Room argument is one in favor of Strong AI. It demonstrates that “understanding” is a property of the rules. The room, per se, really does understand Chinese. A mind can be built out of material components.
As I’ve been arguing regarding the rules of chess, you can “understand” the rules without formally knowing them. A person can play a very good game of chess indeed, laboring under a miscomprehension regarding the actual rules.
It’s a little like the various “black box” games extant, where you have a system which you cannot examine, but which you can probe and observe changes in behavior. In time, you form a mental model of what is inside the black box.
Your model might be entirely wrong, but, as long as the results are correct, you can be said to “understand” the system. If you can make accurate predictions on the basis of your observations, you have this understanding.
But it is not an absurd conclusion at all. He is just hung up on the person. Here are some examples. If the rules are explicit enough, you can build a fairly dumb robot to follow them as well as the man does. But no one would be surprised that the robot does not then understand Chinese. Or the man could have an extreme case of the malady in Memento, and forget everything every then minutes, but still be able to follow written instructions. He won’t ever learn enough to be said to understand Chinese, but the room would. As we’ve said before, the cpu of a computer system that understood Chinese would not understand it, and the neurons of a Chinese speaker don’t understand Chinese either.
So I think his whole “proof” boils down to incredulity that a thinking person might not understand Chinese in this situation.
But the premise is that the system understands Chinese, not the person running the program, so [3] does not follow at all. The guy who pushes the button does not understand Chinese. Neither does the power supply, the disk, the memory, or even the operating system.
Thanks for making it so clear - I agree that this is exactly his argument.
Is there an adaptive FSA I’ve never heard of? I studied them a long time ago, don’t follow that area, and haven’t seen a thesis review on the subject for a long time. If not the situations is exactly equivalent to the card case, where the cards represent states, the writing on the cards the output, and which card to choose next given the current card and the input the next state function.
I’m not clear what you mean here. Are you saying that a TM cannot put its program on the tape? That the TM cannot read and act on its own program on tape? Or that none of this leads to understanding?
I think this capacity is necessary for understanding, but likely not sufficient for understanding. I’m not claiming an understanding TM or program is possible - just that one is impossible without this capability.
The Chinese Room scenario assumes that understanding happens - but then claims that leads to an absurdity or contradiction. It is useless to argue that it is not possible as a counterargument, of course, but assuming it is, I don’t see the contradiction. I think we both agree that if the simple card system were used there would be a contradiction in that you can come up with a question something understanding Chinese could answer that the room can’t. But the man is a relatively speaking component in the system, and his understanding is irrelevant.
I was still in the Chinese room. As a physicalist, I agree, but some might want to make a distinction between what a computer and our brain can do. I think a computer can be programmed with Strong AI, but I’d hate to claim it as a fact yet.
Well, then what’s your answer to the case in which the person incorporates the room? As I’ve asked above, could he translate Chinese sentences to English, or point to an apple tree if prompted in Chinese? Because it seems evident to me that he couldn’t—all he’s thinking is along the lines ‘this symbol follows that symbol, so I must reply with those symbols’, etc.
Then of course you can take the various lines that suppose some kind of ‘emergent mind’ arises ‘on top of’ the person’s mind—as I’ve said above, this seems a reasonable stance to take. But even then, as I’ve said, the ‘ah, so that’s how it works’ just doesn’t seem to be there—I couldn’t for the live of me imagine what it would be like to be such an ‘emergent mind’.
This presupposes the capacity of forming mental models, which is of course exactly the thing that’s in question. You could just take all your data points and fit it with a mathematical curve, which you can do with a simple algorithm without any sort of actual ‘understanding’ present.
Missed the first part of this. In things like GAs the rules evolve based on the inputs, and the acceptance function, so there are no rules to anticipate. Our minds clearly evolved this way.
Not at all. I’m saying that if there is a language understanding system it must understand this, not that a language understanding system is possible. Searle makes that assumption as his premise.
And he never demonstrates that we cannot. And there have been plenty of AI systems which include concepts of lots of things - more likely boxes than trees. They don’t seem to be scalable into ones which can learn the concept of anything, though.
But in the case in which the person does all the work of the room, then the person is the system; the argument as formulated includes this case. But still, I at least cannot fathom how understanding of Chinese would arise in the person in this case.
I’m saying that any adaptive system can be emulated on a non-adaptive one—computational universality is sufficient for that. So the adaptivity does not bring anything new to the table, as exactly the same computations can be performed with an adaptive as with a non-adaptive system.
It seems to me it’s possible to construct a Chinese room thought experiment that doesn’t meet our gut-feeling of understanding, and it’s also possible to construct one that does meet our gut-feeling of understanding (I say gut-feeling because we clearly don’t have a good definition of understanding).
So it seems the lesson of the Chinese room is that we can imagine both.
But it really doesn’t help us get any closer to figuring out what the bare minimum is for understanding.
But you can model the evolution of the rules deterministically (or even probabilistically, perhaps—but probabilistic computation is not more powerful than deterministic computation), and thus, implement the GA on an architecture with fixed underlying rules (which is of course how things proceed in practice). As I said, adaptivity produces nothing new—the same things are computable with or without it.
To lead it to absurdity; the absurdity here being that even though a given system—the man with the room in his head—implements the computation that is supposed to lead to understanding, it does not understand.
Well, it was of course exactly in reply to these systems—in particular SHRDLU—that Searle first proposed his thought experiment to show that they do not have a concept of these things, but merely act as if in a purely mechanical way. The boxes, to the machine, are nothing else but particular kinds of symbols—not ink on paper or electrons in a wire, but simple geometrical objects. And they are manipulated exactly as symbols.
Speaking just for myself—and maybe it’s just my gut feeling differently from anybody else’s—I’ve never seen an example of the latter. I can accept, intellectually, that something like the virtual-minds reply might be coherent, but my gut tells me differently: viscerally, I cannot imagine what it would be like to be an ‘emergent mind’ understanding Chinese, the way I can imagine what it is like to be a person understanding Chinese.
I think I would start at the extreme:
Picture of set of mechanical neurons suspended in the Chinese room such that their connections and functionality mimic real neurons perfectly…and they are connected to input/output that perfectly mimics eyes/ears/mouth (whatever is needed)…and this collection of neurons was modeled on a Chinese person that understands Chinese.
It seems like we could assume that this version of the Chinese room probably understands Chinese (because it’s basically performing the exact same functions in the same manner as a person).
It’s not very helpful, but neither is the non-understanding case, both are opposite ends of the spectrum. We are trying to determine whether there is a line somewhere in between the two extremes that represents understanding, and I don’t think the Chinese room helps us get there other than to set the far extreme point that is not understanding.
Yes, these are familiar considerations. And as I said, I can intellectually accept that something like this ought to give rise to understanding—after all, brains seem to manage! But this doesn’t resolve my problem regarding just how this works, while I seem to be perfectly well able to imagine that there is no understanding associated with syntactic rule-following, or with mechanical causal interactions, and so on.
And besides, there is at least the possibility that understanding might rely on some non-computational (i.e. non-algorithmic, hypercomputational, etc.), non-functional (perhaps for some unknown reason, only squishy bioneurons produce consciousness—there is a position known as ‘biologism’ advocating this sort of position), non-physical (as in Chalmers’ dual-aspect theory or some other naturalistic dualism, or the position of neutral or Russelian monism) or perhaps even non-natural (maybe there is some Cartesian res cogitans after all) property—none of which would be present in these kinds of thought experiments, thus explaining the apparent impossibility of understanding understanding through them. These positions might not seem terribly attractive, but that doesn’t automatically imply they’re wrong; and just stipulating that understanding can somehow be captured computationally/functionally/physically etc. does not suffice as a response to things like the Chinese room.
The more I think about it, the weirder the ‘systems’-reply seems to me—consider for instance a sentence of 12 words, and each of these words given to a different person. Now matter how hard each person concentrated on their word, it seems highly spurious to me that there would be conscious awareness of the whole sentence anywhere—indeed, I would put such a suspicion about on the same level as speculations about group minds, telepathy, and so on.
But it seems to me that a proponent of the systems reply ought to hold that something very much like that happens: if twelve people each implement a part of a suitably parallelized ‘understanding’ program, then there would emerge a ‘group mind’ awareness of such a sentence, even without any of the people understanding their part. Each of the people would have, for instance, in their attention one particular Chinese symbol, and somehow, this produces an awareness and understanding of that sentence. If now based on the meaning of that sentence, one of the persons could be made to execute a particular action—which seems to me to be a consequence of understanding—, say point to an apple tree, then it seems we’d have something very much like telepathy, or a ‘hive mind’, indeed.
Of course, there has to be at least some information exchange between the persons—but since they individually for sure do not understand Chinese through collectively implementing the program (I think everyone’s agreed on that), the information they exchange can’t be semantic, that is, can’t be related to the meaning of the Chinese sentences they receive and produce. So they seem to be able to create a group mind, with an understanding greater than the sum of their individual understanding, merely by, for instance, talking. So then, is there a group mind associated with every instance of social interaction? Does the US have a mind, exceeding the combination of minds that make it up? And how does this differ from telepathy—it seems to be at least in the same spirit: mental content shared between two or more brains, regardless of their physical connection (which must be present, but transfers no semantic information, no mental content).
Additionally, going back to the primitive ‘lookup table’ implementation, I think we’ve overlooked a bit of weirdness there, too, at least if we wish to hold on to the idea that a lookup table doesn’t produce conscious experience. For, say, I were to implement another program using a lookup table—any program run for a finite time can be, in principle, so implemented, as there are only finitely many possibilities to consider. So let’s say I’m playing a computer game, implemented such that my inputs—key presses, mouse clicks and movements—simply call up the appropriate reaction—sounds and images displayed on the screen—via an enormous lookup table. My experience in playing the game would not be different than in the case that responses to my actions are dynamically calculated; in fact, nothing would differ between the two cases.
But not so if the lookup table is supposed to implement some program that ‘understands’ what I type, and responds appropriately. Again, on my end, I wouldn’t notice any difference—for any finite stretch of time, I can carry on a computation with the lookup table that is isomorphic to one carried out with a program that actually performs some computation to produce appropriate results. But while in the latter case, as most people in this thread here seem to argue, there is genuine understanding, there is none in the former, or perhaps only the ‘warmed-up’ understanding of who or whatever drew up the lookup table in the first place. But this is then a very real difference between the computational implementation of a mind, and the computational implementation of a computer game, or anything else: the lookup-table and dynamical-computation cases are equivalent in the latter, but not in the former case. (If someone wishes to insist that they are, because even in the former case, I’m interacting with the ‘warmed-up’ understanding of some original programmer, simply imagine the—vanishingly unlikely—case in which the computer only makes random replies, which happen to perfectly match a conversation in one, and the reactions of the computer game in the other case; the difference seems very clear here.)
Now if this is right, then the computation of a mind must be different from any other sort of computation. That would be an interesting result in itself, but I’m wondering if it can possibly be made coherent: after all, one can essentially view one system as implementing different computations, and thus, construct a mapping between the computation of the conversation, and the computation of the computer game (such a mapping exists if both state spaces are of the same cardinality). But then, under this mapping, I can view my computer implementing the game as implementing the conversation—and note that I’m not doing anything physically to it. Now let’s instantiate two computers, both of which implement the game, one via the lookup table, and the other via dynamical computation. What happens on both computers is equivalent. Now I can look at the setup via my mapping, and view one computer as implementing the conversation via a lookup table, and the other as implementing it via dynamical computation—meaning that both processes no longer are equivalent: one gives rise to understanding, and the other doesn’t. But nothing has changed physically, so the equivalence between the two computations does not seem to supervene on the physical, which strikes me as a rather strange result.
The problem with your comparison between the two scenarios is that the game program wasn’t conscious in the first place and wasn’t supposed to be, so it’s not surprising that more or less dynamic calculation doesn’t add or take away from the consciousness that was never there in the first place.
To me, your example just points out that not all computation creates consciousness.
Well, the difference between computations giving rise to consciousness and all other computations was exactly what I was trying to point out…
But you went on to say “But nothing has changed physically, so the equivalence between the two computations does not seem to supervene on the physical, which strikes me as a rather strange result.” which seemed like you concluded more than is allowed based on the example.
Your examples, IMO, are comparable as follows:
Game:
Method 1 - table - has limited dynamic processing and state
Method 2 - dynamic (with some extra internal sauce, but because we aren’t inside the program we don’t know what that feels like, inside meaning we aren’t connected to and actively sensing the shifting internal state)
Brain
Method 1 - table - has limited dynamic processing and state
Method 2 - dynamic (with some extra internal sauce, and because we are inside the program, we get it, it’s consciousness)
Well, the idea here is that I can view the computer implementing the game (either via the lookup table or via dynamical programming) as implementing the conversation via a suitable interpretation mapping—that is, without making any physical change on it. I simply need to map the states of the one computation to the states of the other—my mouseclicks, -movements and key presses are interpreted as a strange code for Chinese questions, and the audiovisual outputs of the game as Chinese responses. This mapping, as I said, exists if there’s enough states—that is, I have game inputs for all different Chinese inputs, and game outputs for all different Chinese outputs.
But if, in the scenario where I view the two computers as implementing the game, both are indeed equivalent with respect to the computational/functional details, and they cease to be so in the scenario when I view them as implementing the conversation, then, since really nothing is physically different between the two scenarios, it is indeed the case that this equivalence does not supervene on the physical. An alternative would be to view the computer implementing the game as being conscious, as well, thus embracing a panpsychist view—in this case, there would not be equivalence even if they merely implement the game.
Or you could deny that there is the relevant sort of equivalence even in the case when both computers implement the game, that lookup-table computation is somehow fundamentally different than dynamic computation. But then you’d still be left with the puzzle that in the physically identical setting in one case seems to give rise to consciousness, but not in the other.
I’m not entirely grasping what you are seeing with these examples.
I get the whole mapping of game IO to valid conversation sequences, but I’m not fully following the leap of your reasoning to the point of not supervening on the physical.
Understanding and method of conversation are two independent attributes of the systems in these examples. You could have a system that understands but decides to just use the lookup table out of laziness.
There are the external attributes and internal attributes, and we can only draw limited conclusions about the internal attributes based on the external attributes.
It’s just that by looking differently at the setup, I have consciousness in one, and no consciousness in the other case; I have equivalence in one, and not in the other case, even though nothing’s changed physically. The question of whether there is equivalence, or consciousness, then (apparently!) isn’t one decided by the physical setup.
But it’s really just a minor point, and not essentially different from the one Searle raised in his argument that basically, every physical system can be seen to implement every computation, and that hence, if computation suffices for consciousness, every physical system is conscious (and in every possible way, no less). I think that’s a point more easily refuted than the Chinese room: you simply have to find a nontrivial implementation relation, i.e. something that tells you when, exactly, a physical system can be seen to implement a certain computation. Chalmers has proposed such a relation based on the causal structure of the physical system mirroring the formal structure of the computation; I think that’s a bit much, so I prefer a somewhat more lax relation, which says that if it’s not more complicated to map the system to the computation than it is to carry the computation on its own, then you can say that the system implements the computation—because then, it does some nontrivial computational work (otherwise, all the work really is done by the mapping). This I owe to a suggestion by Scott Aaronson.
Under such a mapping, I don’t think my original argument goes through—but as I said, I think it’s only a minor point.
Sorry to be slow, but I would like to understand your point, so be patient.
“I have consciousness in one, and no consciousness in the other case” - ok
“I have equivalence in one, and not in the other case, even though nothing’s changed physically” - equivalence between what exactly? I’ve read your explanation and I’m just not seeing what you are seeing here, can you elaborate on this part?