Why ChatGPT doesn't understand

[Half_Man_Half_Wit]

@wolfpup

The title of the thread says that it’s about the claim that ChatGPT doesn’t “understand”. It seems reasonable to assume that this invites discussion about whether or not this is true in any meaningful sense of the term.

The title of the thread is “Why ChatGPT doesn’t understand”, indicating that a reason will be presented that is supposed to be the topic of this discussion. Furthermore, in the OP (which one generally assumes people read in addition to just the title), it is clarified that this thread isn’t concerned with whether arguments for understanding being present in ChatGPT succeed, but with the negative case as laid out there, by the presented argument. As that seemingly was too subtle a hint regarding the topic, I’ve patiently, if apparently fruitlessly, clarified it for you several times now.

If you want to discuss the topic of ChatGPT’s understanding from another perspective, you’re free to create a thread to do so, but please, stop hijacking this one.

@Sam_Stone

Making any kind of sweeeping conclusion about what an LLM can or can’t do, can or can’t experience, is WAY premature.

And yet, I can make innumerable such conclusions. LLMs can’t tell me whether a set of matrices, multiplied together in the right way, can yield the zero matrix. LLM’s can’t produce more than the initial few bits of an \Omega-number. An LLM can’t determine whether two context-free grammars produce the same sentences. Or determine whether a given strategy in Magic: The Gathering is a winning strategy. Find a complete theory of its own function. And many more.

I know all these things without even having to take a look at the details of the LLMs implementation, because they can be proven. Suppose you now tell me that some LLM produced, say, 10 initial bits of an \Omega-number. I would still know it couldn’t keep on producing them forever. Suppose you then came to me with, well, another 10 digits. Or 100. Or 1000: still, I would keep fast to my claim that eventually, the LLM won’t be able to correctly produce any more. Because I can prove it.

The OP contains a purported proof that there can be no understanding in LLMs. The proof starts out with the assumption that the LLM has a complete knowledge of the structure inherent in language. This will, automatically, enable it to answer any behavioral test with perfect accuracy: it will know exactly which words to use when, because there exists a model such that all the sentences it produces come out true. So none of these appeals to how much it looks like the system understands is going to hold water: that’s what I expect.

The trouble is that there is more than one model that makes all the sentences a LLM produces true, and that on the vast majority of these models, what we mean by a predicate such as ‘…is a cat’ won’t line up with what is picked out in that model by the respective relation. But there is no sense in which any of these models is preferred; so there is no determinate fact of the matter which one connects the words used by an LLM to the objects in the world. If there was some definite model, and we just didn’t know which one, then fine: ChatGPT would just use a language that has all of the vocabulary of English, but in which the words have a different meaning. But since there isn’t any particular model appropriate to its utterances, the words it uses don’t have any particular meaning.

Against this, to try and marshal arguments from various tests and tasks ChatGPT has fulfilled, is simply to misunderstand the argument being offerered: if the proof in the OP is right, then that performance is to be expected, but not indicative of any understanding whatever.

@Pleonast

I’m not sure what the distinction you’re making here is. Isn’t all knowledge a type of abstract structure?

Well, it’s a tempting idea, but if that’s the case, then ‘abstract structure’ needs to be spelled out in some way different from what’s given in the OP. Because if you just tell me that the world has a particular abstract structure in that sense, then what you’re telling me is fully logically equivalent to telling me that there is a certain number of objects in the world. So if you believe that we know more than just how many things there are, then abstract structure won’t cut it.

There have been other attempts to elucidate ‘structure’, most notably using Ramsey sentences, but they run into a different version of the same problem. I’m not aware if anybody has come up with a feasible solution that doesn’t in some way amount to a weakening of the claim that all we know is just abstract structure (such as Russell, who arguably felt contained to appeal to direct knowledge of at least some concrete structures to defend his theory).

@Pleonast

I’m having trouble understanding what the problem of intentionality is. A thought is a state of the brain. That brain state may or may not be associated to any sensations of the world by that brain.

But in what sense does ‘associated to’ imply ‘about’? Thunder is associated to lightning. If you hear thunder, you know there is lightning; thunder, in that sense, could be said to be ‘about’ lightning—it informs you that there’s lightning there, when you didn’t know that before. But to do so, you have to know how thunder is related to lightning. Somebody who’s never heard it might wonder what all that noise is about—they wouldn’t connect it to anything like huge electric discharges without some further knowledge. Thunder isn’t intrinsically about lightning.

But then, what is that knowledge but a thought that’s about how ‘thunder is associated to lightning’? But then, suppose you try to elucidate the intentionality of that thought by means of association: as we’ve seen, association isn’t enough, you need to know—i.e. have a thought that’s about—that association. So then the whole thing just iterates, falling into what’s known as the homunculus regress (which I think is one of the most underappreciated problems in cognitive science, and hence, my avatar is a representation of it): trying to explain a capacity in terms of itself.

[Half_Man_Half_Wit]

@wolfpup

Back in the days of Hubert Dreyfus expounding on the alleged limitations of AI, the murmurings in the AI community were along the lines of “the job of engineers is to build things; the job of philosophers is to explain why they’re no good”.

As for telling AI people that they’re wrong, I’ll just let that towering figure of cognitive science, Jerry Fodor, do the talking:

[I]nstantiating the same program that the brain does is not, in and of itself, a sufficent condition for having those propositional attitudes characteristic of the organism that has the brain. If some people in Al think that it is, they’re wrong. As for the Turing test, it has all the usual difficulties with predictions of “no difference”; you can’t distinguish the truth of the prediction from the insensitivity of the test instrument.
[…]
To say that a computer (or a brain) performs formal operations on symbols is not the same thing as saying that it performs operations on formal (in the sense of “uninterpreted”) symbols. […] If there are mental representations they must, of course, be interpreted objects; it is because they are interpreted objects that mental states are intentional.

The topic of the thread is an argument to the effect that the symbols as used by ChatGPT aren’t interpreted in this sense.

[wolfpup]

@Half_Man_Half_Wit

As for telling AI people that they’re wrong, I’ll just let that towering figure of cognitive science, Jerry Fodor, do the talking:

That link doesn’t work for me as it takes me to a Z-library login page. But the comments you cite are from Fodor’s reply to yet another skeptical paper by the infamous John Searle [PDF]. The parts that you left out (the most interesting bits of which I bolded) are illuminating.

On the first part, the potential issues with the Turing test have been well recognized by AI researchers, which is why tools like the Winograd schemas have been implemented – and in which domain ChatGPT performs very well. But of particular interest here is that the first paragraph you quote is then followed by a great big “However …”. Fodor writes (bolding mine):

However, Searle’s treatment of the “robot reply” is quite unconvincing. Given that there are the right kinds of causal linkages between the symbols that the device manipulates and things in the world – including the afferent and efferent transducers of the device – it is quite unclear that intuition rejects ascribing propositional attitudes to it. All that Searle’s example shows is that the kind of causal linkage he imagines - one that is, in effect, mediated by a man sitting in the head of a robot – is, unsurprisingly, not the right kind.

On the second point, this is Fodor’s comment in full (again, bolding mine):

To say that a computer (or a brain) performs formal operations on symbols is not the same thing as saying that it performs operations on formal (in the sense of “uninterpreted”) symbols. This equivocation occurs repeatedly in Searle’s paper, and causes considerable confusion. If there are mental representations they must, of course, be interpreted objects; it is because they are interpreted objects that mental states are intentional. But the brain might be a computer for all that.

Recalling that the context of these comments is Fodor’s rejection of Searle’s Chinese room argument (and, as I recall, you reject it as well) one must conclude that what Fodor is saying is that when there is unequivocal evidence of understanding, as there presumably is in the Chinese room “system” in its entirety – then the symbols are, in fact, being interpreted in exactly that sense. The cause of ChatGPT’s apparent failures in understanding, then, are not due to the absence of mental representations, but to something else, and I’ve proposed that that “something else” is often due to an inadequate model of the physical world, though there are other reasons as well.

[Half_Man_Half_Wit]

@wolfpup

Recalling that the context of these comments is Fodor’s rejection of Searle’s Chinese room argument (and, as I recall, you reject it as well) one must conclude that what Fodor is saying is that when there is unequivocal evidence of understanding, as there presumably is in the Chinese room “system” in its entirety – then the symbols are, in fact, being interpreted in exactly that sense.

No, that is very much not what Fodor is saying. Rather, he is agreeing with Searle that in the setting of the Chinese room, as originally proposed, there would be no understanding, because the symbols lack the right sort of causal connection to the world, which would, presumably, be available to a robot. (Even in the part you quote, he grants that Searle shows that the connections in his example are not of the right kind, i.e. that the Chinese room argument correctly establishes that even though the system instantiates the right sort of computation, and fulfills all sorts of behavioral tests, there will be no understanding present.)

[wolfpup]

Fine, if that was Fodor’s position, then it was in distinct opposition to the majority of the AI community, which has generally sided with the “system reply” refutation of the Chinese room argument. Fodor, as I’m sure you’re aware, was renowned as a philosopher and cognitive scientist and made important contributions to the computational theory of mind, but was sometimes at odds with AI researchers.

[Half_Man_Half_Wit]

@wolfpup

Fodor, as I’m sure you’re aware, was renowned as a philosopher and cognitive scientist and made important contributions to the computational theory of mind, but was sometimes at odds with AI researchers.

So, just to help me triangulate your position. When you perceive Fodor’s view to align with yours, and somebody dares criticize him, it’s

@wolfpup

He was a towering figure in cognitive science who I daresay greatly overshadowed his various critics.

But once your views diverge, he can be just dismissed by a vague wave in the direction of unspecified ‘AI researchers’?

I should think that the views he holds would at least demand a fair hearing.

[Pleonast]

@Half_Man_Half_Wit

@Pleonast

I’m not sure what the distinction you’re making here is. Isn’t all knowledge a type of abstract structure?

Well, it’s a tempting idea, but if that’s the case, then ‘abstract structure’ needs to be spelled out in some way different from what’s given in the OP. Because if you just tell me that the world has a particular abstract structure in that sense, then what you’re telling me is fully logically equivalent to telling me that there is a certain number of objects in the world. So if you believe that we know more than just how many things there are, then abstract structure won’t cut it.

I guess this touches the root of my problem: I’m not convinced we know anything. Brains (or any sufficiently advanced computational device) only have abstract (that is, non-concrete) structures and do not have knowledge. (I know there’s a huge body of philosophy about “knowledge”. I’m not convinced by any that I’ve read are useful descriptions of the world. I use “knowledge” in the more general sense of “amorphous set of abstract structures”.)

@Half_Man_Half_Wit

@Pleonast

I’m having trouble understanding what the problem of intentionality is. A thought is a state of the brain. That brain state may or may not be associated to any sensations of the world by that brain.

But in what sense does ‘associated to’ imply ‘about’?

None at all. By “associated”, I mean only that a particular brain state has a causal link to that sensation. There may also be brain states that are not associated with any sensations.

[Half_Man_Half_Wit]

@Pleonast

I guess this touches the root of my problem: I’m not convinced we know anything.

Hmm. Do you mean in the sense that none of our beliefs are true with any certainty, or that we don’t even have any beliefs at all? (If the latter, how isn’t that a self-defeating position?)

@Pleonast

I use “knowledge” in the more general sense of “amorphous set of abstract structures”.

I don’t know what that’s supposed to mean or how it is knowledge in a ‘more general sense’, sorry.

@Pleonast

By “associated”, I mean only that a particular brain state has a causal link to that sensation.

Well, but in general, if I have a thought, it seems to be a thought about something, no? If I think about a table, then there’s something my thought is about. It might not be, as one would naively think, a physical table out there in the world, it could be a concept, or a Platonic ideal, or what have you—but there seems to be something, whatever it may ultimately be.

And I’m fully on board with the possibility that the vernacular here is misleading, and it’s not quite right to talk about at thought being about something; but even a skeptical theory has to account for the data, i.e. that it seems to be about something. So how does that come about?

[Pleonast]

I probably hang out with the wrong sort of philosophers, who are always expounding on how knowledge is something like a “justified, true belief”, while I don’t find that definition useful.

In your structure framework, I’d call each ordered tuple <D, R> a piece of knowledge, and the structure W a set of knowledge. Without limiting knowledge to only that framework.

A thought can only refer to other states of the brain. There are no tables in our brains, of course, only thoughts. Yes, the vernacular is misleading, because our brains conflate external things with internal thoughts. This comes about because the evolutionary purpose of a brain is to provide useful reactions to the world. There’s little need to make an internal distinction between a thought and an external thing.

My larger problem is that while brains have states, there’s no evidence that a mathematical description of them is useful. (Note that I’m not restricting a thought, whatever a thought is, as the only type of brain state.) Any treatment of brain states that tries to encapsulate them within a particular mathematical formulation is already failing to describe actual brains, which are not logically consistent.

And that’s the inherent problem: mathematics (including logic) is not a useful tool for describing an inconsistent system.

[Half_Man_Half_Wit]

@Pleonast

In your structure framework, I’d call each ordered tuple <D, R> a piece of knowledge, and the structure W a set of knowledge. Without limiting knowledge to only that framework.

The trouble is that in this case, each piece of knowledge would just be of bare quantity. It doesn’t seem to be the case that all we know is just the number of things.

@Pleonast

A thought can only refer to other states of the brain. There are no tables in our brains, of course, only thoughts.

Well, brains are also just things in the world, and as with all things, their boundaries are only approximate; they interact with a lot of other things, and it’s not always clear where the brain ends and the ‘world’ starts. So states of the brain are states of the world (or of a thing in the world), and I’d think it’d be strange if there were some hard-and-fast boundary in the mind that can’t really be defined in the world.

But whether brain states or things in the world (or Platonic objects, or…), how thoughts can be about anything at all is still a difficult question; wherever one thinks to find such aboutness in the world (such as in a word, or a picture, or whatever), ultimately, one finds it originating within the mind that reads the word or interprets the picture, so there don’t readily seem to be any things in the world that are just about other things—except brain states. So what makes them special?

@Pleonast

Any treatment of brain states that tries to encapsulate them within a particular mathematical formulation is already failing to describe actual brains, which are not logically consistent.

And that’s the inherent problem: mathematics (including logic) is not a useful tool for describing an inconsistent system.

I’m not sure I can believe in actually true contradictions of the A-and-not-A-type, but there’s people who do (well, mainly Graham Priest, I suppose), and there are systems of logic and mathematics designed to accommodate inconsistencies.

[Pleonast]

@Half_Man_Half_Wit

@Pleonast

In your structure framework, I’d call each ordered tuple <D, R> a piece of knowledge, and the structure W a set of knowledge. Without limiting knowledge to only that framework.

The trouble is that in this case, each piece of knowledge would just be of bare quantity. It doesn’t seem to be the case that all we know is just the number of things.

Well, sure. I’m not a proponent of that framework; I’m only explaining my concept within it.

@Half_Man_Half_Wit

But whether brain states or things in the world (or Platonic objects, or…), how thoughts can be about anything at all is still a difficult question; wherever one thinks to find such aboutness in the world (such as in a word, or a picture, or whatever), ultimately, one finds it originating within the mind that reads the word or interprets the picture, so there don’t readily seem to be any things in the world that are just about other things—except brain states. So what makes them special?

I’m denying that “aboutness” is special. It’s simply another brain state.

@Half_Man_Half_Wit

I’m not sure I can believe in actually true contradictions of the A-and-not-A-type, but there’s people who do (well, mainly Graham Priest, I suppose), and there are systems of logic and mathematics designed to accommodate inconsistencies.

I’m not familiar with these. Do they handle situations where “statement-A” could be used to infer “statement-B” which could be used to infer “statement-not-A”, but the system does not make either or both inferences?

[wolfpup]

@Half_Man_Half_Wit

No, that is very much not what Fodor is saying. Rather, he is agreeing with Searle that in the setting of the Chinese room, as originally proposed, there would be no understanding, because the symbols lack the right sort of causal connection to the world, which would, presumably, be available to a robot.

First of all, going back to this point again, Fodor did, in fact, endorse the “system reply” refutation to Searle. If you have a problem with that claim, take it up with the Stanford Encyclopedia of Philosophy. His “robot reply” appears to be a later refinement of that idea in response to further arguments from Searle. I myself see no difference whatsoever between semantics attached to words by virtue of real-world experience versus semantics that are attached to words by various forms of more abstract learning, such as Reinforcement Learning from Human Feedback (RLHF) which was so effectively used in ChatGPT.

@Half_Man_Half_Wit

But once your views diverge, he can be just dismissed by a vague wave in the direction of unspecified ‘AI researchers’?

Wut?? Being a highly respected figure in cognitive science does not equate to being omniscient about AI. One can respect someone for being a significant contributor in their field without necessarily agreeing with every position they hold, particularly in fields other than their specialty. When I criticized you for dismissing Fodor’s views on the computational theory of mind, it was because what you were dismissing was absolutely central to the entire body of work that he was so highly regarded for.

As for “unspecified AI researchers”, the system reply and the very closely related “virtual mind reply” are associated not only with Marvin Minsky, often regarded as the father of AI, but with a whole host of notable philosophers:

Ned Block was one of the first to press the Systems Reply, along with many others including Jack Copeland, Daniel Dennett, Douglas Hofstadter, Jerry Fodor, John Haugeland, Ray Kurzweil and Georges Rey. Rey (1986) says the person in the room is just the CPU of the system. Kurzweil (2002) says that the human being is just an implementer and of no significance …

[Half_Man_Half_Wit]

@wolfpup

If you have a problem with that claim, take it up with the Stanford Encyclopedia of Philosophy. His “robot reply” appears to be a later refinement of that idea in response to further arguments from Searle.

I don’t know where Fodor ever endorsed the ‘systems reply’, but I do know that if he did, it must have come after his endorsement of the ‘robot reply’, because the article is a direct reaction to Searle’s original presentation of the Chinese Room in Behavioral and Brain Sciences. The format of that journal is such as to publish ‘target articles’, combined with reactions to them; Fodor’s endorsement of the ‘robot reply’ was one of those.

@wolfpup

When I criticized you for dismissing Fodor’s views on the computational theory of mind, it was because what you were dismissing was absolutely central to the entire body of work that he was so highly regarded for.

But that’s what you’re doing: recall that Fodor’s maxim was ‘no computation without representation’, and it is exactly the question whether there is any representational aspect to the words produced by ChatGPT that’s the issue. Saying that understanding could come about just by means of rule-following amounts to a complete negation of Fodor’s views.

So, again: Fodor’s view, at least in that original article, was that there would not be any understanding in the Chinese Room, as it is not hooked up to the world in the right way, no matter how much ‘engineer-understanding’ it might demonstrate. It seems to me that by these lights, one should at least take the possibility seriously that the engineer-concept is not the right one when talking about understanding in AI, if one doesn’t want to dismiss what is absolutely central to the entire body of work Fodor was so highly regarded for.

[Half_Man_Half_Wit]

@Pleonast

I’m denying that “aboutness” is special. It’s simply another brain state.

I don’t see how that would work.

@Pleonast

Do they handle situations where “statement-A” could be used to infer “statement-B” which could be used to infer “statement-not-A”, but the system does not make either or both inferences?

Not sure what you mean here. If A implies B which implies not-A (which is the sort of thing these logics deal with), then the system does make these inferences—how would they imply one another otherwise?

[Pleonast]

@Half_Man_Half_Wit

@Pleonast

I’m denying that “aboutness” is special. It’s simply another brain state.

I don’t see how that would work.

There’s no work to do. I don’t see any explanatory utility in distinguishing “aboutness” brain states from other brain states.

@Half_Man_Half_Wit

@Pleonast

Do they handle situations where “statement-A” could be used to infer “statement-B” which could be used to infer “statement-not-A”, but the system does not make either or both inferences?

Not sure what you mean here. If A implies B which implies not-A (which is the sort of thing these logics deal with), then the system does make these inferences—how would they imply one another otherwise?

Think of it this way. A particular brain has a state with statement-A and statement-B. But statement-B implies statement-not-A, so we know that brain is in a logically inconsistent state. This is not unusual, because brains do not follow all possible logical paths to their conclusions, and so may have logical inconsistencies.

Any description of a brain needs to allow for statements to not have inferences made from them, because brains do not always make inferences they could.

[Half_Man_Half_Wit]

@Pleonast

I don’t see any explanatory utility in distinguishing “aboutness” brain states from other brain states.

Well see, that just seems to get things the wrong way around, to me: the ‘aboutness’ isn’t the explanans, it’s the explanandum: mental states have, or seem to have at any rate, intentional properties. How come?

@Pleonast

Any description of a brain needs to allow for statements to not have inferences made from them, because brains do not always make inferences they could.

Ah OK, yes. I think generally, the phenomenon that brains don’t derive all the consequences entailed by their knowledge is known as the problem of logical omniscience. I think the most common approach there is to point to the fact that the mere relation of implication doesn’t pay any attention to bounded resources, i.e. the complexity of deriving what can be derived.

I’m not aware of any attempts at connecting this with paraconsistent reasoning, but it’s not an area I know much about.

[Cervaise]

As Chatbots Spread, Conservatives Dream About a Right-Wing Response

This is a pretty bad article — instead of properly educating the reader on how a language-modeling program like this works, it touches briefly on the underlying mechanics and then quickly reverts to a more familiar form of neutral sports-caller commentary on the partisan bickering — but if one reads between the lines one can recognize confirmation of this thread’s thesis statement. ChatGPT is not generating anything new; it’s just regurgitating our existing corpus of written material back to us in mutated form. To whatever extent the bot produces politically-biased “thoughts,” it is not reflecting bias in programming or as a result of machine “thinking,” it’s simply reproducing the bias already extant in its human-origin source texts, a bias which can easily be manipulated by selecting different sources or enforcing a specific emphasis on certain sources over others. That, in and of itself, is ample demonstration of what the language machine is, and is not, actually doing, and it’s unfortunate that the media “explainers” are missing this golden opportunity to break open the black box for their readers.

[DPRK]

If you spend thirty seconds (today) playing with it, it quickly becomes clear it does not “understand” anything. Furthermore, it has been demonstrated how tuning enables one to generate text in the style of a particular author or genre or political party. IMO theoretical analysis of possible “understanding” vs the specific underlying model would be more interesting for the readers.

[pulykamell]

@DPRK

If you spend thirty seconds (today) playing with it, it quickly becomes clear it does not “understand” anything

Well, you’re a sharper knife than me. I’ve been playing with it since December, and while there are times it doesn’t “understand”, I’m absolutely flabbergasted still at how well it makes sense of anything I type, no matter how casual or convoluted. It may not be “understanding,” but it’s a hell of a good simulacrum in most cases. The first ten seconds I was playing with ChatGPT were probably spent trying to pick my jaw off the floor. And I’m still continually surprised and amazed by it.

[DPRK]

I asked it for some things that can be described in text yet require some thinking, or at least “thinking”, like designing a cryptographic puzzle, an electronic circuit, or philosophical or mathematical concepts. You get superficially plausible output, but, to give one example, the letters in a scrambled letter puzzle are not the same set of letters in the secret word. Or it will happily describe a polygon with five sides (two of which are parallel and two of which are non-parallel) and four vertices.