Why ChatGPT doesn't understand

Over in Cafe Society, the thread on OpenAI’s Large Language Model (LLM)-based chatbot ChatGPT has developed into a lengthy discussion on whether it (and any similar LLM) can be said to understand language, in a way similar to a human being. That thread was mainly concerned with the arguments in favor of that conclusion, and whether they are sufficient. Here, I want to make the negative case: outline an argument according to which LLMs are constitutionally unable to acquire any kind of understanding of language, and rather are what has been called ‘stochastic parrots’ that are basically just a souped-up version of your phone’s predictive text feature, producing a continuation of an input string by means of statistical patterns learned from large volumes of text.

The argument essentially establishes the following: the relationships of terms (names etc.) that can be abstracted from language (a kind of structure) don’t suffice to figure out how those terms are supposed to map to objects in the real world. With this structure being all that ChatGPT has at its disposal, it thus can’t figure out what anything means.

Note that this is not an argument against AI understanding, as such. There could, in principle, be ways of creating AI whose operation is sensitive to the semantic value of the tokens it manipulates. It’s just that LLMs aren’t that way.

So, the important notion in the argument is that of structure. A structure W is an ordered tuple <D, R>, consisting of a domain D and a relation R. The domain is just a set of distinguishable, but otherwise unspecified objects. For concreteness, let’s say D = {‘Alice’, ‘Bob’, ‘Charlie’}, where the single-quotes are there to alert the reader that these are intended to be names, not the objects themselves.

A relation is a set of ordered tuples, for example R = {<‘Alice’, ‘Bob’>, <‘Alice’, ‘Charlie’>, <‘Bob’, ‘Charlie’>}. Such a tuple can standardly be represented in the form of a set distinguishing the first element of the pair, i.e. <‘Alice’, ‘Bob’> = {{‘Alice’}, {‘Alice’, ‘Bob’}}. With this, we can now prove the following:

Newman’s Objection. Knowledge of a structure W = <D, R> is equivalent to knowledge of the cardinality |D|, i.e. the number of elements within the domain (and nothing more).

This is named after the mathematician Max Newman, who put it in this form:

Any collection of things can be organised so as to have the structure W [where W is an arbitrary structure], provided there are the right number of them. Hence the doctrine that only structure is known involves the doctrine that nothing can be known that is not logically deducible from the mere fact of existence, except (‘theoretically’) the number of constituting objects.

Newman raised his objection in 1928 against Bertrand Russell’s proposal that all scientific investigation tells us about the world is its structure (in the indicated sense). A thorough discussion can be found in this paper.

The argument goes as follows. Suppose we know that there is the structure W. Then, in particular, we know that there is the domain D. Likewise, then, every subset of D must exist. Then, the powerset, P(D), the set of all subsets of D, also exists. But then, every subset of P(D) also exists. But then, every relation R’ over D also exists, as all that a relation over D is, is a set of subsets of P(D). So then with W, every W’ = <D, R’> exists just as well. But that already follows from the fact that D exists. So, telling me that not only D, but W exists, tells me nothing new: that’s just equivalent to saying that D exists. W comes right along, and so does every W’.

But D is completely specified by merely the number of objects within it: after all, their sole property is being distinct from one another. So, all that I need to know to specify D, hence, all that knowing that there is the structure W can tell me, is that there are |D| objects.

As an objection against the idea that mere structure can tell you anything substantial about the world, this is decisive. Moreover, while remaining within the structural, there is nothing that can be done to patch things up. For suppose you were to claim that W is, in some way, ‘special’, that it is what ‘really exists’, or whatever. One possibility of doing so is to refer to the extension of R, i.e. explicating by virtue of what R holds of objects. For instance, suppose it’s the relation ‘taller’: then R holds because Alice is taller than Bob and Charlie, and Bob is taller than Charlie. But then, you’ve also told me something about the domain that doesn’t come down to structure, namely, that the objects have a certain size, and what relative sizes there are: you’ve left the realm of pure structure.

Then, suppose that you’re trying to single out R as special by purely structural means. This means you’re telling me that there is some structure W’’ by virtue of which R is singled out. But then, we’re back at the beginning: this is ‘just more theory’, and hence, vulnerable to the same objection.

Now, as far as metaphysics goes, this isn’t much of a threat for something close to the Russellian picture. We can, for instance, hold that we have non-structural knowledge of at least our own, mental properties, and ground relations there. But for ChatGPT, that’s going to be a bigger problem.

Suppose that, from its training data, ChatGPT has learned of the three terms in D, and that they stand in a relation C = {<Alice>, <Bob>}. Now we need to find an interpretation—a model—of this, such that this correctly mirrors the relevant part of the world. One such model is the following: ‘Alice’ refers to Alice, ‘Bob’ refers to Bob, and ‘Charlie’ refers to Charlie. Alice is a cat, Bob is a cat, and Charlie is not a cat. (Names without the single quotes refer to objects ‘out there’, in the world.) Thus, C is the predicate ‘…is a cat’. Can ChatGPT correctly identify this as the ‘proper’ model of the structure it has learned?

The answer, thanks to the Newman objection, is no (in this form, the argument is due to Hilary Putnam and known simply as ‘the model-theoretic argument’). We can appeal here to a construction that, in model theory, is known as ‘Push-Through’: given a structure over some domain D, a domain E such that |D| = |E|, we can imbue E with exactly the same structure. In particular, we can just permute the elements of D, using the permutation h(D) such that h(Alice) = Bob, h(Bob) = Charlie, and h(Charlie) = Alice. (Note that we’re doing this on the things in the real world, i.e. where the ‘model’ of the sentences ChatGPT produces lives.)

With that, the relation C becomes h(C) = {<Bob>, <Charlie>}. But this model makes all the same sentences true as before. The sentence ‘Bob is a cat’, in D, refers to Bob, who is in the extension of the relation C, and is a cat. The same sentence, in h(D), now refers to Charlie, which is in the extension of the relation h(C) (making the sentence true), but is not a cat. Hence, C picks out cats, but h(C) doesn’t; but there is no fact of the matter regarding which is the ‘right’ relation. Both allow absolutely truthful sentences.

To put this in somewhat different terms, there are two levels, the level of language (or ‘theory’), and the level of objects out there in the world (or ‘model’). ChatGPT only ‘knows about’ the language level, where there is the domain, given by the terms (‘names’) of the language, and their relations—the structure it has learned throughout its training (we can here assume that it learns the structure perfectly, even though that’s clearly not true of the real-world ChatGPT). Out there, in the world, there are objects (three in the example, Alice, Bob and Charlie without single quotes—the things, not their names), which will instantiate a number of relations, as discussed above (23 - 1 = 7 non-empty one-place relations, 23*3 - 1 = 511 two-place relations, and so on). The relation C is one out of the 7 one-place relations. These are:

R1 = {<Alice>}
R2 = {<Bob>}
R3 = {<Charlie>}
R4 = {<Alice>, <Bob>}
R5 = {<Alice>, <Charlie>}
R6 = {<Bob>, <Charlie>}
R4 = {<Alice>, <Bob>, <Charlie>}

R4 is the relation ‘…is a cat’, in the real world: it names things that, in fact, are cats. But there is no way to single it out among the other two options. All ChatGPT knows is that the tokens ‘Alice’ and ‘Bob’ stand in relation C. It does not know that ‘Alice’ is supposed to refer to Alice, and ‘Bob’ is supposed to refer to Bob, and thus, does not know that C is supposed to single out cats. It doesn’t—it can’t—know what it means to be a cat, even if it can utter correctly that ‘Bob is a cat’, because that is the sort of thing it does with elements of the relation C. But whether that is supposed to mean that Bob is a cat, or that h(Bob) = Charlie is whatever’s picked out by h(C), simply isn’t available to it.

This is depicted in the following picture:

There, the ‘Language’ level is what ChatGPT has learned from its training data, which allows it to frame sentences like ‘Bob is a cat’—it knows that the token ‘Bob’ is the sort of thing that ‘…is a cat’ should be said of. The arrows then depict two different ways for the terms to refer, the pink, unbroken ones corresponding to the one in which ‘…is a cat’ ends up picking out cats, and the orange, broken ones one in which ‘…is a cat’ picks out whatever h(C) is supposed to be. (Note, again, that whatever h(C) might be, the Newman argument ensures us that it exists, purely as a set of subsets of the powerset of {Alice, Bob, Charlie}.)

ChatGPT can never pick out which is supposed to be the ‘correct’ interpretation, and thus, can never know that Bob is, in fact, a cat. There’s just not enough information there, in the structure, which after all settles only questions of quantity. There is, simply, no fact of the matter whether, when ChatGPT produces the sentence ‘Bob is a cat’ it means, in fact, that Bob is a cat, or that Charlie is a h(C)-sort of thing—in both cases, it is saying something equally true.

The argument here was presented for a simple domain and hence, structure. But it can be equally well carried out for an arbitrary one: as long as ChatGPT only knows words and their relations, it will never be able to infer from there anything about things and their relations: it produces sentences without any connection to states of affairs—it understands nothing.

Newman’s objection is far from being a settled argument (see here [PDF] and here [PDF], for instance). Moreover, it goes much further into the weeds of metaphysical debate than I think most of us would care to venture, and is probably not going to get a lot of traction here.

What I think would be more productive, if you don’t object, is to raise the discussion about the question of ChatGPT’s ability to “understand” to more concrete terms. I’ll begin with a point I made in the other thread, which I think is especially pertinent here:

I think at least some of the disagreements in these discussions stem from the fundamentally different world views that exist in engineering versus philosophy. Perhaps the most salient question here is not so much which side is “right”, but rather, which world view provides the better predictive value for the future of AI.

I think the distinction is important because the two world views are so different that they don’t even share the same language. To a philosopher, “understanding” is a sort of epistemic state connected to the physical world. To the AI engineer, however, “understanding” is a straightforward behavioural quality wherein the AI exhibits – in every discernable way – intelligent responses to the facts presented to it. This, in turn, is an important distinction because the latter implies that understanding (i.e.- intelligence) is not an intrinsic quality of a system’s internal implementation, but entirely a quality of its observable behaviour. It implies the viability of different substrates for the implementation of human intelligence and, ultimately, superhuman intelligence.

Those who disagree with this conclusion are those who have always argued that AI is not “real” intelligence, and, for the past 50 years, have been constantly moving the goalposts redefining what “real” intelligence supposedly is. Every time there’s a significant advancement in AI, the goalposts move again. Eventually, the skeptics are going to run out of land for any further goalpost movement, and fall into the sea.

Sure. As noted in the OP, as a metaphysical question regarding scientific realism, there are various ways to contend with it, although it is still, even charitably speaking, an open problem for epistemic structural realism. (And just parenthetically, the second of these papers doesn’t attack Newman’s objection, but accepts it, and proposes that Russell reformulated his views accordingly, by postulating a ‘direct acquaintance’ with at least some relations—i.e., knowing their intensions, rather than merely their extensional characterization, which is basically what I noted above. The first, on the other hand, depends on a notion of finiteness that’s difficult to square with what physics tells us about the world. Neither of them attack the logic of the objection, which is all we need here.)

But we’re not doing metaphysics here, where what our capabilities are and what our access to the world is, is open to debate. We know exactly what ChatGPT has access to: large amounts of text. There is nothing that’s discoverable from that that’s not terms and their relations. So Newman’s argument and Putnam’s argument applies in a very sharp form.

I can, of course, understand that that’s too abstract for many to bother with. That’s fine. This is just an offer: here’s why ChatGPT doesn’t understand, and if you care about that, either you’ll have to dig in, or just accept that the matter isn’t settled so simply as to point to how it seems to understand.

There’s I think a decent notion of ‘understanding’ present in the OP, which just boils down to knowing what words refer to, and hence, what any given text talks about. It seems to me that’s a perfectly ordinary notion, not something way off in philosophical la-la land: it just means that when you use the word ‘dog’, you’re talking about dogs. If you feel that there’s some relevant ‘engineering’ concept of understanding that somehow doesn’t boil down to that, you’re free to open a thread and make that case; this one is about this concept.

By this definition, ChatGPT certainly understands dogs.

Ok, so what do you think goes wrong in the argument in the OP? Granted, I used ‘…is a cat’ as an example, but I don’t think that’s a salient difference… At any rate, the construction explicitly shows how ChatGPT could utter the sentence ‘Bob is a cat’, which comes out true, yet has a model in which it refers to Charlie, which isn’t a cat. Why do you think that wouldn’t apply?

It could, but it generally doesn’t, because of the data it was trained on. However, I’m not questioning your conclusion there - I’m not weighing in on that yet.

I’m just pointing out that by the specific criteria you set:

Generally, if you talk to it about a dog, it responds meaningfully, with sentences that make sense to use in context about dogs. To me, that meets the criteria laid out by you above.

But that’s exactly what the argument in the OP is intended to show doesn’t suffice to decide whether it’s talking about a dog. Again, for a sentence like ‘Bob is a cat’, there is model—an association of things in the world to the terms used—such that the sentence is true, but what it talks about isn’t a cat. It’s just that the name ‘Bob’ is mapped to the individual Charlie, which isn’t a cat, and the predicate ‘…is a cat’ is mapped to a relation that doesn’t pick out cats, but which is true of Charlie. The same goes for anything further predicated of ‘Bob’, such as ‘…has four legs and a tail’. So just saying things like ‘Bob is a cat and has four legs and a tail’ doesn’t entail that ChatGPT knows anything about cats whatsoever.

Fwiw, I just asked it:

What weighs more, an imperial pound or a dog pound?

And got:

An imperial pound weighs the same as a dog pound in terms of mass, which is approximately 454 grams or 16 ounces. However, they are used to measure different things - an imperial pound is a unit of weight or mass in the British system of measurement, while a dog pound is a unit of weight used to measure the weight of dogs in the United States. So, although they have the same numerical value, they are not interchangeable and are used in different contexts

Which feels a lot more like a stochastic parrot than a conversation with an entity that knows what a dog is.

That’s true. But, so what?

Take colors. Let’s say that you suffer from a rare genetic mutation that means you see red as blue and blue as red.

We go to an art gallery together. Is there any way for us to determine that your perception of blue and red is different from mine?

I would argue that no, no there is not. If I point to a patch on a painting and ask you what color it is, we would both agree that the patch is “red”. However, you would be seeing blue, not red. But every time in your life when you have seen blue, you were told that the color you’re looking at is red. And in fact, there is absolutely no way for us to communicate this incongruity to each other. You wouldn’t even know anything was wrong.

Does that mean that you or I don’t understand what the color “red” is?

Nope, not understanding on my behalf there.

That’s the inverted qualia-thought experiment. But it doesn’t have anything to do with the case at hand: when I talk about ‘red things’, I will be talking about entirely the same things as you are, because we understand what red is, out there in the world—even if what it is like for me to see red might be different from what it is like for you. My ‘red car’ refers to the same object as your ‘red car’.

But ChatGPT won’t be talking about those things when it talks about red things. In fact, there’s no determinate fact of the matter regarding which things it’s talking about, as long as there are the right number. ChatGPT’s ‘red car’ need not be red, nor a car, at all: there is an interpretation that makes it come out to be virtually anything.

There is, admittedly, a great deal about the world that ChatGPT doesn’t know – or more accurately, has not yet learned, and it’s notorious for just making stuff up. This is not intrinsic to how such an AI works, however; DeepQA (the Watson engine) is very good about scoring its own responses with confidence ratings.

The best argument for understanding in ChatGPT I think is its ability to solve problems in logic. There were many impressive examples in the thread over in CS. For instance, a sequence of numbers in which it’s asked to find the pattern – it succeeded not only with the simple example I posted (each number in the sequence is increasing by 3) but also in more complex patterns I tried on it later in which the pattern is defined by a non-monotonic function. It succeeded in solving analogies, which are routinely used in IQ tests (obviously, I couldn’t give it visual pattern analogies, but it solved verbal analogies; IQ tests typically have both kinds). It correctly solved a logic problem I gave it that was a bit convoluted and that I struggled to phrase as clearly as I could; it understood it perfectly and gave the correct answer. And it solved what I referred to over there as the “fish question”, wherein a man has five fish and they’re described in relation to each other’s weight – it solved that one by setting up the appropriate equations and then solving the equations. I’ve never seen a parrot do that!

ChatGPT has also solved “theory of mind” problems that demonstrate at least the intrinsic intelligence of a nine-year-old child. It also dealt with questions that someone posted in which they made up a world inhabited by beings attributed with made-up behaviours, and was questioned about the logical outcomes of certain scenarios.

At this point, given this clear demonstration that ChatGPT is correctly solving logical problems of precisely the kind that we routinely use to test intelligence (and obviously understanding them) the claim that it possesses none of those qualities seems rather specious at best. Granted that it’s not a towering intellect, to be sure – in some respects no better than a nine-year-old child judging by its “theory of mind” performance, but solving these problems cannot be done with mere token-matching or “stochastic parroting”. The fact that it has evident flaws and sometimes says nonsensical things is not of particular relevance, because people have those flaws, too, only they usually manifest differently.

I asked the exact same question and got this answer:

An imperial pound and a dog pound are not comparable in terms of weight because they are completely different things.

An imperial pound is a unit of measurement used to measure weight and is equal to 16 ounces or approximately 0.45 kilograms.

A dog pound, on the other hand, is a facility where stray dogs are taken to be held until they can be reclaimed by their owners or adopted by new families. The term “pound” in this context refers to a place where animals are held in confinement. The weight of a dog pound would depend on the number of dogs being held, as well as their individual weights.

So, in short, it is not possible to say which weighs more between an imperial pound and a dog pound as they are two completely different things with no basis for comparison.

You baldly assert these things. Do you have any argument to make your case, or do we just have to take your word for it, on the basis of how obvious that is to you? Better yet, there is a proof in the OP that entails that the impression you have that there must be some understanding is inaccurate (which is, after all, what this thread is supposed to be about). Can you point out where this proof goes wrong?

FWIW, I asked it:

ChatGPT responded,

I then asked ChatGPT your question, in the same chat, so that it was “primed” to think about wordplay.

It gave a similar answer to the one you provided:

I actually did get the incorrect answer about “dog pound” earlier. So it may have to do with whatever randomness is in the system, or perhaps it has “learned” but I don’t think there’s any real-time training, is there? I mean, within a user’s conversation, there is some persistence, but not across the whole.

While there is a philosophical argument (essentially the “Chinese Room” argument, where understanding is an emergent property, even if not explicitly programmed in), I don’t think that’s actually all that necessary here. All you need to do is look at the errors ChatGPT makes. If it does actually understand a concept, it’s not going to make mistake about that concept. Understanding something provides consistency.

ChatGPT repeatedly makes errors that people who actually understand the topics do not. It does this so frequently on so many topics that it suggests that understanding is not part of it. Looking at the similar bot from Bing that has access to the Internet, and it’s even more clear. The bot constantly seems convinced that people are attacking it when that hasn’t happened. It’s clearly just saying things others have said before.

Both ChatGPT and Bing “hallucinate” in ways that seem to show a lack of understanding of the topic.

Right. I don’t think that ChatGPT actually understands things. But I think by the bar that @Half_Man_Half_Wit sets, none of us can be proven to understand anything.

If you built a much larger Neural Network than ChatGPT, and then you trained it on data coming from video and audio sources, could it achieve understanding?

But I’d also suggest that the types of mistakes suggest that the method employed for ChatGPT is incapable of ever understanding. The more I see of it (or its cousin at Bing), the more it seems to me like the Blurry JPEG model of what ChatGPT does is correct.