Over in Cafe Society, the thread on OpenAI’s Large Language Model (LLM)-based chatbot ChatGPT has developed into a lengthy discussion on whether it (and any similar LLM) can be said to understand language, in a way similar to a human being. That thread was mainly concerned with the arguments in favor of that conclusion, and whether they are sufficient. Here, I want to make the negative case: outline an argument according to which LLMs are constitutionally unable to acquire any kind of understanding of language, and rather are what has been called ‘stochastic parrots’ that are basically just a souped-up version of your phone’s predictive text feature, producing a continuation of an input string by means of statistical patterns learned from large volumes of text.
The argument essentially establishes the following: the relationships of terms (names etc.) that can be abstracted from language (a kind of structure) don’t suffice to figure out how those terms are supposed to map to objects in the real world. With this structure being all that ChatGPT has at its disposal, it thus can’t figure out what anything means.
Note that this is not an argument against AI understanding, as such. There could, in principle, be ways of creating AI whose operation is sensitive to the semantic value of the tokens it manipulates. It’s just that LLMs aren’t that way.
So, the important notion in the argument is that of structure. A structure W is an ordered tuple <D, R>, consisting of a domain D and a relation R. The domain is just a set of distinguishable, but otherwise unspecified objects. For concreteness, let’s say D = {‘Alice’, ‘Bob’, ‘Charlie’}, where the single-quotes are there to alert the reader that these are intended to be names, not the objects themselves.
A relation is a set of ordered tuples, for example R = {<‘Alice’, ‘Bob’>, <‘Alice’, ‘Charlie’>, <‘Bob’, ‘Charlie’>}. Such a tuple can standardly be represented in the form of a set distinguishing the first element of the pair, i.e. <‘Alice’, ‘Bob’> = {{‘Alice’}, {‘Alice’, ‘Bob’}}. With this, we can now prove the following:
Newman’s Objection. Knowledge of a structure W = <D, R> is equivalent to knowledge of the cardinality |D|, i.e. the number of elements within the domain (and nothing more).
This is named after the mathematician Max Newman, who put it in this form:
Any collection of things can be organised so as to have the structure W [where W is an arbitrary structure], provided there are the right number of them. Hence the doctrine that only structure is known involves the doctrine that nothing can be known that is not logically deducible from the mere fact of existence, except (‘theoretically’) the number of constituting objects.
Newman raised his objection in 1928 against Bertrand Russell’s proposal that all scientific investigation tells us about the world is its structure (in the indicated sense). A thorough discussion can be found in this paper.
The argument goes as follows. Suppose we know that there is the structure W. Then, in particular, we know that there is the domain D. Likewise, then, every subset of D must exist. Then, the powerset, P(D), the set of all subsets of D, also exists. But then, every subset of P(D) also exists. But then, every relation R’ over D also exists, as all that a relation over D is, is a set of subsets of P(D). So then with W, every W’ = <D, R’> exists just as well. But that already follows from the fact that D exists. So, telling me that not only D, but W exists, tells me nothing new: that’s just equivalent to saying that D exists. W comes right along, and so does every W’.
But D is completely specified by merely the number of objects within it: after all, their sole property is being distinct from one another. So, all that I need to know to specify D, hence, all that knowing that there is the structure W can tell me, is that there are |D| objects.
As an objection against the idea that mere structure can tell you anything substantial about the world, this is decisive. Moreover, while remaining within the structural, there is nothing that can be done to patch things up. For suppose you were to claim that W is, in some way, ‘special’, that it is what ‘really exists’, or whatever. One possibility of doing so is to refer to the extension of R, i.e. explicating by virtue of what R holds of objects. For instance, suppose it’s the relation ‘taller’: then R holds because Alice is taller than Bob and Charlie, and Bob is taller than Charlie. But then, you’ve also told me something about the domain that doesn’t come down to structure, namely, that the objects have a certain size, and what relative sizes there are: you’ve left the realm of pure structure.
Then, suppose that you’re trying to single out R as special by purely structural means. This means you’re telling me that there is some structure W’’ by virtue of which R is singled out. But then, we’re back at the beginning: this is ‘just more theory’, and hence, vulnerable to the same objection.
Now, as far as metaphysics goes, this isn’t much of a threat for something close to the Russellian picture. We can, for instance, hold that we have non-structural knowledge of at least our own, mental properties, and ground relations there. But for ChatGPT, that’s going to be a bigger problem.
Suppose that, from its training data, ChatGPT has learned of the three terms in D, and that they stand in a relation C = {<Alice>, <Bob>}. Now we need to find an interpretation—a model—of this, such that this correctly mirrors the relevant part of the world. One such model is the following: ‘Alice’ refers to Alice, ‘Bob’ refers to Bob, and ‘Charlie’ refers to Charlie. Alice is a cat, Bob is a cat, and Charlie is not a cat. (Names without the single quotes refer to objects ‘out there’, in the world.) Thus, C is the predicate ‘…is a cat’. Can ChatGPT correctly identify this as the ‘proper’ model of the structure it has learned?
The answer, thanks to the Newman objection, is no (in this form, the argument is due to Hilary Putnam and known simply as ‘the model-theoretic argument’). We can appeal here to a construction that, in model theory, is known as ‘Push-Through’: given a structure over some domain D, a domain E such that |D| = |E|, we can imbue E with exactly the same structure. In particular, we can just permute the elements of D, using the permutation h(D) such that h(Alice) = Bob, h(Bob) = Charlie, and h(Charlie) = Alice. (Note that we’re doing this on the things in the real world, i.e. where the ‘model’ of the sentences ChatGPT produces lives.)
With that, the relation C becomes h(C) = {<Bob>, <Charlie>}. But this model makes all the same sentences true as before. The sentence ‘Bob is a cat’, in D, refers to Bob, who is in the extension of the relation C, and is a cat. The same sentence, in h(D), now refers to Charlie, which is in the extension of the relation h(C) (making the sentence true), but is not a cat. Hence, C picks out cats, but h(C) doesn’t; but there is no fact of the matter regarding which is the ‘right’ relation. Both allow absolutely truthful sentences.
To put this in somewhat different terms, there are two levels, the level of language (or ‘theory’), and the level of objects out there in the world (or ‘model’). ChatGPT only ‘knows about’ the language level, where there is the domain, given by the terms (‘names’) of the language, and their relations—the structure it has learned throughout its training (we can here assume that it learns the structure perfectly, even though that’s clearly not true of the real-world ChatGPT). Out there, in the world, there are objects (three in the example, Alice, Bob and Charlie without single quotes—the things, not their names), which will instantiate a number of relations, as discussed above (23 - 1 = 7 non-empty one-place relations, 23*3 - 1 = 511 two-place relations, and so on). The relation C is one out of the 7 one-place relations. These are:
R1 = {<Alice>}
R2 = {<Bob>}
R3 = {<Charlie>}
R4 = {<Alice>, <Bob>}
R5 = {<Alice>, <Charlie>}
R6 = {<Bob>, <Charlie>}
R4 = {<Alice>, <Bob>, <Charlie>}
R4 is the relation ‘…is a cat’, in the real world: it names things that, in fact, are cats. But there is no way to single it out among the other two options. All ChatGPT knows is that the tokens ‘Alice’ and ‘Bob’ stand in relation C. It does not know that ‘Alice’ is supposed to refer to Alice, and ‘Bob’ is supposed to refer to Bob, and thus, does not know that C is supposed to single out cats. It doesn’t—it can’t—know what it means to be a cat, even if it can utter correctly that ‘Bob is a cat’, because that is the sort of thing it does with elements of the relation C. But whether that is supposed to mean that Bob is a cat, or that h(Bob) = Charlie is whatever’s picked out by h(C), simply isn’t available to it.
This is depicted in the following picture:
There, the ‘Language’ level is what ChatGPT has learned from its training data, which allows it to frame sentences like ‘Bob is a cat’—it knows that the token ‘Bob’ is the sort of thing that ‘…is a cat’ should be said of. The arrows then depict two different ways for the terms to refer, the pink, unbroken ones corresponding to the one in which ‘…is a cat’ ends up picking out cats, and the orange, broken ones one in which ‘…is a cat’ picks out whatever h(C) is supposed to be. (Note, again, that whatever h(C) might be, the Newman argument ensures us that it exists, purely as a set of subsets of the powerset of {Alice, Bob, Charlie}.)
ChatGPT can never pick out which is supposed to be the ‘correct’ interpretation, and thus, can never know that Bob is, in fact, a cat. There’s just not enough information there, in the structure, which after all settles only questions of quantity. There is, simply, no fact of the matter whether, when ChatGPT produces the sentence ‘Bob is a cat’ it means, in fact, that Bob is a cat, or that Charlie is a h(C)-sort of thing—in both cases, it is saying something equally true.
The argument here was presented for a simple domain and hence, structure. But it can be equally well carried out for an arbitrary one: as long as ChatGPT only knows words and their relations, it will never be able to infer from there anything about things and their relations: it produces sentences without any connection to states of affairs—it understands nothing.