The problem here is with the dismissive implication of “all that these models do”. In fact the consequences of that paradigm in neural networks on very large scales are unpredictable and, as we’ve seen, often produce amazing results. Raphael Milliere, an AI researcher who specializes in studying generative models based on deep artificial neural networks, makes the point that characterizing the LLM paradigm as “only” next token prediction is misleading because it fails to acknowledge the fact that in certain instances very sophisticated capacities may be required to correctly achieve it. He calls this kind of dismissal a category error.
It’s not dismissive. It’s completely free of valuation. It’s just a factually accurate statement about how LLMs work: they complete a sequence by generating a new word (or token, more generally).
Except that’s kind of like saying, “all a writer does is write one word after another until a sentence is formed.” At one level, it’s true. But an accurate assessment of what’s going on would conclude that the actual putting of words on paper is a tiny part of the whole picture.
Csn you explain how next word prediction, all by itself with nothing else going on, can solve a mathematical proof that it has never seen solved before and requires calculus to solve it and the proof has no natural starting point? Can you explain how next word prediction can allow GPT-4 to understand why a picture of an RS-232 cable plugged into an iPhone might be funny?
I remember when the first image LLMs like DALL-E came out, and people said it was just a trick, and DALL-E might be able to build pretty pictures but had no idea what any of the pixels actually meant. They would describe the diffusion process, and say, ‘and that’s all it’s doing’. Now we know these models actually recognize not just objects in the pictures, but their properties and the relationships between them, in sophisticated ways.
ChatGPT has 96 layers of neural net. The ‘next word prediction’ part makes up 6 of those layers on output. It’s just a small part of what these things are doing.
They are. LLMs use the text prompts to mine their training set. The bigger the training set the more data to mine. The more text in describing the problem the more clues it has for mining. I could not find any of your high level math problems, but I’ll bet they are very textual. Also, from the link above, LLMs are not solving math problems they are just getting 70% of it right, whatever that means. The successful example I posted in the other thread required some coaching.
If GPT is doing math then it is getting the easy stuff right and that is not the case:
Prompt:
multiply (3+3+1+1) by (3+3+1+1)
GPT:
To multiply (3+3+1+1) by (3+3+1+1), we can use the distributive property of multiplication as follows:
(3+3+1+1) x (3+3+1+1)
= 3(3+3+1+1) + 3(3+3+1+1) + 1(3+3+1+1) + 1(3+3+1+1)
= 9 + 9 + 3 + 3 + 9 + 9 + 3 + 3 + 3 + 3 + 1 + 1 + 3 + 3 + 1 + 1
= 16 x 16
= 256
Therefore, the product of (3+3+1+1) and (3+3+1+1) is 256.
It is just following the one word it got in the prompt - multiply. If it had added terms, it had the right answer. But it doesn’t add. So, it got the wrong answer. No reason to give it partial credit, because if it had actually known what it was doing in the first half, it would have been right in the second half.
Explicitly and by construction, all you do is survive and pass your genes on to the next generation. That statement is just as true as your statement because you and a LLM were both created through an interative process to reach a goal (next word generation for LLM, reproducing for your genes).
If evolution can produce something like “understanding” as an emergent property on the path to successful reproduction, why can’t the algorithms powering a LLM not evolve something capable of understanding, if it turns out to be the best or even only way of performing next word completion at a high enough level?
It’s not. A writer goes back and edits things—crosses out, replaces words, rewrites entire sentences. Also, a writer looks ahead—while writing this, I have an overall sense of the argument I’m wanting to make, a rough structure, the next few sentences more or less planned out, and this one basically thought through to the end. I sometimes need to stop for a while, collecting my thoughts, reordering my arguments, and clarifying the points I’m trying to make. ChatGPT—and I hope we’re in agreement at least there—can’t do any of these things.
In exactly the same way that it solves the problem of completing a sentence that it has never seen before: it utilizes the patterns present in its training corpus. Let’s go back to addition. It’s an operation described by a simple formal set of manipulations on certain symbols. With increased training, it will be able to ‘fake’ following that rule on an increasingly large sample of cases, until at some point, it has discovered that rule—which is itself not materially different from a rule of grammar—and applies it correctly, the same way it applies the rule that noun and verb should agree in number.
That sort of teleology is entirely misplaced when applied to biological evolution. For one, not all humans actually do so—and yet, they still do things. If an LLM were not to do word prediction, it wouldn’t do anything. Evolution means that entities with a high likelihood of passing on their genes proliferate; but how they do so, and what they do to achieve this, and what else they do that doesn’t negatively impact on this, it doesn’t care. Thus, there’s a difference between you and an amoeba, even though as far as evolution is concerned, you may both be just gene-passing vehicles. But that is the minimum of what you both do; in ChatGPT, word prediction is the maximum.
That LLMs only do next word prediction has no bearing on whether they’re able to understand one way or the other, at least not that I could see. Sure, it would be an understanding of a very different kind than that present in humans, and I can’t imagine what it would be like, but so what, there’s lots of things I can’t imagine.
That LLMs can’t actually understand anything is a simple consequence of the fact that it’s provable (
) that the data they have access to don’t contain the necessary information to determine the referents of the symbols used in language. Something which only does next word prediction in some way might be capable of understanding—I don’t know—but LLMs aren’t, because of how they do next-word prediction: by means of data regarding relations between words alone.
I’m just going to point out a few more things in addition to my earlier point about the open-ended nature of the skills that can potentially be developed through semantic competence alone. First, the view that LLMs are inherently not capable of understanding because they lack real-world referential information is not without its dissenters, even if the latter point is taken as true. For example:
Large language models (LLMs) represent a major advance in artificial intelligence and, in particular, toward the goal of human-like artificial general intelligence. It is sometimes claimed, though, that machine learning is “just statistics,” hence that, in this grander ambition, progress in AI is illusory. Here I take the contrary view that LLMs have a great deal to teach us about the nature of language, understanding, intelligence, sociality, and personhood. Specifically: statistics do amount to understanding, in any falsifiable sense. Furthermore, much of what we consider intelligence is inherently dialogic, hence social; it requires a theory of mind. Complex sequence learning and social interaction may be a sufficient basis for general intelligence, including theory of mind and consciousness. Since the interior state of another being can only be understood through interaction, no objective answer is possible to the question of when an “it” becomes a “who,” but for many people, neural nets running on computers are likely to cross this threshold in the very near future.
Do Large Language Models Understand Us? | Daedalus | MIT Press
Second, it’s not clear that LLMs are completely devoid of referential semantic information. For example:
Landgrebe and Smith (2021)’s analysis of the Transformer architecture’s expressivity and generalization concerns inferential semantics. This part of their diagnosis is shown to rely on misunderstandings of technical properties of Transformers. Landgrebe and Smith (2021) also claim that referential semantics is unobtainable for Transformer models. In response, I present a non-technical discussion of techniques for grounding Transformer models, giving them referential semantics, even in the absence of supervision. I also present a simple thought experiment to highlight the mechanisms that would lead to referential semantics, and discuss in what sense models that are grounded in this way, can be said to understand language.
Understanding models understanding language | Synthese
Sure it is. There’s a proof, right there, in the OP. Which, you know, this thread is about.
According to the title, this thread is about “why ChatGPT doesn’t understand”. It’s not about why the relevance of your proof must not be challenged. (This is, after all, a forum called Great Debates.) The cited papers address precisely the issue under discussion (as do many others, and as do some of the comments upthread). It’s regrettable that you don’t even have the curiosity to look at them, but perhaps others may be interested.
We’ve been over this:
There is more to the topic of a thread than just its title. So one last time, I beg of you: contribute to the topic, or create your own. And while we’re on the subject of curiosity, it’s curious you challenge me on my alleged lack in not directly responding to every hastily googled up bit of info you regurgitate, whereas you don’t even bother engaging with the argument that’s the topic of the very thread you’re posting to, and that’s highly relevant to an issue you otherwise spent lots of time and energy discussing.
Given your reminder about how you qualified the purpose of the thread, fine, I will relent and withdraw from continuing this line of discussion.
To address your perplexity, and leave you with some parting thoughts, I’m not challenging your argument on the stated terms because the soundness of what it asserts seems both true and not especially relevant, as I said up in Post #86 and elsewhere, and the more salient question is whether this particular argument from metaphysics is necessarily fatal to showing that advanced transformer architectures like ChatGPT actually do exhibit practical understanding, as evidenced by many persuasive tests. As you know, there is a suite of tests for cognitive competence called BIG-Bench (BIG = Beyond the Imitation Game) in many of which LLMs are still falling well short of 100% success, but it’s expected that LLMs will pass them all in the relatively near future. It’s unclear from your argument just exactly what specific tasks LLMs will not be able to do, except that they’re doomed to be dismissed as uncomprehending zombies no matter what they do.
So again, if your request is to keep the discussion to the indicated narrow metaphysical scope, I will respect your prerogative to do so, though it’s not clear what it achieves since (a) you’re already convinced that the proof is unassailable and you’re probably right, and (b) meanwhile the world of AI is rapidly changing and advancing even as we speak, and is doing so despite similar proofs we’ve had in the past (from Dreyfus et al.) that it isn’t possible. To be fair, notwithstanding his humiliating defeat against the MacHack chess program, Dreyfus was partially right in critiquing the limitations of the GOFAI models of the 60s, but entirely wrong in completely failing to anticipate how such models would undergo revolutionary advancements over the ensuing decades. Today they’re on a roll that seems to be exponential.
So I’d still think you’d have some curiosity about some of these counterarguments – the second one cited, Understanding models understanding language, is a more formal treatment than the first, but there are many others, some of which hint at the emergence of at least a limited sort of referential competence.
I’ll now leave you to your preferred line of discussion. Thanks for your patience.
I just tried it with the old “if a brick and a half weighs a pound and a half and half a brick, how much do 10 bricks weigh” puzzle chestnut. It flunked completely.
I’m fairly convinced there is no “understanding” going on here. Just VERY sophisticated sentence completion.
Like the old trigram programs on super-steroids, but throwing vastly more memory and computational power at the problem.
Stephen Wolfram has an interesting article on this (google ‘steven wolfram chatgpt’).
I met someone who supposedly taught himself Russian, who knows how well, using nothing but a bilingual dictionary and books in the source language. And I can believe that a really good linguist can decipher an unknown language—but, of course, he or she already knows what a cat is (no hope for hapax legomena though)
I met someone who supposedly taught himself Russian, who knows how well <<
‘How well’ would presumably have to be evaluated by native speakers?
It’s odd that linguistic ability does not seem to be a one-dimentional thing. I am fluent in my native languge (English, of course), but I’m not good at learning other languages. I can stumble by in French, having had a few years in school. But I struggle to learn basic phrases and idioms in other languages, while some other people seem to be able to absorb them easily. It’s probably a genetic trait?
There are people supposedly gifted in this respect, able to pick up a language in a matter of weeks, but pretty much all humans have a genetic trait for language acquisition, otherwise how did they learn a language in the first place?
Ask them what? Just if they understand what a dog is? The jury is looking skeptical. The witness alleges they saw a dog attack a person, but the opposing lawyer is saying that the witness just calls any medium-sized animal a “dog”, and it could have been a mountain lion or something else. It’s really crucial to the questioning that when the witness says or hears “dog”, they’re referring to a real-world instance of a dog and not some other concept.
Well, I don’t know what the test is yet. I think you already know my answer: you ask questions about dogs that establish within a reasonable degree of confidence that we mean the same thing. 4 legs, long snout, wet nose, the usual. That the test is fallible is actually beside the point–I actually have no expectation that the mutual understanding would be exact at all. We might have different answers for whether a coyote is a dog, for example. But that’s ok, we all have an approximate view of the world and all we need is a reasonable degree of overlap.
I don’t remotely agree that ChatGPT doesn’t look ahead. It must, in some sense, since it gets things like “a” vs. “an” correct before the next word has been outputted. And yes, once “a” or “an” has been written, that constrains the next word, and so it can choose one that fits–but if that was only after the fact, we’d see it get the simplest things wrong.
Really, lookahead is just a special case of next-token prediction, where the previous N tokens are considered unknown. As a simple example, I gave this prompt to ChatGPT:
Please describe a fruit with exactly two adjectives before it. The fruit should be the one traditionally associated with Adam and Eve.
It came up with:
The forbidden, succulent apple.
Regenerated:
Forbidden, tempting apple.
Clearly, the word “apple” is totally constrained here as the third word (aside from “the”). But the two adjectives are not; they could have been anything (though it liked “forbidden”, which isn’t surprising).
So in effect, the net is predicting ahead. The weights that contributed to “apple” are not influenced by the adjectives that came before it. But the reverse is likely true: given that the third word is “apple”, it chose adjectives that were good fits.
This is speculative, to some degree. But I’d argue that whatever the most accurate interpretation is, it is a distinction without a difference–it behaves as though it can predict far in the future, because otherwise it would too easily find itself with boring garden-path sentences.
I’d be interested in testing this more experimentally. It should be possible to train an LLM to consider “unknown” tokens; i.e., ones that haven’t been set in stone yet. Then, compare predictions with gap tokens with ones where the intervening words have been generated. I’d posit that it does quite well with short gaps (with long gaps, neither human nor computer is likely to do well).
“Do you know what a dog is?” Sure, they might be lying, but there’s nothing I can do about that. The point is that in general, I’ll have sufficient grounds to form the hypothesis that the person indeed has some understanding of the symbol ‘dog’—that there is something it refers to, to them. That might be a different thing from what it refers to for me, but that doesn’t matter.
Compare ‘Do you know who Tenzin Gyatso is?’: if somebody were to answer ‘yes’, I’d expect that there is something that, to them, that particular string maps to. That might not be the same thing as in my case; the might know he’s from Tibet, but not that he’s the current Dalai Lama. But there’s something there that that symbol refers to.
On the other hand, if they answer ‘no’, I’d expect that that string is just a meaningless array of letters. However, once I tell them ‘he’s the current Dalai Lama’, then, provided they know who that is, I’d expect them to also gain an understanding of the string ‘Tenzin Gyatso’.
And they’re free to make that case, but the onus of doing so is on them.
According to what I think I understand of ChatGPT’s operation, it can’t possibly do that. I also don’t see why it would need to: given its training data, if it had perfectly learned the structures inherent in it, it will only follow a given string with a word such that it could be present in its training data in the same way—i.e. complete the sentence the way a completion could possibly occur in the data. Thus, it will never back itself into any corners, because such corners don’t exist in the training data.
I could swear I regularly see Chat GPT back over the last word or three and delete or correct what it had just written. Am I mistaken and seeing something else?
Do you consider this point addressed by the multi-modal models coming out now, such as PALM-E?
Really? I thought this aspect was utterly crucial. The whole point to the Newman objection is that if you can remap referents, then understanding is meaningless. What’s the point of communication at all if you can’t establish that two referents are at least approximately equal?
As a practical matter, I hope you aren’t a teacher of any sort, or at least that you don’t use this method in teaching. I have a hard time thinking of a worse method of establishing understanding than “just ask”. Establishing actual understanding is a “meeting of minds” of sorts, and can only be done iteratively and by a query/response approach.
You seem to be repeating the false “stochastic parrot” interpretation of LLMs, but never mind that; it’s actually not crucial to my point.
Regardless of how it works, there are some small number of sentences that could “reasonably” be used to complete a sequence, as compared to the extremely large number of raw token combinations. Of these, we can look at the probability of a token showing up at some point down the line across all possible sentences (possibly with some extra constraints about positioning). If that probability is close to 100% for some token down the line, then we can say that token depends weakly or not at all on preceding tokens and that it is essentially being predicted.
Another way of saying this is that if you simply shifted the input tokens to the left by N slots, and filled in the gaps on the right with “wildcards”, it should predict the same word as if it had been run normally, given the appropriate prompt (i.e., one that always puts the same word in that slot).
I tried getting ChatGPT to do this, but it didn’t quite understand my prompt. Bing chat has been apparently been crippled and wouldn’t accept my “game” proposal. I would like to try again with GPT-4 to see how well it can do.