Yeah, you are very clearly testing it and marking it on different things than I was when I first started playing with it.
It is not at all clear that ChatGPT doesn’t ‘understand’. In the GPT 3.5 thread I outlined why we can’t be sure.
Today I read about another piece of evidence. In humans, there are special ‘object’ neurons that seem to associate things together. For instance, a ‘Halle Berry’ neuron that fires when her name is mentoned, or a picture of her is shown, or even a picture of her dressed as Catwoman, so long as the subject knew that Cat Woman was Halle Berry. This is assumed to be part of a complex memory system where a single neuron can recognize a complex picture.
So, researchers went looking for ‘Halle Berry’ type ‘neurons’ in Multimodal language models like GPT-4, and found them.
For example, a ‘spider’ neuron that fires when the word spider appears, when a picture of a spider is shown, or the word ‘Spiderman’ or a picture of Spiderman is seen.
The wild thing here is that we didn’t design this. There is nothing about the architecture of LLMs that encoded this behaviour. The Neural Net in GPT-4 was completely tabula rasa when training started, filled with random data. And yet, through the learning process it seems to have evolved similar structures as the human brain, and structures that we assume are part of higher-order thinking like concept formation, abstraction, thinking in context, etc.
There is a lot more going on in these models than just spitting out next likely words. That describes the end process. It’s like saying people don’t really think when writing, because they are just writing down the next most likely letter in,a word sequence, then amother, until they are done. What we don’t understand about LLMs is how they construct those probabilities in the first place, since some of what they do can’t possibly be done by simply predicting the next word in a sequence.
For example, you can show GPT-4 an image of someone leaning, and ask it 'What is happening, and what will happen next?" It will accurately describe the leaning person, AND it will tell you that the person will likely fall over. There is no ‘next word prediction’ that can solve this, at least until somewhere in the LLM the computer recognizes a person, that the person is in an odd angle, and that people at such an angle tend to fall down. So it’s not just seeing a bunch of pixels, but it’s recognizing objects AND their relationship with the environment.
‘Next word prediction’ has become a handy way of explaining how these things output text, but it’s just a small part of what’s actually going on inside an LLM. And for thr vast majority of it, we have no idea what it’s doing.
If I had to guess, I’d say that LLMs are actually exposing something quite profound about intelligence and how it formed in animals and how neural nets evolve. The complexity in these models is undesigned and mind-blowing, yet shares startling similarities to human brains.
You’re engaged in a kind of anthropomorphic line of reasoning, where you assume that because it can’t do things that you consider simple (because most humans can do them), then nothing else it does is real. But there’s no reason to believe that the strengths and weaknesses of the system will be the same as humans, so applying your own sense of easy vs. hard isn’t likely to be predictive.
Incidentally, GPT-4 seems to be much better at letter-based puzzles than GPT-3.5, namely in Wordle:
GPT-3.5 basically didn’t get the game at all (aside from suggesting 5-letter words). GPT-4 understood the rules and actually played rather well.
Many of the apparent weakness in the current system will go away with better training and more weights.
Only saying that the admittedly short amount of time spent playing with it felt exactly like talking to the proverbial Chinese Room. Especially when it spouts arrant nonsense instead of saying “I don’t know” (or saying anything at all coherent—not expecting it to pretend to be a human). If you talk to it, you will quickly see what I mean, but this is not a formal conjecture of anything. Doubtless this effect would be somewhat mitigated with an increase of a couple of orders of magnitude in context and complexity, but reexamining the architecture (is it published for GPT-4?) seems potentially more productive.
Well, in this thread, I gave a proof that it doesn’t understand, so unless that’s shown to be wrong, we can be sure.
I don’t know why that’s supposed to be surprising. Perhaps that the encoding is stable across modalities? But then, how else would it connect words to images?
Anyway, that there are such neurons in an LLM seems a straightforward consequence of word-vector encoding, where words used in a similar context will cluster in adjacent regions of a high-dimensional vector space. If there is a single neuron that’s highly active in such a cluster, that just seems like it would mean that it’s close to one of the axes/proportional to a unit vector. Which doesn’t strike me as all that strange, since the vector space is very high-dimensional and mostly empty…
Suppose that your proof, as far as it goes, is unassailable. For purposes of this discussion, let’s grant that this is true. Where does that take us, specifically with regard to this conclusion:
… as long as ChatGPT only knows words and their relations, it will never be able to infer from there anything about things and their relations: it produces sentences without any connection to states of affairs—it understands nothing.
I want to explore what this says about what ChatGPT and its successors will or will not be able to do in the future.
What I think is perhaps wrong with your conclusion is an unnecessarily constrained view of the concept of “understanding”. We might gain better insight if we substituted, instead, the more pragmatic term “semantic competence”.
One could argue that ChatGPT possesses no “understanding” as you defined it, but nevertheless has a high level of semantic competence. This enables it, first of all, to discern nuanced context-dependent meanings from natural language, and then to act on those meanings by solving difficult problems, accurately translating language, and performing many other remarkable tasks – tasks that were once thought to be in the distant future of AI capabilities or indeed not possible at all.
There’s an interesting paper that touches on this that’s worth a look. I’ll summarize it but first a couple of definitions of terms:
Contextualism: the view that context-sensitivity generalizes …There is context-sensitivity whenever a distinction has to be drawn between an expression’s lexical meaning (invariant across occurrences) and its (contextually variable) semantic contribution.
Computational semantic minimalism (CSM): that the semantic content of a phrase P is the content that all implementations of P share.
The article argues that while CSM is a familiar characteristic of classical GOFAI (“Good Old Fashioned AI”) it no longer applies to the deep learning paradigm of deep neural nets. There, the author argues, a new characteristic emerges that he calls “radical contextualism” which has powerful implications, not just in supporting highly nuanced semantic competence, but fundamentally in its logical processes:
In the case of [artificial neural networks], complex contexts breed[ing] metalinguistic vagueness as parts of the machine’s workings are obscured from interpretation. In other words, metalinguistic vagueness is vagueness about what counts as literal and non-literal. ANNs have no need for fixed meanings from which others are modulated …
Thus, there is an indeterminacy about which connections or weights are generated from others. This process seems to be completely pragmatically determined by the machine, allowing for the possibility of decisions on significance being nonstandard (nonlinear) and even ad hoc. For example, in predicting who the next likely president of the United States could be (given a range of candidates in the test set), the machine might focus on characteristics we do not usually consider relevant or salient such as age, gender, race, birthplace, education and so on. It sees thousands of data connections we are unable to appreciate. So what is meaningful in the decision or output might have no obvious parallel in our limited reasoning, like having a particular ancestral background.23 One of the main differences between GOFAI and deep learning is that the latter can automatically design high-dimensional features to probe the data.
In short, we can grant a lack of “understanding” in the sense meant in the OP, and still show – as in this excellent overview paper [PDF] that I posted in the other thread – that semantic competence can take us a very long way toward genuine human-like behaviours and maybe even AGI – artificial general intelligence.
I mean, the concept used here is pretty much just that words mean things. That the string ‘dog’ refers to a dog. That symbols are, you know, symbolic. I don’t see how understanding could be any more common-sense or generic.
If you think that’s an interesting topic, then by all means, open up a thread dedicated to it, so others can better discover it and join in the debate.
I gave some thought as to whether I should post this argument here (as opposed to maybe in the other ChatGPT thread) as you seem to want to limit this discussion to its original philosophical context. But the reason I did so was that it bothered me a fair bit that your argument seemed to me – particularly in its scathing conclusion that “it produces sentences without any connection to states of affairs—it understands nothing” – to be making unsupportable implications about things that AI systems will never be able to do.
If you mean this to be a purely metaphysical discussion, carry on, and I won’t intrude, though I’m unclear about the purpose. But if it’s meant to be commentary on the future potential of AI then I think the counterargument is relevant.
To put all of this another way, until such time as we have an AGI that can perceive the world with its senses and interact with it through its manipulators, building progressive mental models of the world like a small child, semantic competence is the AI functional equivalent to human understanding. It’s exactly what we would have if we existed as a brain in a jar. It has limitations, to be sure, but its capabilities are turning out to be far greater than might be casually imagined. Thus the argument that ChatGPT has no “understanding” is both true in some appropriately defined philosophical sense and, at the same time, if it’s meant to imply intrinsic limitations on AI performance, is also a fundamental category error.
In fact, I would revise my concluding sentence to add the bolded part:
… semantic competence can take us a very long way toward genuine human-like behaviours and beyond, into superhuman intelligence and maybe even AGI
I don’t think we need to repeat our previous discussions, but I’m genuinely curious about something, because this is not remotely common-sense to me.
Suppose you were given the task of establishing if someone understood what a dog was. I’m not asking as a kind of trick question or abstract thought experiment–suppose you were a lawyer and you had to establish understanding for whatever reason, maybe because there was a witness to some event, and there was some question if they really knew what a dog was. What are the concrete steps you would undertake to do that? What do you think would be convincing–one way or another–to yourself and others?
I have just found a query that really seems to break it.
Q: "A popular mnemonic for pi is “how I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics”. Devise a similar mnemonic for the constant ‘e’. "
It came up with: " “Every apple tastes really great, especially freshly picked juicy ones!” or
“Now I know a lot of digits, eagerly calculated by my eager mind.”
Nothing it came up with was remotely correct. I tried additional hints, eventually arriving at:
Q: “A popular mnemonic sentence for the digits of pi is “how I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics”. Devise a similar mnemonic sentence for the digits of the constant ‘e’. This should use the same technique: the number of letters in each word should correspond to the digits in the decimal representation, including the first before the decimal point. No verbal representations of numbers such as ‘one’ or ‘three’ should be used.”
And it responded with:
"“Every day Greta’s housecat jumps quickly over sixty-five fences, vanishing without a trace.”
In this sentence, the number of letters in each word corresponds to the digits of ‘e’, starting with “e” and ending with “ce”. The sentence creates a vivid image of a housecat jumping over an impressive number of fences and disappearing without a trace, which may make the sentence more memorable."
The mnemonic is just plain wrong, and the ‘explanation’ is gobbledegook. Unless there’s some meta-level where the thing is smarter than I am?
I’ll ask them. Sure, that way, I can’t be exactly sure that my opposite isn’t an AI, and only ‘faking’ understanding, but I can’t be sure I’m not a brain in a vat, or any number of other skeptical hypotheses. The difference is in what’s reasonable to believe: it’s reasonable to believe fellow humans possess understanding in the same way I do, because it’s not reasonable to believe that I should be somehow special. It’s not reasonable to believe a system such as ChatGPT possesses understanding in the same way, because a proof to the opposite effect means it’s reasonable to believe it doesn’t.
It’s not an infallible test, but why should it be?
That is true. and one of it’s limitations is that it does not understand anything.
GPT:
“Therefore, (3+3+1+1) multiplied by (3+3+1+1) equals 32.”
If GPT had any understanding of context, it would correct that statement before printing it.
It’s just word order:
“What is () multiplied by ()” yields correct answer
“Multiply () by ()” yields incorrect answer
Sure you could be sure, not that it really gets us anywhere. If I was a defense lawyer and an AI was an expert witness on the subject of dogs that I wished to discredit, I would proceed just as I did with ChatGPT (edited for brevity):
What is a dog?
A dog is a domesticated mammal and a common household pet that belongs to the Canidae family. They are known for their loyalty, intelligence, and ability to form strong bonds with humans …
Have you ever played with a dog? Have you ever even seen one?
As an artificial intelligence language model, I don’t have the capability to see or experience the physical world like humans do. However, I have been trained on vast amounts of data, including images and descriptions of dogs, which allows me to understand what dogs are and provide information about them.
How can you be sure that your knowledge of dogs is accurate?
My knowledge of dogs is based on the vast amounts of data that I have been trained on, which includes information from a wide range of sources such as books, scientific research, and online content. However, my responses are generated using statistical patterns found in this data, and as a result, there is always the possibility of errors or inaccuracies.
It’s important to note that while I strive to provide accurate information, I am not infallible and may sometimes provide incorrect or incomplete information …
At which point, having discredited the expert, I would rest my case. 
But in reality, the difference in dog knowledge between someone who lives and works with dogs and a putative expert whose knowledge of them is entirely theoretical is not necessarily very great. In AI theory this would be described as referential versus inferential knowledge, but there would be a great deal of commonality between the two. In general the dog owner’s knowledge would tend to be narrower in scope but deeper, and the expert’s knowledge would probably be broader in scope and shallower in places. But both would be able to provide a useful account of dogs.
For certain interpretations of “understand” and “anything”. It’s currently notoriously bad at many math problems. Do you think this condition is intrinsic and permanent, or do you think it will get much better? Note that it was shown in the other thread that with nothing more than additional training, generic calculation capabilities have spontaneously emerged in such models.
And for something that “doesn’t understand anything” it’s doing remarkably well at high-level tests that might soon qualify it for advanced degrees …
GPT-4 passes basically every exam. And doesn’t just pass…
The Bar Exam: 90%
LSAT: 88%
GRE Quantitative: 80%, Verbal: 99%
Every AP, the SAT…
https://twitter.com/emollick/status/1635700173946105856
It’s not bad at math. It doesn’t do math. It’s like they keep telling us. It’s just word order:
“What is (3+1+1) multiplied by (3+1+1)” yields correct answer
“Multiply (3+1+1) by (3+1+1)” yields incorrect answer
A test question contains all of the information GPT needs to mine it’s training set for the answer. The wordier the question the better. Recall of text is what it is weighted to do.
But, recall is machine intelligence. What’s wrong with that? Why does it require understanding?
If it didn’t do math, its answers would almost always be wrong. But in fact, at a certain scale arithmetic abilities magically emerge and become progressively more accurate. The article below describes the emergence of three capabilities at appropriate scale: arithmetic, the ability to pass college-level tests on a broad suite of subjects involving world knowledge and problem solving ability, and the ability to understand the meanings of words from context. Some of these capabilities were observed in some LLMs at lower performance levels at smaller scales, but others were not, and emerged suddenly at certain threshold scales.
I would add that we’ve found ‘neurons’ in the models of these things that associate objects. Not words, not letters, but objects. For example, in the new multi-modal GPT4, there’s a neuron that will light up if you say “Spider”, or show a picture of a spider, or even a picture of Spiderman. Over time as it read things and saw images, it created these associations to link things at a symbolic level, not just as tokens. The human brain has the same kinds of features.
These neurons are evidence of ‘understanding’. When you say “Spider” to GPT-4, it can actually picture what a spider looks like, and might even conjure up Spiderman.
More evidence of understanding in these models comes from the testing of Palm-E, an ‘embedded’ large language model that has been allowed to run a robot with vision. Palm-E can solve visual problems, recognize visual jokes, move objects around in the real world intelligently based on prompts, etc. There’s no ‘next word prediction’ at all in the output, because it’s outputting actions, not words. It CAN output words, but it’s not necessary to its ‘understanding’ of tasks and the space it’s in.
Even more evidence of some kind of ‘understanding’ comes from the fact that ‘mental maps’ of geography have been found in GPT-4. Describe a room, and GPT-4 will have a map of the room in its model. It needs this to be able to finish certain sentences like, “I turned left at the table, then turned right and walked five feet, and bumped into…”
Also, you can describe a scene to ChatGPT, and it will build it in 3D in Unity. That implies it understands shapes, relationships between them, geometry, etc.
And we KNOW it’s not just doing ‘next word prediction’ for addition, as we’ve found code inside the models that actually solve math in a general way. That too is emergent and unexpected. The addition code appeared in ChatGPT after 10^23 FLOPS of computing. Now GPT-4 can do advanced calculus, and you’re not doing that with next word prediction. It even solved a mathematical proof from the 2022 International Math Olympiad that had no obvious starting point and required calculus to solve. It was not in its training data.
It’s really somewhat telling how you take me to task for supposedly overstating my conclusions, which I both qualify and actually produce arguments for, while you just blithely claim highly nonobvious things like this as if they were self-evident truths.
I said it was ‘evidence for’. I didn’t say it was a slam dunk. No one really knows what’s going on in these things for sure, but we have evidence that there’s more going on than just ‘next word prediction’.
I don’t even know what ‘understanding’ would look like in a large language model. Maybe nothing like humans. But when we see the model forming associations between objects, drawing mental maps of places, being able to tell from an image that something is about to fall over or that a DB-15 plugged into an iPhone is a joke and why, does that not constitute evidence for some form of understanding beyond just relationships between words? How would you explain it?
Yes, exactly. Why would that be evidence for understanding? A bell tied to a string at 1.5m height rings whenever something taller than that goes past. Is that evidence of understanding the concept ‘taller than 1.5m’?
I’m not sure I get what you’re trying to say here. ‘Next word prediction’ is explicitly and by construction all that these models do. That’s just how they work.