I agree for the most part, as I hope my other responses have made clear. Which is why I phrased my comment as “to the extent that”, without trying to make a hard claim of what’s going on here. But word analogies in particular seem like a relatively easy problem compared to, say, summarizing several paragraphs of text (which is something it can also do). The degree of association it has to dig through is not as deep. I think the analogies are probably a prerequisite for everything else: they represent a particular way of abstracting meaning: if you have F(A) = B, then what is F(C)?
Well, OK. The contention in this thread was that ChatGPT’s behavior implies its ability to understand; if you don’t hold that view, then that’s fine.
That’s the other direction of implication, though: whether what understands must necessarily behave a certain way.
Again, it’s trivial to have the correct answer to any question be produced without an understanding of the question. Just have the appropriate string be the key in a database, linking to a value representing the correct answer. Since the act of string comparison involves exclusively criteria related to the string’s form, and none related to its meaning, whether it is understood simply has no bearing on whether the correct answer is produced.
That can be true in trivial cases, but this is where the sophistry comes in when applied to novel and complicated questions. Are you trying to claim that ChatGPT already had my carefully re-worded and somewhat complicated question already in its database as a string to match with, and an answer ready to go tagged by that string? That’s clearly an absurd assumption. The reason “understanding” applies here is that it was parsing a novel question that it couldn’t realistically have ever seen before, and then generating the correct response.
ETA: I had another look at the question, and now I’m just annoyed about how this is being trivialized. I was striving to get across the idea of two sets of words; I was conveying the concept that one set was tagged as “Group A” and the other as “Group B”. Understanding these sets and what was being asked of in regard to them was, I think, a more impressive feat than merely being able to look up the meanings of the words. But ChatGPT did all of that.
No. I was answering your question regarding how the production of a correct answer could possibly be consistent with having no understanding of what the question is, by pointing to an example where that’s the case. What this means is that the logical implication you’re trying to draw—‘giving the correct answer’ implies ‘understanding the question’—is false. Hence, giving a correct answer doesn’t tell us that the question was understood.
This doesn’t depend on a claim that ChatGPT uses such a system, because there may be other ways to produce an answer without understanding. Compare: you claim that, because Alice’s hair is wet, she must have been out in the rain. I say, no, it could be that Bob just threw a bucket of water at her. To which you reply, well, not only would Bob not do that, he doesn’t even own a bucket, and is in fact currently in Australia. Does any of this help to shore up your original claim? No, because what I’ve done with the example is to show that the implication does not hold in general; this does not mean that it can only be violated in the specific way my example stipulated. Alice could simply just have taken a shower.
My contention is that to whatever degree of certainty we can apply that to humans, we can equally apply it to ChatGPT.
Observation of behavior is the only method I have of testing human understanding. Even if an oracle told me that all humans were capable of understanding, I still would not know if a specific human understood a specific thing. For that, I can only ask questions and work out if they seem to have a model that matches mine. This can never be an absolutely certain process. In fact, I’d contend that any sufficiently complex model in another mind is virtually guaranteed to differ from mine in small details. So the small differences do not matter as far as I’m concerned; “close enough” is good enough. And the same is true for an LLM.
There are also cases where ChatGPT quite clearly doesn’t understand the question and is obviously parroting stuff it doesn’t understand:
What’s interesting about this is that if the questioner had used the standard question instead of specifying 2 pounds of bricks, the answer given would have looked like understanding as it told us that a pound is a unit of weight or mass etc etc… But if this answer doesn’t show understanding, which it obviously doesn’t, then what understanding would there be if it gave the “correct” answer to a question about 1 pound of both bricks and feathers?
NB there was a follow-up, in which ChatGPT further demonstrated it doesn’t understand the difference between 1 and 2.
Why, though? What makes that a reasonable assumption? In humans, we know that there is understanding, simply from self-examination: when I read a word I understand, there’s a qualitative difference to when I read a word I don’t understand. When I manipulate symbols according to their syntactic properties, there’s a qualitative difference to when I react to their semantic contents. I can expect the same to be true in relevantly similar entities, i.e. other humans. Indeed, it would be the failure of that assumption that would require an explanation.
But simply transferring this assumption to another kind of entity would just be sloppy thinking: I have no reason to suppose that whatever goes on in the human brain to understand goes on in ChatGPT, as well. I know the assumption I’m making is false, in general; so what would drive me to make it in a case where I have no reason to believe it to be true?
After all, this is exactly the sort of animist assumption that makes creationists disbelieve evolution: things that require purposeful actions for humans to construct surely must require purposeful action to arise in nature. But that’s just not the case: we can give explicit examples where apparently purposeful designs emerge from blind, random behavior. Thus, I’m inclined to be extremely skeptical of this sort of anthropomorphism.
(All that, of course, neglecting that we have good reasons to doubt that ChatGPT understands a damn thing on entirely independent grounds.)
Two amazing Monty Hall efforts.
(The “this” in the first tweet and the linked article refer to a human beating an AI at Go by playing a strategy the AI simply couldn’t recognise as a strategy).
With all respect, this argument seems to come from some far-away philosophical fantasy-land where the important thing is to show that a claim you made could be true in the right set of specific circumstances. No argument with that. It was certainly true in Eliza. But unless you can show how the same kind of trivial string-matching can result in ChatGPT successfully exhibiting the genuinely impressive performance just demonstrated, that argument is both irrelevant and misleading. Moreover, such an argument can be applied to any AI and indeed to any intelligent human at any time and in any circumstance – that there is no actual understanding and no actual intelligence there. Hence, sophistry: a subtle, tricky, superficially plausible, but ultimately fallacious argument.
ETA: Just to add, one of the difficulties of this question (the two word groupings) is conveying to the subject and having them understand the idea that there are two sets, labeling them as “Group A” and “Group B”, and then defining the matching task required between words in the two groups. To me, the fact that ChatGPT grasped the concept of the two sets and the required task was far more impressive than its ability to look up the definitions of the words. But it did all of it successfully.
Again, that’s beside the point: to show that behavioral criteria don’t imply understanding, it’s sufficient to show that there is at least one case in which they don’t. That doesn’t commit me to showing that the understanding was faked in exactly that particular way.
I don’t think I can explain it more clearly, not that explaining the same thing over and over again will get us anywhere, so I’ll just repeat this point from above:
There could be billions of ways to fake understanding to each way to really understand: we simply don’t know. Hence, the only way not to make unwarranted assumptions about the relationship between understanding and behaving-as-if is to say that the latter fails to reliably indicate the former.
I don’t think the situation is anywhere near as clear-cut as that. There are times when I think I understand something, and later find out that I clearly don’t. I have intermediate degrees of understanding. I have skills that I can perform that I have no conscious understanding of. I have skills–pretty much all language, for one–which I do have understanding of, but in practice only rarely employ (I don’t read by diagramming sentences, for instance). And of course things like my senses and memory are faulty.
All of which adds up to self-examination being faulty. Hell, even for stuff I know I understand deeply, I find it difficult to convince myself of that except by proving my capability (say, by writing a computer program), which is exactly the same experiment I’d use on another person to demonstrate their understanding of something.
My own view is that we should be careful with anthropomorphizing humans.
I have never in my life seen a human do exactly the same thing.
I mean, if you really want to argue that you’re not ultimately sure whether you actually understand anything, I’m not sure if I should offer much in the way of opposition 
… Except to note that this would only serve to make the move to assume understanding in others more dodgy, not less.
And how do you find that out? By understanding that your earlier interpretation was erroneous. If you can correct your understanding, that only serves to bolster the case that you are capable of understanding, not undermine it.
ChatGPT isn’t human.
More to the point, your sarcasm is literally true. You have never seen a human fail to understand that they know what’s on the other side of a door they can see through, and you never will.
You will never see a human who knows of the Monty Hall problem tell you you should swap a car that you have and want for a goat because doing so will improve your chances of getting a car.
You will never see a human tell you that 2 is the same as 1 because pounds are a measure of mass.
Whatever process led to ChatGPT telling us these things, “understanding” was not a part of it. Which does raise the question of whether “understanding” can be part of the process that leads it say things that appear correct, given that it’s the same process in both cases.
No, it absolutely does not. This is sophistry masquerading as logical rigour.
We do seem to be going around in circles, so let me conclude in the following way. The problem with your statement is that it appears to attack the idea that one particular behaviour is going to be reliably reflective of “true” understanding or “true” intelligence. In fact, judgments of understanding and intelligence – just as in Turing’s proposed test – are realistically based on an appropriately large range of observed behaviours. Eliza gained very little traction even in its day, not because everyone dismissed it based on “how it works”, but because its behavioural limitations were almost immediately apparent.
Conversely, ChatGPT is getting widespread attention, not because of any one remarkable answer it gave, but because of its consistent performance. I don’t think you would (or could) plausibly raise this same argument against an AI that could successfully respond to any imaginable intelligence test that you could throw at it, including tests that very few humans could pass. Nor could you raise that argument against another human (not because they think just like you do – how do you really know that?) but because they exhibit intelligent behaviour across such a broad spectrum.
So where do you draw the line? At what point of broad-spectrum demonstration of understanding and intelligent behaviour does one give up this reductionist silliness and accept that the qualities being demonstrated should be accepted as genuine? I don’t know where that point is, and ChatGPT is certainly not there yet, but it’s getting there and at some point that acceptance is going to have to happen, or we’re just going to be in a perfidious state of denial.
So you’ve never seen a human give a completely inappropriate response because they didn’t understand the question? Does that invalidate the whole premise of human understanding?
Well, at least I have the logical rigor, whereas all you have is ‘well it really looks like understanding, so it must be’.
These responses are not “inappropriate”. They are incoherent gibberish that betray not just a failure to understand a question, but a failure to understand some very basic concepts like the difference between 1 and 2 or what it means to actually have a thing. And no, I have never seen a human produce responses of that kind when they didn’t understand a question.
This essay by the guy I tweeted has loads of illustrations of similar problems - where a simple prompt causes ChatGPT to produce nonsense.
ChatGPT: Automatic expensive BS at scale | by Colin Fraser | Jan, 2023 | Medium
But it’s not just a list of amusing failures. He explains why these failures occur and why they are fundamental to what ChatGPT is:
It’s an error to describe unintended text in LLM output as “mistakes” in the first place. This in itself is a sneaky anthropomorphism, a way of implying that the model was trying to produce the right answer, but failed due to insufficient capabilities or expertise. But ChatGPT wasn’t trying to solve the Dumb Monty Hall Problem or the quadratic equation; it was trying to recursively predict the next word given the previous words and the joint word frequencies in the training data. I have no reason to claim that it failed to do this. The training data is littered with explanations for why you should switch in Monty Hall-style problems. For a model that produces output based entirely on the joint word frequencies in its training data, it would be miraculous if it didn’t produce the wrong answer to the Dumb Monty Hall Problem. It produced text consistent with almost all Monty Hall Problem-style prompts, which is exactly what it was programmed to do. No mistakes were made.
No, the problem is that you have a cherry-picked narrow definition of “behavioral criteria”. A productive argument requires at least agreement on the meanings of terms. Your argument is exactly like giving an example of an apparently intelligent or intriguing response that Eliza might have given, and then declaring that the Turing test is useless because “it’s sufficient to show that there is at least one case” in which the response the evaluator sees on a terminal doesn’t imply intelligence. Which sounds beautifully rigourous, but doesn’t actually address the real question, which is much more nuanced.
The real issue in both cases is that “behavioural criteria” must be defined as a sufficiently broad range of behaviours to allow for a reliable assessment, not just one which is shown to lead to the wrong conclusion. Essentially the more behaviours the evaluator can observe, the greater their confidence that the underlying qualities are being consistently represented, which is another way of saying “that the qualities are real”. Because if not, we would have to dismiss both human intelligence and superintelligent AI smarter than humans as all being “not real”.
No, they don’t betray anything except that LLMs, and AIs in general, do not function the same way as humans, and so when they make mistakes due to faulty understanding, the mistakes might seem really strange to us. Sometimes, strangeness and surprises emerge even when they’re successful. Expert players at the game of Go described some of AlphaGo’s winning strategies as “alien” – they work really well, but not in the way that a human would ever have thought of.
One of the weaknesses of ChatGPT at present is that it doesn’t assess the accuracy of its responses. A key feature of the Watson DeepQA is its ability to confidence-rate its responses, which was important in driving its strategy in the Jeopardy championship.
I have not picked any definition at all; I leave that completely open to those making the claim that a behavioral test is sufficient. Indeed, I even allow that this test, according to whichever criteria, is 100% effective, so that the case can be brought forward in its strongest form.
Whatever behavioral criteria you care to put forward, whatever test you may consider is the best in an utopian setting where you can do whatever you want with a candidate system: that’s exactly the test I have mind when talking about membership to the second set in the example. It’s completely up to you!
My claim then follows from noting that there’s at least one example of a system in S2 but not in S1, and that, given such a system exists, there could be arbitrarily many such examples, as far as we know, and hence, that any assumption that testing for membership in S2 yields reliable information regarding membership in S1 is unwarranted.