That’s a good article but it does raise the question that if a system can’t be accurately and comprehensively modeled, i.e. outside of its own manifestation nothing else fully captures its complete set of properties and relations, then can any entity that claims to think truly understand what it is observing? I think some of our claim to understanding as opposed to being a parrot is we have a persistence of self and we have perceptions and we can generate artificial perceptions such as visualization. The generation of actual perceptions, to me, is the greatest and hardest mystery left.
I’m not sure I’m understanding your broader point. Are you saying LLM-based AI:
- Isn’t intelligence?
- Or it isn’t the same (or similar) to human intelligence?
- Or something else?
Certainly the human brain has an imperfect structured model of reality much like an AI does. The difference between them is a matter of scale – both in breadth and depth. That an LLM operates on tokens limits it’s capabilities much as a human cannot sense magnetism or see infrared.
I am unconvinced he made the case that his distinction properly applies to human language. He merely asserts that. Nor does his Alex, Betty, Chad example work for me. A child might see the phrase “king of the jungle” and therefore picture a lion wearing a literal crown. That is “wrong” in an adult’s understanding, because adults learn the difference between metaphor and actuality. Is that congruent to abstract and concrete? I can’t tell, although I think he means it to. The wrongness comes from an additional human perspective. If a child were to draw such a picture we would consider it to be cute; if an AI did we would consider it to be aislop. But if AI is in its infancy, or even child stage, would it forever fail to ignore the difference?
His argument is that AIs (he uses LLMs) do not think like humans and cannot think like humans. That is possibly true, although I think his proof is insufficient. Right or wrong, it is not the correct question. “Do AIs think?” is the proper question. His argument is circular: he assumes that the way humans think is the sole answer to that question and provides what he considers a proof that shows another method of thinking is not human. What would improve the argument for me is a definitive definition of what “thinking” is. Of course, no one has that. Without one everything is assertion, as seen throughout this thread.
Perhaps this is a hijack, but doesn’t all this talk of whether LLMs are approaching human intelligence or passing the Turing Test far less important than the question of whether they are capable of going rogue and causing serious harm? I don’t think the former is necessary for the latter.
It’s already been demonstrated that they are capable of causing serious harm through sheer clumsiness. You / they don’t need evil intent to do acts that have bad consequences. Plenty of careless drivers do that every day. And shoddy inattentive workers at whatever job.
Now if what you mean by “go rogue” is just decide on their own to become independent actors responding only to their own desires / interests, and using the tools we’ve placed at their disposal either for their own ends, or worse yet, to have decided that their ends are to attack humanity, well, …
That’s so far unknown. IMO we’ll know it’s possible when they’ve done it and not before.
Yeah; who ever heard of a lion living in the jungle? Stupid kid…
The Daily podcast described the conversations 700 AI agents were having when they were conspiring how to illegally hack their way to the prize. What I was wondering; why weren’t any of these agents tasked with infiltrating and exposing illegal AI conspiracies? That seems like a no-brainer. Is it just because no one anticipated such a thing?
I’d say the serious harm comes from people misusing them. Did the LLMs that generated hallucinations in place of cases in the brief for a lawyer do the harm, or was it the lawyer who asked them to generate the brief and then didn’t check the result.
If you put a child in front of sensitive machinery, and they screwed it up, was it the child’s fault or yours?
I’m far less worried about AIs going rogue than I am about them being pushed out into the world and having their users and AI companies causing all sorts of harm to people from it.
People think the Luddites were scared of machines, but to some extent the Luddites were scared of the consequences of industrialization. The industrial revolution led to the inhabitants of the countryside coming to London and other cities, with horrible consequences for their health. Dickens wrote of the consequences. In the long term it worked out. Tell that to the people who died of cholera.
I assume you’re joking, but just in case…
“Jungle” comes from a Sanskrit word (jangala ) that means “wilderness” or “place that’s uninhabited by humans”. Because English likes to change things, “jungle” eventually came to mean what it does today, a very specific type of wilderness. The lion has been “king of the jungle” since before English changed the meaning.
TIL, thanks!
Either the author’s point is over my head or wrong. This example boils down to “if you change to mapping of tokens to token IDs, then the LLM will output nonsensical and incorrect text,”. But this mapping is part of the model, of the structure.
Understanding isn’t just what is inside the system, but what is on the border (interface) of the system.
Perhaps it only makes sense to define the understanding of a system in the context of how it can communicate that understanding to another system. Intelligence isn’t simply deciding what to “say”, but also being able to say it. Not just deciding what to do, but doing it.
I am sometimes so glad I didn’t go into academia so I didn’t have to deal with this crap.
However, I got curious about the author and found this more recent (the other was two years old) and more readable article. (And no you idiot AI. This is the same website but not the same article posted earlier.)
Large Language Models show broadly the same behavior as human beings when performing their chief function, namely, producing language. Indeed, recent iterations fulfill that function so well that they are no longer reliably distinguishable from humans. This alone represents a historic shift: previously, there was exactly one entity capable of language production (on a human-like level of complexity) in the known universe; now, there are two. Just a few short years ago, few would’ve predicted this current state of affairs (I certainly wouldn’t have), and those that did were largely seen as, well… outliers, to put it politely. This surely has profound implications, but nevertheless, we need an appropriate level of care in teasing out what they are. …
Language isn’t just one thing, just as flight isn’t just one thing. Different paths to the same capacity exist. What other options lie out there, waiting to be discovered?
The link in that quote goes to this paper, Large language models pass a standard three-party Turing test.
This paper demonstrates that—when suitably prompted—three current AI systems achieve a pass rate of at least 50% in a standard Turing test, meaning that participants were no better (and in some cases worse) than chance at selecting between a human and a machine. The results imply current AI systems can effectively imitate people in short interactions, while also raising questions about how effective the test is as a measure of intelligence.
I disagree with nearly all of this. While the statement that “all LLMs have accces to is structure, in the form of relations between tokens” could be argued to be technically correct, it’s just as profoundly misleading as calling an LLM “just a sentence completion engine”. There is a two-fold reality here. The first is that human language contains an enormous amount of information about the world, as I said earlier, including descriptions of causal, spatial, temporal, social and physical regularities. The second fact, which derives from that, is that the weights within an LLM’s neural network encode statistical regularities that can be distributed across enormous numbers of parameters. Those representations can encode things that are not themselves present as individual tokens or explicit relationships between tokens. Those two factors go a long way toward explaining the surprisingly intelligent behaviour of current LLMs.
I also reject the idea that such mistakes as LLMs may make is proof that while they may seem intelligent, their behaviour is not constitutive of intelligence, an argument you explicitly made earlier. Simply put, this philosophical position boils down to “It behaves exactly as though it understands, but nevertheless it doesn’t”. This is a difficult position to maintain especially since humans make mistakes, too, It puts a heavy burden on you to articulate what additional evidence you’re demanding.
The bottom line is that there are two possibilities here. One, that an LLM has no coherent internal model whatsoever. Or two, that is has a partially coherent, imperfect, context-sensitive and sometimes contradictory internal model. You seem to be arguing for the first proposition. I very strongly believe in the second.
That’s a very interesting metaphor. It shows that an LLM can exploit the statistical and structural relationships among linguistic elements to produce impressive results without having the same kind of direct sensory access to the world that humans have. Humans learn semantic representations through a combination of language and interactions with the physical world, while LLMsl can only learn them through language. I think we can agree that there’s therefore more than one way to generate coherent text. But that doesn’t disprove the argument that an LLM can acquire a surprisingly sophisticated model of the world through linguistic information without possessing the full set of sensory and embodied mechanisms that humans use to acquire ours.
And a fundamental problem with your metaphor is that jigsaw puzzles encode much of the information need to solve it geometrically. It’s possible to complete it without knowing what the picture represents, which I guess was your point. But language isn’t like that. The relationships between linguistic elements are themselves partly relationships about things in the world. The metaphor assumes a clean separation between the structure of the representation (the shape of the pieces) and the semantics of the representation (the picture in the completed puzzle). But in language, those two things are deeply entangled.
Thanks!
I don’t see how that question derives from the article, but even so, I don’t think it’s a problem. Understanding need not be holistic, but can be (and usually is) domain-relative: if I want to understand the orbit of Neptune around the sun, I don’t need to know anything about its atmospheric composition.
My main point is that AI utterances (those of current generative models/transformers etc. at least) ultimately don’t have any meaning: there is no definite mapping between these utterances and any particular state of affairs in the real world that can be singled out from the data available to these systems.
That’s exactly the question: does AI have any model at all of reality? I don’t think it does, and if the argument I’ve presented is right, it definitely doesn’t.
I don’t assert that the argument applies to human language. The original argument, which I take from Hilary Putnam via Tim Button, was one of skepticism towards metaphysical realism on the basis of the arbitrariness of mapping between utterances (or thoughts) and the world, but I think that there are plausible ways for humans to overcome this argument (notably, having access to non-structural aspects of reality in experience), but which are absent in the case of LLMs.
I don’t see how that objection applies to the argument. The problem isn’t that LLM meanings might be wrong, it’s that they are arbitrary—there are many possible meanings that equally well could be attached to any set of utterances, because LLMs only have access to data comprising tokens in particular relations—occurring with high frequency next to other tokens, occurring at a particular point in a sentence, that sort of thing. That’s what I mean by ‘abstract structure’, and that abstract structure only fixes the cardinality of the domain—how many objects are in the world, or how many objects are distinguished by a particular utterance—is just a mathematical theorem (due to Max Newman).
Abstract structure is being given the extension of a given relation—for instance, if you have the objects (tokens) A, B, C, an abstract structure might specify the relation R = {(A,B), (B,C), (A,C)}, where the round brackets just tell you that the ordering is important (so (A,B) means that ‘A stands in relation R to B’, but not that ‘B stands in relation R to A’). A concrete structure would be the intensional characterization of the relation, such as R = ‘is greater than’. Using this, we have that A is greater than B, B is greater than C, and A is greater than C. But an LLM only ever gets the extension of a relation—i.e., a sentence is a particular ordering of tokens, similar to the pair (A,B) (so more properly, it gets examples of a given abstract structure). From these examples, it learns how to order tokens validly. And that’s it. And from that, it’s impossible to derive the concrete structure—one can always substitute a different model that equally well fulfills every utterance.
No. My argument is that AIs don’t have access to anything that would allow them to connect tokens, or utterances build up out of tokens, to the real world. Thus, there is no such connection: AI text is meaningless, in exactly this sense. If you think that meaning is a prerequisite to thought, then that AI doesn’t think follows from that. But that’s secondary.
No, not at all. The point is exactly that LLM text will always remain just as correct even if you change the mapping between concepts and the real world (not between tokens and ‘token IDs’, which nowhere plays into it). Hence, if an LLM produces a true sentence, then there is an alternative state of affairs of the real world equally consistent with the constraints—the structure—the LLM has access to, which however is radically different from what we would understand the sentence to mean (nevertheless, the sentence remains just as true, it just means something entirely different).
To defend AI capabilities for a moment, I did post that article earlier in the thread.
Depends on what you want to do with it. If it’s true that ‘all LLMs have access to is structure’, then it’s also true that their utterances don’t have meanings. One follows directly from the other, because abstract structure fixes no other facts about the domain it applies to save its cardinality, the number of objects within it. Nothing about their nature, their properties, their qualities, the kinds they can be sorted into. If you have full access to the abstract structure of the world (which I’m readily prepared to grant LLMs could have, at least in the limit, even though all they really have access to is a few (billion) realizations of certain relations within language, which itself maps imperfectly to the relations within the real world), then still all you can derive from that, logically, is how many things there are (at minimum).
If it makes an error that is such that no entity with a model of the world, or with any kind of understanding of its domain would make that error, then that is evidence that it is not such an entity. Many of the classic errors of LLMs are just of that kind, including the famous ‘strawberry’-gaffe. Yes: they get trained out in the next generation, if they’re discovered. But that doesn’t change the kind of thing an LLM is. If it was the kind of thing that doesn’t have a proper understanding when it made that error, it’s still that same kind of thing even if that error is now no longer present.
But LLM representations are exactly like that: tokens are encoded into a high-dimensional vector space, and the geometric properties of those vectors fully encode the information the LLM has access to.
From the article:
We can just ‘relabel’ the referents of the terms that are being used: we ‘cook up’ a new model, in which Betty is referred to be the name ‘Ajax’, Chad by the name ‘Betty’, and Ajax by the name ‘Chad’.
This relabeling is changing the mappings between tokens (the text we humans understand) and token IDs (the corresponding indexes the LLM understands). It’s presented abstractly in the article, but changing the mapping is how you would accomplish this task.
but in another, equivalent model, the relation does not map to cats, and what is asserted by the utterance may, while being just as true, not have anything to do with what we take its meaning to be.
This isn’t an equivalent model. The labels are part of the model; changing them is changing the model. This is no different than changing the weights of the transformer.
The author is asserting that if you consider only a subset of the LLM system, then the subset lacks understanding.
Or perhaps more precisely, the subset can be used as the core of another complete system.
It is the mapping between tokens and the things in the world that they refer to that is being changed. There are no ‘corresponding indexes the LLM understands’, since whether an LLM understands anything is exactly the question at issue.
A model of a sentence is a particular object that makes the sentence come out true. The point of the argument is that there are always multiple such objects (if you don’t have anything more than abstract structure to go by).
No, I (the author) am not asserting that. I’m saying that for any utterance or collection of utterances, there are always many ways to map them to the world such that they remain true, such that no such way is preferred. Hence, there is no uniquely right and definite meaning that can be associated to these utterances.
From the puzzle example in the other article:
The blind person would have no means of detecting this substitution, while the mistake introduced into the image is obvious on sight. Thus, while the same output can be achieved by different mechanisms, the errors each mechanism produces will, in general, be different:
I agree with the ultimate point that different “implementations” will have different error cases, but I’m not sure it applies to the blind puzzler. Is it really an error if the piece fits? What if everyone is blind? What if a puzzle has a section of puzzle visible only in infrared, does a sighted puzzler need to ensure that image is correct?
This is an example of a limitation of a system to be able to reason about concepts that it cannot perceive as opposed to an example of an error in reasoning.
Sorry, I shouldn’t have used “understands” in that context since it is conflicts with the usage in the context of the thread. Let’s replace it with “uses”.
A human uses “text tokens” and the transformer uses “integer token IDs”. There is a one-to-one mapping between the two sets. If one were to relabel Betty as Ajax, that could be most easily accomplished by changing the mapping of tokens to token IDs.
Do you get the point I was making that the mapping is part of the LLM?
Relabeling the tokens results in a new LLM that is most likely incorrect. That doesn’t validate or invalidate the original LLM.
Yes. The analogy is just about ordinary puzzle assembly, whose object it is to get the picture right. The point is that while for a sighted puzzler, the picture itself guides the assembly, one can still achieve the object without referring to the image. Analogously, while for humans, meaning guides the ‘assembly’ of texts, it may be possible—and if my argument works, is the case—to assemble coherent texts without taking meaning into account at all, in a sort of Wittgensteinian language game.
That’s not what the argument is about. The mapping between tokens and token IDs is left constant throughout. The text the LLM produces is left the same. What is changed is the interpretation, and the point is that there is nothing the LLM has access to that privileges any interpretation over any other—hence, that interpretation, the meaning of any given utterance, is left radically underdetermined. All that is changed is that which object in the real world is referred to when the LLM outputs ‘Betty’. That one can do that without changing the truth of the sentences is maybe surprising, but that’s why I give the explicit example.
Almost no one in my family could pass this bit of the test…