I don’t see that Othello case as in conflicy with the assertion that LLMs are just completing strings. An internal world model just helps facilitate the string completion. The output of LLMs is (I’m not sure if this is always true, but generally) a probability distribution over next tokens. An engineer might then figure out clever ways to sample from that distribution to improve performance, but that’s what they are. Building an internal representation of Othello seems like a good way to output legal next strings.
That’s a valid viewpoint. But if you take that viewpoint, for consistency reasons you must also take the view that almost all human intelligence is also just predictive string completion. Every author is still just engaging in a process of putting down the next word. Every painter puts down a series of brushstrokes that can be modeled as a handful of coordinates and other tokens. Every mathematician is putting down a series of symbols that are some transformation of previous ones.
When people say “LLMs are just statistical predictive models”, they’re usually trying to make some argument that somehow humans have some secret sauce that makes them intelligent, while the LLM is just mindless statistics. Half_Man_Half_Wit above said that ChatGPT had no internal model (or imagination), but the evidence suggests otherwise (though it’s hard to be certain).
That LLMs are trained statistically doesn’t seem to preclude them from building internal models. If their internal sophistication can continue to increase, we might reach general AI just through more advanced LLMs (or similar construct).
This is not required for consistency. LLMs are trained with a specified objective of predicting tokens. Humans are not. The fact that LLMs are different than humans also doesn’t mean that they cannot be intelligent. No reason that emergence can’t come through token prediction as well as evolutionary fitness survival objectives.
Why does the objective matter? All that should matter is what we can measure. If a human and LLM can respond to a prompt in similar ways, then why do we call the latter prediction and the former… creation, or something?
Our evolutionary fitness objectives are all about prediction, so it should be no surprise that rubs off on more abstract things as well.
I’m not sure I follow. LLMs are predictive string generators because that is their objective. Since humans have a different objective why are you assuming the mechanism is the same?
First, I don’t think you (or anyone) has established that the objective of humans isn’t ultimately predictive. It certainly appears predictive in numerous ways, and the objective function for much of our evolution is also clearly predictive.
But more importantly, I think it’s a distinction without a difference. If the result in the case of an LLM vs. a human writer is to just produce the next word, something that fits with all preceding words, in what measurable way are they actually different?
The same LLM that drives the predictive string generator can drive a diffusion image generator - you just have to train it on the image data. We will soon have LLMs that will train on all of it together.
We understand transformers. We don’t understand the internal logical structure of the model driving it, or why it does what it does. And that’s where all the complexity is. If there is any real intelligence, it’s somewhere in the 175 billion paramaters and the connections between them.
Since we really don’t know what consciousness is, I would not be comfortable saying categorically that there is or isn’t some form of fleeting ‘consciousness’ happening when Bing Chat crawls the web each day all day, reading the stream and fine tuning itself based on what it reads, or while it is answering millions of questions.
My gut feeling, offered with no evidence, is that consciousness is just the next level of emergence when neural bets get to a certain scale and start processing questions that require abstract reasoning and rational choices.
Below that is Kahneman’s system 1, the automatic functions that happen below conscious level, like basic pattern recognition, understanding how to walk or throw a ball, etc. Things that get trained until suddenly we just ‘know’.
Early language models looked pretty system-1. But these latest ones seem to be going beyond that. To where, I don’t know.
Well, it was in fact the Minsky quote that suggested to me that you wanted to repeal the concept of ‘understanding’ that can’t be behaviorally assessed (i. e. the association of semantic values to tokens), in saying that it must be ‘explained away’. (Incidentally, that particular quote is a good example of a pithy saying that sounds kinda edgy and deep on first brush, but which just seems to be wrong for most concrete examples: explaining the second law in terms of statistical mechanics does nothing to explain away the second law, but merely elucidates it, providing just the sort of understanding of its terms ChatGPT lacks when it comes to words.)
But if you agree that there clearly is a meaningful concept of ‘understanding’ that isn’t assessable behaviorally (consider two people on the couch, watching an English-language documentary; both seem equally immersed, but actually, only one of them speaks English, the other is simply lost in thought—there’s a material difference in the understanding between both, but no behavioral one), then that’s fine.
Again, it’s not an evidential matter. We know ChatGPT doesn’t know the meanings of words, because those simply play no role in how it produces its output. Moreover, it couldn’t possibly learn meanings, simply because they weren’t in its training set.
Meanings are correlations between symbols and objects (where ‘objects’ can be concepts, sense data, whatever: the only important element is that they’re representations of something). To somebody like Fodor, who defended a causal theory of reference, the meaning of a symbol is whatever caused that symbol to be produced within an agent. That theory, I think, is pretty much dead in the water, but there are other naturalized ways of thinking about representations—for instance, Ruth Millikan’s biosemantics, where symbols get their meaning essentially from what those representations where evolutionarily selected to accomplish. But the details don’t matter: neither the causal antecedents nor the evolutionary selection pressure is present in text.
The Othello-example doesn’t indicate anything else. The problem is that it has a very shallow semantics, with coordinates mapped to board positions, and a single-bit difference for the color (or something like that; I couldn’t immediately find how the representation used works). Beyond that, the game is just structure. And this semantics is basically ‘taken for granted’: we could use the identical in- and outputs of their model to play a different game, where, say, the mapping to colors is reserved, or where the coordinate system is flipped, and the rules changed accordingly: it would be none the wiser.
So what their Othello-GPT has learned are the structural relationships between tokens representing moves; these are taken to Othello-positions by keeping the mapping between the tokens and the board positions fixed: but that’s exactly the semantic content of the encoding. So this tells us nothing about the semantic capacities of the model.
I’m not denying that LLMs form an internal model (of a sort) of the distribution of tokens they’re working with—it’s how those tokens map to real-world concepts that’s the issue.
Sure, but successful prediction alone does not demonstrate an understanding of the domain. Take the example of predicting a time series of data for radioactive decay: whether that’s got anything to do with radioactive decay, or was, in fact, the discharging of a capacitor, simply doesn’t matter. Likewise, whether ChatGPT is talking about cats or dogs does not matter for its word predictions.
Our senses are themselves a part of the world; they’re just that particular part of the world we have direct, unmediated access to. Anything else just collapses to epistemic structural realism, which then suffers from Newman’s objection: if you’re actually willing to say that all we have access to are relationships between things, you’ll have to accept that all we can ever know is just the cardinality of the domain over which those relations are defined—i. e., how many things there are. This isn’t ‘just how things are’, it’s a contentious philosophical position that’s very hard to defend, and in this form, is all but abandoned, I think. (Indeed, you seem to be the first person I’ve ever met willing to accept that horn of the dilemma, so just to make sure, are you actually willing to accept that the sole item of knowledge we have about the world is ‘there are N objects’?)
What is needed to ground the relations is access to some part of the world that isn’t mediated in structural (or theoretical) terms, and this is given by our knowledge of our own minds (senses and all).
But it’ll never know which word maps to which concept. Consider the following domain, with a set of people (Alice, Bob, Charlie), and a set of relations (hates, loves, hits, kisses). Then consider a ‘language’ intended to model that domain: (t, u, v, w, x, y, z). There is then a corpus of sentences, ‘xwy’, ‘ytz’, ‘xuy’, ‘ztx’, ‘zvx’, ‘ywx’, and so on. A model like ChatGPT could easily learn the ‘rules’ of the domain, and produce novel, valid sentences; it would create something of an internal model for these, which encode rules like, terms x, y, z always go at the beginning or end, the other ones in the middle. But it couldn’t ever divine whether, say, ‘x’ stands for ‘Alice’ or ‘Bob’—because the information simply isn’t there. It has no handle on these mappings, and they’re completely irrelevant to its operation. Whether it can successfully produce valid sentences depends in no way on their interpretation.
Now, of course, real language is vastly more complex, and there seems to be a common intuition that somehow, if you pile on enough complexity, thinks start to, well, just kinda work out. Enough constraints are added onto each term to ‘pinpoint’ its meaning. But in fact, the addition of complexity just makes things more underdetermined, because the number of potential mappings (interpretations) scales with the factorial of the number of elements (the number of permutations), while the number of constraints added with each elements only scales linearly (it can maximally stand in a relationship with each of the elements already in the domain). This is generally known as the ‘just more theory’-problem: whatever you add to pick out the ‘right’ way of fixing reference, will itself be some structure in need of interpretation, and only fix reference in the ‘right’ way under your intended interpretation, and consequently, fix it in a different way if that interpretation itself is changed.
Well, I can tell you that it’s false that the same thing must be true in our brains, because I have constructed an explicit counterexample to that claim: in this article, I present a model of how mental states acquire meaning by having a certain sort of non-theoretical, non-structural access to their own properties, modeled by a self-referential, self-representing sort of process. (See also the popular-level account here.)
Naturally, I don’t know that things actually are that way, but there does not seem to be any prior grounds according to which they can’t be.
I’m not sure how you reach that conclusion. There are other properties that things share beside quantity. We perceive many properties of objects that we can use for groupings or relationships.
Yes, exactly. And what I claim is that everything is just structure. Our entire understanding of everything lies solely in the structure of relationships between objects. And while people may have access to richer sensory input than what has been fed to LLMs so far, it is not different in any fundamental way. Words, sounds, pictures, and other input are just data.
We can go much deeper than that. There’s no reason, for example, that it should have any notion of a board, or a grid, or anything like that at all. And certainly not any notion of colors or pieces that flip or even the fact that this is a “game”. Because all that matters is the connectivity of the nodes and the rules that dictate the transformation of states for each node.
If you fed the same problem to 1000 people, and later asked them to explain themselves, perhaps some would describe Othello as we know it. Some would offer minor distortions of the rules, like using different colors. Some might not appreciate that this is a game, perhaps claiming instead that this is a primitive simulation of bacterial growth. Others may refuse to make any concrete claim about it at all, simply showing their rules for how one game-state transforms into another.
The thing is–these are all still Othello. Any two of these people playing the game over the phone by exchanging moves would not–could not–discover that they have happened upon different interpretations of the ruleset.
The game of Othello is that structure. The plastic tokens and the gameboard are just irrelevant details.
Of course not. I find that a strange claim to make since you just acknowledged that the LLM did discover structure in the game of Othello that, while fairly simple, was still substantially richer than just a count of things. It would not be possible for the LLM to make accurate moves if that’s all it had access to.
I’ll have to get to your other points later (it’s getting late). But I deny that “grounding” is necessary or even possible in the way that you seem to describe it. What reality offers us is consistency, which allows our senses to take multiple “readings” of a thing and develop a theory of its existence. If reality gave different answers each time we looked at (or heard, or tasted, etc.) it, we’d be in a hopeless situation. But neither are our senses perfect, and we have to deal with ambiguity and some false readings. Likewise, the corpus fed to ChatGPT describes–imperfectly–a somewhat consistent reality. There is a fairly consistent grammar, and the symbols are largely used in a consistent way. It very much appears that however imperfect its input, it is sufficient to give it a model of cats and dogs and other things that matches the one in my brain.
Well, once again we seem to be miscommunicating rather badly. Let me re-iterate my clarification of the previous misunderstanding and then address this latest one where you attempt to attribute to me opinons that I absolutely do not hold, as well as apparently misconstruing Minsky’s statement.
You accused me, with a good deal of sarcasm, of “dodging the question” by responding to a different question than the one under discussion: “Sure, if you substitute a completely different question for the one being asked, your answer can be whatever you want it to be! Am I a millionaire? Yes! So, do I have a million dollars? No! I merely substituted the question ‘Am I a living human being?’ and answered that instead!”.
As I clearly explained in post #735, we were talking about two different issues, a fact that I hope you at least recognize and are willing to acknowledge. I was using the Turing analogy to address the question of what “understanding” means, and that, like intelligence, it can only be assessed in behavioural terms. I was making no claims whatsoever about any equivalence between the internal states or processing mechanisms of ChatGPT and human cognition.
Yet you somehow concluded that I was claiming that “every meaningful question about machine cognition should be decidable in behavioral terms” – a claim that I absolutely never made – and then sarcastically attacked that strawman.
Now, as to your statement that I quoted. No, I absolutely do NOT agree with that at all and your conclusion to that effect is so outrageous that I rather suspect that it’s quite disingenuous, a supposition which seems to be supported by the absurd example you gave.
You’re well aware of Turing’s “imitation game” proposal and the reason for it. You must be equally well aware that the closely related concept of “understanding” can also only be assessed behaviourally. More specifically, the concept of whether a human or artificial entity “understands” something can only be assessed subjectively on a sufficiently broad set of relevant behavioural responses. This is how we evaluate everything from machine intelligence to a child’s first spoken words, from PhD dissertations to the support needs of mentally challenged individuals. Your example of the two people on a couch is absurd because none of the behaviours are sufficiently relevant responses to be informative. If you want the behaviours to be informative in assessing the two people’s understanding, ask them questions about the documentary and correlate their responses with the content of the documentary. Your original premise is so absurd that I honestly believe you’re just playing some sort of silly contrarian mind game here.
As for Minsky’s statement, I am old enough to have been present when he said it at a small gathering I attended years ago (although I’m sure it’s been said many times) and it was not meant to be “pithy”, “edgy”, or “deep”. It was meant only to express an enduring truth about artificial intelligence that had frustrated him his entire career, that once the underlying principles of any AI machine are explained, there is a tendency to dismiss them as mechanistic. This is as true today as it was in the 60s. It takes the form of dismissive statements like “it’s just {doing this rote thing}”, or "it’s only {doing brute-force calculations}. Look out for “it’s just …” and “it’s only …”. It harks back to Hubert Dreyfus claiming that no computer would ever play better than a child’s level of chess. Yet we see it here in this very thread about ChatGPT.
Well, it’s a simple mathematical theorem: if all you know about a domain is its structure, all you know about the domain is its cardinality, because that’s all the structure fixes—every other domain with the same cardinality supports the same structure, by means of push-through. That is, structural facts only fix facts of quantity.
As Max Newman originally put it in his objection to Russell’s brand of structural realism:
Any collection of things can be organised so as to have the structure W [where W is an arbitrary structure], provided there are the right number of them. Hence the doctrine that only structure is known involves the doctrine that nothing can be known that is not logically deducible from the mere fact of existence, except (‘theoretically’) the number of constituting objects.
(As quoted in Ainsworth, Newman’s Objection.)
Ainsworth goes on to elaborate:
For example, being told that a system has domain D = {a, b, c} (where a, b, and c are arbitrary names for three distinct but unspecified objects) and instantiates a relation R = {<a, b>,<a, c>,<b, c>} tells us no more than that the system consists of three objects, because some elementary set-theory reveals that any three objects instantiate seven non-empty one-place relations, 511 non-empty two-place relations (of which R is one) and 134,217,727 non-empty three-place relations.6 Being told that they instantiate R is both trivial (insofar as it follows from some elementary set-theory) and perversely specific (insofar as R is just one of the 134,218,245 non-empty relations they instantiate). Thus being told that the system has structure <D, R> is being told no more than that it contains three objects, because any system containing three objects can be taken to have this structure, along with a vast number of other structures (any tuple whose first member is D and whose other members are amongst the 134,218,245 relations instantiated by the members of D is a structure that can be taken to be possessed by any system containing three objects).
Now, you may be tempted to argue, well, but <D, R> isn’t just any structure, it’s the right structure, and special in some sense. But any way to pick out that structure is, as noted, just more theory: just more structure added, which will be subject to the exact same problem. Hence the need for something non-structural to avoid this utter trivialization.
Well, of course. You are talking about a concept of ‘understanding’ that can be behaviorally assessed, whereas the concept of ‘understanding’ at issue is one that can’t be. That’s why the couch-example is pertinent: differences in understanding can exist without differences in behavior. There are thus facts of the matter regarding understanding that are not accessible via facts of the matter regarding behavior.
That doesn’t entail that understanding can’t be manifested in behavioral differences, just that the latter don’t run the full gamut of understanding. In contrast, Turing’s proposal was to create a setting that’s all behavior. This is a legitimate move, as long as you’re upfront about what you bracket out (as Turing was, and you aren’t).
So no, I’m not ‘well aware’ that ‘understanding’ can only be assessed behaviorally, because that’s just false. You can choose to only focus on behavioral differences, but then, you’re asking a different question. Trying to pass off an answer to this question as being relevant to the original one then entails a commitment to the thesis that the behavioral level is all there is.
I know nothing about computer science, coding, and whatnot, so I can’t speak of how ChatGPT works, only that the results are quite impressive at this stage of the game, and one can only imagine what AI holds for humanity in the future.
If we can loosely equate apps like ChatGPT to the cognitive part of the human brain (the cerebral cortex), and the memory control part (the hippocampus), then I believe we can equate apps like self-driving cars and the software that controls robots (e.g. Boston Robotics’s Atlas), to the motor part of the brain (the cerebellum).
I think the writing is on the wall that AI will significantly surpass all three of those parts of the human brain in a relatively short period of time. If we want to retain something superior to AI in the future, like emotional intelligence, then we best not try to emulate the amygdala, or the emergent mind may be something we can’t predict or control.
At the end of the day, all this has really gotten me (and others) really thinking about what the nature of consciousness is. And if a simulacrum of consciousness is indistinguishable from “real” consciousness, is that a real difference? After all, we developed consciousness after millions of years of evolution and biological processes made up of organic matter with no consciousness in and of itself. Yet through complexly interactions, here we are.
A few years ago I would never have thought a computer can develop anything resembling consciousness, but now … I think it is a real possibility, if not an inevetibility. Or at least something that is indistinguishable from this fuzzy concept of what it means to be truly conscious.
Of course there’s a behavioral difference between them. If you later ask both of the viewers about the movie, their responses will be very different. A response to a question about a movie is a behavior, too.
And it’s all well and fine to say that “measurable behavior isn’t everything”, but whatever else there is besides measurable behavior, what can you ever say about it, beyond just wild guesses?
Maybe you never made that claim, but I will. There are plenty of questions about cognition that can’t be decidable in behavioral terms… but those questions also can’t be decided in any other terms, either. And I would maintain that a question that can’t be decidable at all, doesn’t count as a meaningful question.
I can’t make heads or tails of the Ainsworth quote except that it seems to say we’re obligated to consider all possible relations on a set. Which… we aren’t.
Let’s try something more concrete. There is a mathematical structure called the Rubik’s Cube group. It reflects the set of permutations on the cube and has cardinality 43,252,003,274,489,856,000.
There is another group, the cyclic group of order 43,252,003,274,489,856,000, which has a different structure (that of a clock).
Are you telling me that these two structures are indistinguishable because the only thing I can know about them is the cardinality (which is the same)?
Yes, if I have 43,252,003,274,489,856,000 abstract, unlabeled objects, they can support the structure of both the Rubik’s group and the cyclic group. That is trivial and useless. What is not useless is that if I’m presented with a set of legal operations over that domain, eventually I can with high probability discover the structure behind it. Especially if I have reason to believe that the structure is “simple” or “elegant” in some way.
Sure. I’m not claiming that a difference in understanding can’t lead to differences in behavior (which is obviously false). I’m saying that there can be a difference in understanding without an attendant difference in behavior. For certainly, even if neither of the viewers is ever asked about the movie, if will still be the case that one of them has understood the film, and the other hasn’t.
And even for simple automata, studying their behavior is generally insufficient for finding out their internal state—this is a theorem due to Edward Moore from a famous paper, ‘Gedanken-Experiments on Sequential Machines’.
The point is that once you’ve told me, ‘there are three things’, there’s no point in telling me, ‘they instantiate relation R’. Because, duh, I know that already: that follows from the fact that there’s three things! So all that relational knowledge tells me is the number of objects, because the relation immediately follows.
If you’re then saying, ‘but R is a special relation among all of those instantiated by those three things’, then you’ll have to tell me what you mean by ‘special’. Either, you can tell me ‘the relation is ‘hates’, and hence, Alice hates Bob, Alice hates Charlie, and Bob hates Charlie’. Then, yes, I’ll know what makes R special among all the others, in what sense it is the ‘true’ relation fulfilled by those objects. But then, you’ve also given me a piece of non-structural knowledge, namely, the intensional characterization of R (i.e. by virtue of what two elements of the domain stand in the relation), as opposed to its extensional characterization (i.e the elements that stand in that relation).
(Incidentally, Russell’s reaction to Newman’s criticism has been described as a 'classic Homer Simpson ‘Doh!’-moment, acknowledging that the objection is devastating to any account of knowledge as pure structure. Russell’s later views have been interpreted in various ways, but I think the most plausible reading is that he championed a view of ‘direct acquaintance’ with perceptual relations, which yields the necessary grounding.)
If, then, you want to maintain that we only have structural knowledge, then you’ll be obliged to give the explication of ‘special’ in terms of structure. That is, you’ll have to give some relation that characterizes what is ‘special’. But then, the problem just iterates.
As for the groups, what you have structure-wise is a set of elements and a relation modeling the group operation. As you say, the abstract set of elements supports the structure of both groups. So, the question is: using only structural knowledge, how do you pick out one of those relations, and not the other?
To put this back into a language context: Some folks have asked whether, given only a dictionary of some language written in that language, it’d be possible to learn that language. Of course, it would be very difficult… but we would still be able to learn at least something. For instance, an alien perusing an English dictionary would still be able to notice that the word “the” appears in many different definitions, but the word “xylophone” appears in very few definitions, and thus conclude that the string “the” represents a much more common word in English, possibly even one that represents a special grammatical role. We would, in other words, be able to determine the number of words in English, but that wouldn’t be all we could determine: There would also be some level of structure that we could discern. I don’t know if it would be possible to determine all of the structure, sufficient to be able to translate that language, but we could certainly determine some.
I suppose the test would be to train an AI like this, on a dataset that contains ordinary text in most languages, but with one language represented in the training data only in the form of an extensive dictionary written in that language (of course, for ease of the researchers, that one dictionary-only language would have to be one that’s known to scholars). If the resulting AI were able to translate between that dictionary-only language and others, we would know that a dictionary is enough to learn a language from (at least, given known commonalities between all human languages: It might not be enough for a true alien to decipher).
The conclusion is false because the premise is a misleading straw man. You’ve taken my claim that relevant behaviours (such as responses to questions) are the ONLY way to assess understanding of some specific issue, and distorted it into a claim that ALL behaviours must reflect that specific understanding, which is a claim that no one has ever made. Let me be as clear as possible on this – your couch example fails because sitting on a couch like a dummy is not an informative behaviour. But the two people’s understanding of the movie can be assessed behaviourally by asking them appropriate questions about it. It’s not necessary to probe their brains to examine changes to their neural structure to see if they’ve understood the movie.
Of course! I’ve never claimed otherwise. Indeed, one of the important corollaries of my contention that abstract qualities like intelligence or understanding can only be assessed behaviourally is precisely the fact that different implementations of cognitive mechanisms (e.g. human vs machine) can produce exactly the same behaviours. Thus the manner in which they were achieved is irrelevant to assessing whether or not they exist – which was precisely Minsky’s point, and is probably the fourth or fifth time I’m repeating this.