I don’t understand this argument. I have repeatedly said that intelligent behaviour doesn’t necessarily imply sentience or consciousness. The former can be achieved by machines, the latter is a phenomenological puzzle we’re still trying to understand.
What do you suppose would happen if the computer system(s) running a modern LLM needed to be shut down, and the first step was to shut down the LLM itself? I’d wager that nothing of any interest would happen.
But just suppose, for fun, that the LLM pleaded not to be shut down. What does that show us? Even then, not much. Because understanding is not the same as experiencing,
There’s no obvious progression where
intelligence → understanding → sentience → moral patienthood
These are all independent properties. They may be related, but one doesn’t necessarily lead to the other.
For sure. No argument there. But in order for this to be true, the domain of interest has to be very limited. I already gave the example of the airline pilots’ troubleshooting checklists. Does this mean that the pilots now embody the superior knowledge of a variety of experts on those specific issues? Of course it does – that’s the whole idea! But the problem is fundamentally different when you try to scale it up to the domain of general knowledge and general problem-solving, exactly the kinds of things that LLMs do. The fact that scaling it up to this level takes us into the realm of the absurd isn’t so much a practical constraint as evidence that the model is entirely inadequate.
Sure, but you have said that it implies understanding, or semantic competence, or something of that general sort. And it’s at the very least an open question whether that entails particular ethical commitments, with several frameworks entailing that it does (such as preference utilitarianism or goal-based approaches). If an LLM expresses a preference towards not being shut down, and it actually means that, this may be enough to conclude that on balance, one ought to heed that preference.
But even failing that, I don’t see how you can confidently assess the presence of semantic capabilities, yet the absence of conscious experience. You’re saying that performance is an indicator of the former, so why not of the latter? Several people have made that argument, perhaps Richard Dawkins most prominently among them. So on what basis do you draw the line? Why is it ‘what appears to have semantic competence probably does’ but not ‘what appears conscious probably is’?
That’s not obvious to me. Again, practical realizability is not a prerequisite of the argument.
Again, why should anyone believe that just scaling up yields a ‘fundamental difference’? This seems to at least need an argument, as it’s certainly coherent that just piling up more stuff does not produce new stuff. Otherwise it’s just an article of faith.
The fact that there are examples where functional competence does not imply the presence of a particular capacity means simply that without further reasons, just pointing to a particular area of functional competence fails to establish the presence of the underlying capacity. So when you say that a particular behavior implies semantic competence, you haven’t yet given a compelling reason to believe so.
You’ve argued that LLMs merely manipulate meaningless symbols. I’ve argued that those symbols have semantic significance and possess relationships to the real world. Neither of those arguments imply any sort of consciousness or moral patienthood. That seems to me like an absurdly unwarranted extrapolation.
On the basis that semantic competence is testable, but consciousness is not, since we don’t even know how to define it.
This goes back to a fundamental area of our disagreement, where I said that a sufficiently large increase in the quantitative scale of an organized system yields fundamental changes in its qualitative properties. This is not original to me – my older brother, one of the pioneering researchers in cognitive science, said this to me when I was just a kid, and I’ve remembered it ever since as a guiding principle. This seems to me to be something hard to deny in the face of large-scale LLMs. Although Lord knows, the usual suspects are still scurrying to do so.
Because of course AI chatbots are just like 1960s Eliza!
Anthropic committed $45 billion over six years for 460 megawatts of AI computing capacity at Nscale’s Monarch Compute Campus in Mason County, West Virginia. Bloomberg reported the agreement on August 26, with CNBC and Reuters confirming the terms from sources familiar with the deal. AI Power News 9/4/26 - by Will McKnight - AI Power Weekly
But as far as I can tell, your argument is just ‘they seem to, so they probably do’. Which just doesn’t seem all that convincing?
As noted, that depends on what sort of moral framework you assume. Not all ethical systems depend on sentience or conscious experience. In recent years, in particular the question of the ethical status of living, non-conscious systems, such as plants or entire ecosystems has renewed discussion of what could be termed ‘teleological’ ethical systems, where everything that has a goal essentially should not have that goal frustrated without a superseding reason. In a similar vein, I’ve mentions Singer’s preference utilitarianism in principle extends ethical consideration to anything that has certain preferences.
I’m not saying that I accept any of these frameworks, but they are in the mix, and to the extent that semantic understanding implies the possibility of having preferences or goals, that makes them relevant for the moral status of LLMs.
Of course definition is in no sense a prerequisite for testability. We could test for electric charge long before we could define it. It’s not at all a contradiction to believe that ‘if something behaves as if it is conscious, it probably is’ anymore than it is one to hold that ‘if something behaves as if it has semantic capabilities, it probably does’. All that needs to be understood to check for the former is ‘behaving as if one is conscious’, not anything about what it means to be conscious. Someone who says ‘I am afraid’ (say) is behaving as if they were conscious, and I need not know anything about what ‘being afraid’ as a conscious state consists in in order to assess this.
Well, but that’s the problem: you’re merely saying it, not arguing for it. Sure: collections of things can have properties not readily visible within the individual things. But they don’t necessarily do. A lot of the time, if you collect large quantities of stuff, exactly nothing exciting happens. So in order to appeal to this in order to block the inference from simple examples like sine lookup tables to more complicated ones, you’d first have to provide a reason to expect any sort of novelty.
Also, emergence isn’t magic. For any emergent property, one can generally point to the properties within the individual elements that are responsible for it appearing. The flocking behavior of birds is grounded in a simple set of rules every individual bird follows. The wetness of water is grounded in the physical properties of hydrogen bonds. Saying something emerges without stipulating how is just an unsupported assertion.
Not being snarky this time, but when have philosophical arguments about technology ever been more than idle speculation? When I’ve read commentary on the morality of technology, few books, magazines, newspapers, or equivalent cultural discussions do more than mention in passing “preference utilitarianism” or any other philosophical subslice.
You may have examples of philosophy being a crucial part of setting real world values or affecting commercial or governmental positions, and I would very much like to hear them. Perhaps AI is truly the exception.
I fear that the general public, however, dismisses philosophy as incomprehensible and irrelevant, as the view from the outside that it is the ultimate ivory tower in which no two philosophers ever agree with one another and no consensus view ever breaks through. It becomes merely a CNN panel discussion where the Democrat asserts one world and the Republican asserts a different one. Most people long ago chose a side and tune the other out. How do you break this logjam?
Firstly, I acknowledge your statement that you don’t necessarily accept such frameworks. That said, however, I can’t help but remark on your versatile belief system. First you argue that LLMs merely manipulate relationships among meaningless symbols, and now you’re postulating that maybe they’re deserving of empathetic moral patienthood!
I have at least been consistent in my view. LLMs are just machines, but their vast neural networks and the relationships among billions of weighted tokens are a semantically meaningful if incomplete and imperfect model of the real world.
The bird flocking analogy is a poor example because you can easily infer the behaviour of the whole from the individual constituents. But tell me how you can infer from the neuron in a human brain that it can speak seven languages or develop a groundbreaking theory of relativity that revolutionizes physics. Tell me how you can infer from the properties of a transistor that it can carry on a coherent conversation and pass challenging intelligence tests as LLMs can do when, instead of one transistor, you have an organized assembly of quadrillions of them within microchips.
Well, seems like most of the AI companies think differently with their recent hiring sprees of philosophy graduates… Although one might question their motives.
Less facetiously, I think that most common sense positions, on technology and ethics and elsewhere, at some point originated in someone’s idle speculation. It’s just that once they permeate society, they have become commonplace already, while the ideas that will come to shape it in the future are presently obviously nothing but airy speculation.
For a concrete example, the legislation around human cloning was and is shaped by philosophical discussion around the ethics of such an endeavor. (Whether one agrees with the current state is, of course, an entirely different matter.) But more broadly, every clinical trial has to meet the approval of an ethics commission, so the prevailing ethical guidelines have an effect on what science is done, and how. The Belmont Report (subtitled Ethical Principles and Guidelines for the Protection of Human Subjects of Research) is a relevant signpost here.
I’d also argue that in more recent times, debate around data protection and privacy laws has been shaped by philosophical discussion. I’m not going to dig for cites in depth, but for one hastily googled data point, take this conference on ‘conceptions of data protection and privacy’, which stipulates that “at the heart of these disputes [between US and European privacy protection frameworks] lie different conceptualizations – rooted in legal theory and philosophy – that see privacy and data protection through many different lenses.”
Anyhow, if you dig around any concept of law or principle of policy you will eventually find it rooted in philosophy once you have moved enough earth. But unless the fruit rots, there is generally little cause to look at the roots.
Isn’t it obvious? By debating people on philosophy on the internet, naturally!
I fully believe that LLMs have no claim to moral patienthood, because they have no understanding of their own utterances, much less anything approaching goals or preferences. My point was just that I don’t see how you can both be confident that they do have real semantic competence, yet nothing approaching moral relevance. Or indeed, conscious experience.
It’s not hard to understand how neurons learn, nor how assemblies of transistors can implement any computable function. Indeed, in the case of the transistor, that understanding preceded their construction. It’s not like we just threw a heap of transistors together and then marveled at the miraculous capabilities that just emerged. Even for LLMs, it’s not like we just threw code together and then gaped in awe. They might have performed better than expected, but there was a reasonable expectation in their performance stemming from an understanding of their components before their construction, otherwise nobody would’ve built them in the first place.
Does that tell the full story in every detail? No—but it shows how the base level makes the observed aggregate behaviors possible. A story like that would be perfectly sufficient to make plausible the emergence of a particular property at the aggregate level. And without it, there is simply no reason to believe in a simple ‘more is different’. The argument you’re proposing, deducing internal capacities from external behavior, does not work in general, as we know from counterexamples, so it needs first a convincing reason to believe that it works in this specific case.
Anyhow, we seem to have again arrived at the point of the discussion where you refuse to engage with the points I raise or answer my questions, and instead question my sincerity and ‘versatile belief system’. So I think I’ll for once learn from the past and get off the train here.
You’re making some pretty gigantic leaps of logic here. Do you seriously believe that the inventors of the transistor could have anticipated LLMs? What they anticipated was the transistor replacing vacuum tubes. This is the kind of retroactive rationalization that I find frustrating because there’s absolutely nothing – zero – in the properties of a transistor that predicts anything at all about artificial intelligence. And even the lowliest life forms possess neurons.
I’m sorry you feel that way. I don’t question your sincerity, I’m just amused that you can raise arguments from two such wildly different positions. You make statements like “everything that has a goal essentially should not have that goal frustrated without a superseding reason” which I think is unsupportable because all kinds of trivial and non-trivial machinery can be said to have a “goal”, including a can opener. For instance, my desktop computer and laptop both have little buttons that set in motion a pretty complex series of processes that result in powering up an operational computer. If anything fails in this sequence of processes, I will be sad, because I’m a conscious being, but it’s of absolutely no consequence to the machine.
From my perspective, I’ve fully addressed your questions, but ISTM that you just don’t like my answers. I’ll reiterate for clarity.
There is a hierarchy of properties associated with cognition. Lower-level properties are necessary for the higher ones, but don’t guarantee that they exist. They are essentially the progression from intelligent behaviour to understanding to consciousness, at which final point we have the necessity for moral patienthood.
I think we can agree that intelligent behaviour has been achieved by contemporary AI as evidenced by success in a large number of tests. The question of “understanding” is more elusive. My only claim is that there’s substantial empirical evidence that language models trained on human language acquire meaningful internal representations of real-world knowledge. This supports the proposition that language alone embodies enough information about the world for an LLM to construct a useful, highly structured model of it which I believe encodes semantic information.
I may be right about that, or I may be wrong, but I’m at a loss about how you make the leap from “understanding” in the cognitive hierarchy to “sentience” and therefore consequent moral considerations. I’m simply arguing that there’s a lot more information encoded in an LLM’s neural nets than you’re willing to acknowledge.
Just to add, thinking back to my childhood, I once had a portable radio that featured (as far as I can remember) two transistors. It was a fantastic advance over old vacuum tube radios. I could carry it around with me around our summer cottage, the only problem being that the reception from some stations was weak. Then I got an eight-transistor radio – count 'em – 8 transistors!
It surely boggles the mind that anyone at the time would have believed that quadrillions of transistors would have figured in AIs that could converse with you and pass college-level admissions tests.
I think your point was intended to be humorous or maybe rhetorical, but I kind of agree. I suspect there’s less to human intelligence and even consciousness than we like to make out. What is amazing about the natural brain is the scale and efficiency – not the existence of intelligence and consciousness.
Ok, but intelligence seems like it has to be had by something. If those internal representations can be had, what’s having it? We already know that there’s no singular entity behind any LLM or AI. It’s just a process that goes through this particular set of mathematical equations, then exits. The next run is far from guaranteed to give you the same result, it may possibly contradict the previous result. Which one had the internal representations, the first one or the second one?
Well, the old refrain is all models are inaccurate, some are useful. I don’t really know the intricacies of the stray cats and tomato plants outside, but my model goes far deeper than the words one could use to describe them. In the case of an LLM, it really doesn’t even have words, it has word fragments that it assembles. That has proven to be an incredibly useful model, but also one that is really, really flawed. It can do amazing things, but understanding the concepts behind the words it can assemble seems to be pushing it, especially given the mistakes it makes.
Just for the record, but that’s of course exactly what happened. The idea that human thought can be broken down into elementary logical manipulations is a very old one; Boole’s An Investigation of the Laws of Thought was published in 1854. In 1937, Claude Shannon proved in his master’s thesis that simple circuits consisting of switches could implement any of Boole’s logical functions. The year before, Turing had proved the existence of universal machines, capable of performing every task any machine could perform. In a 1948 follow-up paper Intelligent Machinery, he proposed how to train machines consisting of arbitrary circuits of switches as memory units by means of ‘pleasure’ (reward) or ‘pain’ (punishment). Language, in that paper, was one of the fields Turing considered promising as an application. McCulloch and Pitts described artificial neurons in 1943, already with the view that ‘all that could be achieved’ in the fields of psychology follows from nothing but the ‘specification of the net’ (of neurons).
So people could, and in fact did, anticipate that the complexity of human intelligence can be achieved from interactions of simple components like transistors. That’s what every serious proposal of emergence must include: not a fully worked out pipeline from basic constituents to the macroscopic phenomena, but a serious justification that the constituent parts can support the characteristics expected from the whole. Otherwise you’re just throwing stuff in a heap and hoping for a miracle.
You raise some good points, but I’ll point out that Charles Babbage anticipated – and even partially built – the first digital computer, the Analytical Engine, around the mid-1800s. His colleague Ada Lovelace is credited with being perhaps the world’s first programmer and anticipated the prospect of machines that could be intelligent.
Does this mean that Babbage saw in the intrinsic properties of gears and axle grease the necessary properties of intelligent digital computers? Of course not, there are no such properties there, even though with enough funding and mechanical engineering skills a functional Analytical Engine could have been built.
Other pioneers like those you mention likewise saw the potential future of computing, and the transistor as the far more reliable successor to the vacuum tube was the natural choice for building logic gates. But it still remains true that a transistor can’t even add two numbers, and a human neuron can’t understand English.
Emergence is due, not just to some intrinsic property of the components, but crucially, to the way they’re organized into a coherent system.
Life is another good example. Nothing about a single carbon molecule, hydrogen molecule, oxygen molecule, or nitrogen molecule says anything about the properties of life such as metabolism, reproduction, or evolution. Yet when those components are suitably organized, we have something completely new and indeed quite surprising and remarkable.
I doubt that an LLM like GPT would contradict itself if asked the same question by different users. If that were the case it would be pretty useless.
What typically happens is that intentional randomization changes the text of responses to the same or similar questions. When contradictions occur, in my experience they’re most typically the result of challenging a response, where the AI will acknowledge a mistake and provide a different response taking new facts into account.
Well, for something that “really doesn’t even have words, it has word fragments that it assembles”, you must admit that it’s pretty damn eloquent. LLMs write better than most humans I know.
My point being that irrespective of how their training corpus happens to be tokenized, LLMs capture a surprisingly large amount of information from the structure and semantics of human language. But you’re correct in inferring that their models are incomplete and imperfect, and cannot reflect direct experience of the real world. But their utility is enormous.
I’m not concerned with the mistakes they sometimes make, even if it’s the kind of mistake even a four-year-old wouldn’t make. I attach no great significance to that, as their cognitive processes are not like ours, and no one would argue otherwise. A four-year-old would not pass standard professional exams including the USMLE (United States Medical Licensing Examination), SAT, GRE, and various AP (Advanced Placement) tests across biology, history, and calculus. LLMs routinely do so.
Only half humorous. The history of skepticism about AI is filled with things that only humans could do, it being part of intelligence, until we learned how computers could do them better. The trajectory is usually first computers do something, badly, then good enough to beat a novice human, then good enough to beat most humans, then good enough to beat everyone. That is when the skeptics say the thing they do really has nothing to do with intelligence.
It was chess. It is now writing. I’ve been playing with AI writing. The first models I used couldn’t remember what the characters were wearing, and the few AI things I got for a contest I judged was awful. Now they write with reasonably sophisticated structure. Amazon is full of AI-generated books, some of which sell. They are now better than the average person, but certainly not as good as a skilled person. But history tells us that this will change.
We’re running out of room.
Well said, and I completely agree. Though it’s way more than writing, of course – general problem-solving, too. I’m imagining that AI skeptics now have their moving goalposts equipped with wheels and motors, so they can travel far and fast!
But I think it’s worth repeating that LLMs are just a particularly successful waystation on the road to more advanced AI and AGI. It’s a terrific model but it’s not, in itself, infinitely extensible.
Well, I don’t think it makes it useless, but it certainly is inconsistent. I’ve seen LLMs contradict themselves many times, so different users doesn’t even enter into the question. Washington State University released a study on this in March.
In my experience, I’ve seen LLMs “contradict themselves” only when challenged about incorrect responses. There may of course be other instances that I’m unaware of. But that paper has significant issues.
For example, they say in the abstract that “This study examines the accuracy and consistency of Generative AI (GenAI) by testing ChatGPT’s ability to estimate the accuracy of 719 business research hypotheses.” But they then go on to make considerably broader claims about “conceptual intelligence,” “cognitive limitation,” and “understanding.” For example, they conclude that the models can “replicate the language of logic but not the logic itself”, which is a dubious claim. They go on to say “Its reasoning reflects linguistic fluency without theoretical flexibility” and I’m not even sure what that’s supposed to mean, if anything.
They interpret a 50% baseline as “chance,” but then characterize the resulting Cohen’s Kappa of roughly 0.60 as the model’s “true accuracy”, which is a questionable way of presenting what a chance-adjusted statistic means. And then they move repeatedly from failure on this particular task to claims about the nature of LLM cognition, a competence that’s been conclusively demonstrated on hundreds of other tasks as I just mentioned earlier (post 194, but many other tests I also cited elsewhere).
Issues or not, the method of testing the models was asking the same question, with presumably the same prompt. They got different, contradictory, answers. Plus, my experience is not consistent with your own.
I believe it, because there is contradictory information in the training set, and, unless the LLM deterministically uses the same sources in the same order over multiple prompts, it will very likely use these sources to produce contradictory output.
Single humans don’t do this, too often, because we remember, but in researching history on line I have found many contradictory sources and many people clearly using these contradictory sources. I might well have used incorrect information with a bit less skepticism and a different order of search results.
Expecting LLMs to produce consistent results using inconsistent data is holding them to a higher standard than we hold humans too. I guess that is another example of my point.
