The answer depends entirely on your definition of the word “know”. Does the Chinese Room know Chinese?
Then you come up with a definition that we’d all agree on. I gave you the first one that popped up on a search, so I’m not married to it.
Otherwise, for certain definitions of reasoning, knowledge, and intellect, my big left toenail is sentient.
Edit: I should probably answer your question as well. No, the Chinese room does not know Chinese.
I will, right after I succinctly resolve the hard problem of consciousness.
My point is that it is incredibly hard to define what it means to know something. Epistemologists have argued about the definition of knowledge since literally Plato.
Sure, I mean, it’s indisputable that all of the information required to construct your big left toenail (and every other part of your body) is contained within the cells that make up your nails (and that the living cells are the bottom of your nail are actually undertaking that process as we speak). I don’t know about sentience, but on some level your toenail “knows” how to make a human.
Ok, great - why not? What about the Chinese Room makes it so that the system as a whole does not know Chinese? What if the “system” was some kind of translation chip in a person’s brain that translated Chinese for them? If the chip worked well enough, how would it be possible for anyone else to distinguish between that person and a “true” Chinese speaker?
A person who is cortically blind has eyes that function but an occipital cortex that cannot process visual information. They experience a phenomenon called blindsight, where even though they cannot experience sight (as in, their conscious mome is totally unaware of any visual stimuli) if you toss a ball at them they will catch it by reflex. Does a cortically blind person with blindsight know that a ball is headed their way?
Whether I can distinguish the difference doesn’t matter. One is a simulation, the other is knowledge.
I guess if you aren’t going to go along with a standard definition of knowledge and reason, nor give us one that we might work with, there’s not much use in continuing the conversation.
I’m sorry that I can’t casually do what Plato, Aristotle, Descartes, Locke, Hume, and Kant all failed to do.
What sort of objective meaning does a vague term like “knows” have here? If it repeatedly gives me correct answers to those sorts of arithmetic questions, then yes, it knows how to do arithmetic. If it fails to correctly do much larger sums, then yes, its model is lacking beyond some limits.
Again, muddying with waters with vague terms like “knows”. I’ll turn the question around. If I was an alien who had no experience of earth life, and I asked an AI this question, and it identified the animal, and then I prompted it for further information – its general behaviour; is it friendly; do higher level beings on the planet enjoy them as pets; what do they eat; how long to they live; what makes them attractive to the higher level beings? If it provided answers to all those questions, the salient criterion here would be, not does the AI “know” things in some abstract experiential sense, but does it provide useful knowledge.
John Searle has somehow managed over the years to elevate this nonsense into a pretentious ball of philosophical bullshit. He was an AI skeptic on the level of Hubert Dreyfus, both of whom were initially wrong about most things about AI, and eventually, wrong about everything. Of course the Chinese Room “knows” (i.e.- can read and write) Chinese. We know this because it’s doing it.
I know the Chinese Room argument is somewhat more involved than this simplified summation, but in my mind the fundamental fallacy is peering inside and observing that the little man inside is just receiving notes in Chinese, cross-referencing them, and writing prepared responses. The little man certainly doesn’t understand Chinese. But the overall system self-evidently does. Because the only difference between the overall system and a native Chinese speaker is the size of the corpus of reference material. If the corpus is large enough, then there is no distinction.
The fundamental “argument” of AI skeptics always comes down to “AI is mechanical, therefore it’s stupid”. The argument was plausible in the days of Eliza. It’s becoming less and less tenable today.
I asked my wife to speak to my left foot, as my big toenail down there would understand her and respond back by controlling various muscles in my foot and toe. After flipping me off, she decided to humor me as I looked at least somewhat sincere in my request. I told her to ask it what 2 + 2 was. Suddenly, my toe tapped on the floor four times. I asked her if she could prove that that wasn’t my toenail doing the calculation and passing instructions on to my toe muscles. She could not (nor did she give it much thought at all).
So, apparently my big left toenail is indeed sentient.
Thanks!
I agree with you that at least for some definitions of “know” this is true. However…
I don’t think that peering inside is a fallacy. I think it demonstrates how a non-conscious system could act as if it “knows” things that the potentially conscious subsystems within it do not know.
The Chinese Room thought experiment imagines a system with conscious subcomponents (the human following the instructions) and there are analogues in the real world, like markets that “know” things that no individual making up the market knows alone. ChatGPT is more like a fully automated Chinese Room that doesn’t need anyone inside. There is no conscious subsystem, but it is able to act with intelligence. Like a person with blindsight who catches a ball without any part of their conscious mind knowing what they are doing or why until they find themselves holding a ball; or they can walk through a room, weaving past obstacles they don’t consciously see (and with no ability to explain to you why they deviated from the straight path - although they likely won’t know that, and will try to offer an explanation if prompted.
Does the person with blindsight “know” where the obstacles are? On some level, they must, or they’d have bumped into them. But if so, they definitely don’t know that they know where the obstacles are… Even after they successfully avoided them.
No one said that your toe was sentient; in fact, I specifically said it wasn’t. What I said was that the cells in your toe contain all the instructions needed to create a complete copy of you, so you could certainly argue that your toenail “knows” how to build a human, even if it doesn’t know what 2+2 is.
Then why don’t you say, for example, that conventional computer CPUs are really brilliant at math? After all, they are able to do stuff like finding all the prime factors of a large integer, or differentiating complicated multivariable expressions, in mere milliseconds, and produce reliably accurate results. So, computers know math, right?
Well, we generally don’t talk about computers executing conventional deterministic algorithms as “knowing” what they’re doing, even though their outputs are superhumanly fast and accurate. Many people used to perceive them that way, of course, in the breakthrough days of processing power acceleration, because it was so darn impressive that computers could simulate complex processes of logical and quantitative thought the way they did. We now have more widespread familiarity with the concept of how computers execute deterministic algorithms, and it doesn’t look so magic anymore. Even if the processor can do literally every mathematical calculation that a human can perform, we don’t think of the computer as “knowing” how to calculate, in the way that the human does.
Neural networks and the accompanying tools of deep learning in LLMs are far less familiar to most people, and the increase in power and extent of their simulation of human thought functions is incredibly impressive. So once again, a lot of people are focusing on the impressiveness of the outputs, and how closely they mirror what we’d expect from human outputs, rather than on how to understand the underlying processes.
Of course the LLM is going to seem sentient, if you ignore the details of what we know about how its algorithms are designed and implemented to simulate sentience, and how they ultimately depend on the same programmed operation execution that causes a computer to perform complicated math calculations. Almost anything that performs complicated actions can look sentient if you’re refusing to pay attention to how it actually works. “I saw an autonomous vehicle going down the street the other day, and it knew to stop at a red light!” 
In some very limited and useless sense, yes. Computers are essentially enormous collections of organized logic gates. Sometimes those gates are organized to perform specialized functions like high-precision floating-point operations, but those are really just performance optimizations that could just as well be performed in software. Any calculator, by definition, can “do math”. A programmable stored-program computer at a sufficiently large scale with the right software and self-enhancing neural nets can … well, do just about anything! At a sufficiently large scale – which relates directly to my earlier point about emergent properties.
You’re really misrepresenting my point. CPUs at the hardware level – even the supercomputers that power GPT – are pretty useless by themselves. What makes them powerful is the software that can run on them – often layers upon layers of very complex software that produces amazing results as with AI, and specifically with the emergent properties of massively scaled LLMs. At that point the underlying infrastructure is just as irrelevant as a psychologist asking you “how many neurons and synapses do you have in your brain?”. To which the proper answer is “a sufficient number to recognize the irrelevance of that question. Now ask me something meaningful that measures my intelligence!”
Ultimately, an honest human working in the role of a Q&A oracle cares whether they deliver accurate answers to whatever question is put to them. An honest human oracle will assign a certainty factor to their proposed answers. And will hedge their answers in proportion to that uncertainty. And finally say “I don’t know” when that factor is too low.
The current LLMs do not “care”. In scare quotes so I can explicate that word. The don’t have any notion of correctness. Being factually accurate is not part of their objective function or training. Might that change at some point in the future? Sure.
But right now down inside the LLM’s innards it’s a self-referential feedback loop with no damping driving towards truth over “hallucination”.
In a recent thread about artists I commented that a good artist doesn’t need to accurately reproduce the tree in the scene they’re looking at. Instead they just need to produce a plausible tree. The result (and the goal) is art, not a precise photograph.
But imagine the problems that will occur if some architect and construction company start trying to build a real building using the artists’ plausible landscape painting as if it was an accurate depiction of the virgin construction site. Not good.
I see LLMs as similar to artists. Right now their objective is satisfied if they produce a plausible answer, not an accurate one. But a real problem ensues when people start acting on plausible answers rather than accurate answers. A couple weeks ago an LLM told me the population of the USA in 1850 was 340 million. That’s the population in 2020; in 1850 it was about 23 million, ~1/15th as much. The answer was grammatically correct, confidently stated, and superficially plausible. But utterly wrong. To someone who knew little about US population numbers now or in any earlier times that would have been taken as correct without the slightest concern.
And that’s all it was: plausible. That’s all it was “intended” to be. Both by the LLM itself to the limited extent it can be said to have intent, and by the humans who built it and operate it and sell its behavior as a useful product.
Letting these wild Impressionistic artists out to define and redefine what the world really looks like is a major mistake. Because humans in all their laziness and ignorance, will utterly mistreat plausible as if it was accurate.
Such fundamentally defective products should not be offered by honest businesses nor accepted by a discerning marketplace. But they are being both those things. As one early AI put it so eloquently:
If an LLM can answer 2+2 correctly, but not 1234890349 + 9883279200, then it’s not performing anything that can be called arithmetic. Arithmetic can sum integers, period, and if your approach breaks on large integers, then arithmetic isn’t what you’re doing. What it’s returning is synthetic consensus, i.e. what do most texts say is the answer to 2+2? This question has a reliable answer because 2+2 is a very common idiom, but not all expressions are that well known.
That’s what LLMs do. They use a lot of opaque accumulated data to do so, but ultimately all it’s doing is returning a synthesis of the consensus of a very large body of text.
Some of the newer chat AI’s are starting to build “escape hatches” to offload certain queries to functions that are ideal for that purpose – i.e. it’s stupid to ask a computer to utilize an LLM to sum large numbers when there are any number of libraries that could do it deterministically and inexpensively – but they have to be gated to avoid attacks that could be too computationally costly, and it’s also not very practical to have a machine implement every variety of computation that it could possibly be asked.
I asked ChatGPT your math question.
So I guess ChatGPT is performing arithmetic?
Nope. The difference is that we understand in detail how the software is working to generate the answer outputs, whether via deterministic algorithms or LLM deep learning models, and that the operational context of those software processes is different in significant ways from what we call human “knowledge” or “understanding”.
Sure, if you insist on looking only at the outputs themselves rather than the processes, and if your criterion for “sentience” is simply “how well does this output simulate the output of human thought on a particular circumscribed set of tasks”, then yeah, many processes are going to qualify as “sentient”.
And if that’s how you think sentience ought to be defined, fine, no skin off my nose. I just disagree with that choice because I think it’s wilfully ignoring a lot of informative detail about how and why the processes are designed and implemented specifically to simulate the outputs of sentience.
Ok, if we understand in detail how it all works, what is the mechanical process by which neurons in the brain create “knowledge” or “understanding” and how specifically does it differ from the process by which “neurons” in a neural network give rise to everything an LLM can do?
Impossible to say. The broader point I’m making is, if you catch any tool, LLM or not, getting arithmetic questions wrong, then what it’s doing isn’t arithmetic. That doesn’t imply that if it gets a result correct, then it is doing arithmetic.
To repeat what I said before, many earlier AI chat tools couldn’t get this right, so they started to build in capabilities to recognize when it’s being asked an arithmetic question, and then delegate that to a non-LLM function that’s specialized for math. So the LLMs are no longer trying to do arithmetic, they’re delegating out to a boring math library that can do it for them. Just as they’re now actually pulling information from the internet as context, rather than trying to synthesize a complete answer from scratch.
LLMs are synthetic-consensus machines.
@wolfpup makes the excellent point about emergent behavior being a factor in the development of AI.
I’m quite confident that’s what’s going on between my ears, so IMO in principle that can be equally true of enough computational power regardless of whether it’s made of meat or semiconductor.
But …
Life has a bias towards getting things correct. Correctness matters deeply and always has. Animals that confuse pebbles with berries starve. Animals that confuse predators with harmless competitors die violently. Animals that confuse harmless competitors with conspecifics mate but fail to produce offspring. etc. And it’s not only about perception. It’s about all aspects of animal decision making. Migrating east for the winter Does. Not. Work.
For humans with their big conscious cortex riding on top of all that accumulated animal nature, the bias to be correct enough is real deep-seated. Life gives us plenty of motivations to be correct. Correctly assessing whether the stove is hot is a darn useful skill most toddlers pick up the hard way. But they inherently “get” the idea that being correct has utility. They just need to discover what is “correct” in this particular situation.
So the challenge for AI, whether implemented via LLM or some other tech, is to incorporate positive reward for correctness and negative punishment for incorrectness. The eventual emergent result will be a machine motivated to be correct.
But until we humans buying this stuff motivate the IT humans making this stuff to include correctness as a deep-seated unshakeable virtue, the IT humans won’t have that motivation and neither will their products.
Absent the motivation, all the emergence in the world won’t grow correctness. It’ll grow something, that’s for sure.
New revelations on the wonders of LLMs.
Ultimately, the question of how an LLM arrives at an answer is a red herring when we’re talking about therapy.
Like, if I want to bomb a building then the “correct” way to do it is to aim a bomb and lob it. This is correct because 1) I want to do it, and 2) I’m using math and engineering to figure out and match the trajectory.
But I could train a bird to recognize a spot on a map and peck at it, then put the bird in a chamber inside a bomb, so that when he looks down on the world he’ll see the target and peck at it, and then my bomb can detect the pecking and adjust its fall after I drop it in the general region from a plane.
This is the “wrong” way to bomb a building because the bird doesn’t want to destroy the location and doesn’t having any understanding of how to move the bomb to destroy it - let alone using math and engineering to accomplish it.
But… What if the bird method proves to be the most effective means of bombing buildings and despite thousands of years of engineering, lasers, microcontrollers, etc. all we can do is make the bird better, but not make a better system that doesn’t involve a bird?
Then you’d be a fool to not use a bird, if you’ve really got to hit your target.
If an LLM is a cheat hack with no heart and bad reasoning…but saves more lives and marriages than any therapist known to man, then that would just be what it is.