It’s even worse than that. To pass a Turing test, an ‘artificial intelligence’ system just has to function in a way that convinces the user that it is actually demonstrating sapience. Of course, someone who is predisposed to look for indications of sapience will find evidence for a ‘spark of consciousness’ and ignore contraindications that the system really isn’t sapient and is just producing word-streams per a complex algorithm with no ongoing cognitive processes. This is especially true with an LLM because as humans our main mode of interaction is through language, so the default assumption is that a tool that can competently manipulate language must be capable of extensive cognitive functions even if it makes basic errors in simple reasoning, fails to distinguish between fact and fiction, and generates fake references for the ‘information’ that it ‘hallucinates’.
Nobody believes that non-LLM deep learning systems are on some edge of sapience or sentience but a thundering herd of people have convinced themselves that LLM-based chatbots are or are approaching AGI despite a complete lack of any ongoing processes that a neuroscientist would recognize as cognition, or any ability to interpret the world or develop ‘lived experience’ of it beyond its training data set and reinforcement learning, which requires massive amounts of carefully curated data that are many orders of magnitude more than a person would read in a lifetime, and still needs extensive correction and constraints to keep from generating pure nonsense or spiraling completely out if control. That LLMs can manipulate natural language in a context-sensitive way is an impressive feat of deep learning computation to be sure, but it is not indicative of more extensive intelligence or knowledge models; rather, it is an empirical confirmation of the speculations of computational linguists of the extensive metasemantic content embedded in the structure of how language is used.
Stranger