Can you explain what you mean by “taking subjective experience seriously”? I’m thinking that may be the initial difference in our camps from which other disagreements are emergent.
Since every current conscious animal on Earth shares a common evolutionary ancestor, and we don’t even understand the subjective experience of non-human creatures (what does it feel like to be a bat?), then predicting the qualia experience of an artificial intelligence is near impossible. It will probably have a will to survive. Beyond that is anyone’s guess.
There’s a problem here. In testing the results of neural net training, it is important to not test the resulting weights with anything from the training set. Testing with the training set is just a table look up.
From your link:
"This experiment was not intended to see whether training GPT-3 on Dennett’s writing would produce some sentient machine philosopher; it was also not a Turing test, Schwitzgebel said.
Instead, the Dennett quiz revealed how, as natural language processing systems become more sophisticated and common, we’ll need to grapple with the implications of how easy it can be to be deceived by them."
GPT can express nuances about information within it’s training set. It can generalize the sequence and rhythm of words in poems. But it can’t generalize textual content. The real test would be asking questions that relate to the views of an iconoclast that was not in it’s training set.
Ask it to critique Jeffers poem Ink-Sac. The poem is about political propaganda.
If that’s supposed to relate in some part to my (rather futile, it would seem) attempts at clarifying the notion of emergence to you, it’s a huge distortion. Basically, emergence isn’t a magic wand: faced with a system showing no indication of a quality, proposing that it might emerge once you pile up enough of it is vacuous. Sure, maybe it does. Maybe it spontaneously acquires consciousness. Maybe it grows wings and flies away. I mean, who knows, right? 
Emergence!
But that’s not how it works. First of all, there clearly is emergence in the sense that large-scale phenomena show qualities not apparent from their constituents. Water is wet, where water molecules aren’t; the flocking of birds is not apparent from single individual behavior. But it’s always the case that ultimately, fixing the facts at the microscopic level fixes the facts at the macroscopic level. Everything else—sometimes called “strong” emergence—is basically magic, or at least an expression of dualism.
This also means that the microscopic constituents put bounds on the sort of phenomena that possibly could emerge. For instance, you might say that because of emergence , piling on more parameters to a large language model might enable it to produce genuinely random numbers. I mean, it might, right? Who knows what could happen of these things get large enough! You don’t know it couldn’t happen, because there’s no way to survey the whole system in all its complexity!
Except, of course I do know. The basis of the system simply doesn’t support the emergence of genuine random number production. No matter how many parameters you pile on, it’s not gonna happen. Likewise, no construction out of Lego pieces, no matter how complex, is ever going to spontaneously emerge the ability to invert gravity and levitate. The building blocks just don’t support the emergence of such phenomena.
So, what we know about the base can be used to put boundaries on what could emerge. If you’re saying that anything at all could emerge, you’re essentially appealing to magic; and when you’re doing that, then basically you’re just not making a claim with any evaluable content, because then, anything goes anyway.
Basically, just accepting that it exists, rather than getting into the eliminativist tangle of holding that it just ‘seems’ to exist, which doesn’t really net you anything, but saddles you with having to explain why it seems the way it does, and on top of that how it can seem any way at all to us if there’s no such thing as subjective experience.
This wasn’t directed specifically at you, though I do recall having had those discussions. One name that pops immediately to mind in connection with emergence is David Chalmers, who has a sort of paper (it looks more like an unpublished set of ruminations) in which he stresses the importance of the fundamental difference between weak and strong emergence. It can, in fact, be argued that this is not actually a meaningful distinction at all, as I note below.
This seems to be the core of the argument – the rest of your post just giving examples – so let me address this. There is clearly a sense in which this is obviously true, but what does it actually mean? Does it mean that all emergence is weak emergence, and the properties of a system can always be inferred from the properties of its components?
The theoretical physicist Philip Anderson addressed this question in a thoughtful paper published in Science in 1972. He demolishes the distinction between weak and strong emergence using the hierarchy of sciences as an example. We cannot possibly develop an understanding of biological systems merely from an understanding of chemistry, nor can we develop an understanding of human psychology from an understanding of biology. The distinction between weak emergence creating phenomena that are merely “unexpected” versus strong emergence creating phenomena that are fundamentally unknowable, as proposed by Chalmers, becomes a moot point. Or at least, a metaphysical point that makes no useful predictions about the world.
A key extract from Anderson’s paper, with bolding added by me:
The ability to reduce everything to simple fundamental laws does not imply the ability to
start from those laws and reconstruct the universe … The constructionist hypothesis breaks down when confronted with the twin difficulties of scale and complexity. The behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of a simple extrapolation of the properties of a few particles. Instead, at each level of complexity entirely new properties appear, and the understanding of the new behaviors requires research which I think is as fundamental in its nature as any other. That is, it seems to me that one may array the sciences roughly linearly in a hierarchy, according to the idea: The elementary entities of science X obey the laws of science Y.… But this hierarchy does not imply that science X is “just applied Y.” At each stage entirely new laws, concepts, and generalizations are necessary, requiring inspiration and creativity to just as great a degree as in the previous one. Psychology is not applied biology, nor is biology applied chemistry.
He ends with the following observation:
In closing, I offer two examples from economics of what I hope to have said. Marx said that quantitative differences become qualitative ones, but a dialogue in Paris in the 1920’s sums it up even more clearly:
FITZGERALD: The rich are different from us.
HEMINGWAY: Yes, they have more money.
I believe a subjective experience exists (at least, it does for me! The rest of you could be philosophical zombies for all that I know). But it also pretty clearly appears to be an emergent property of physical, chemical, and electrical interactions between the components of your brain. We know that changes to the physical structure of the brain can impact subjective experience, sometimes dramatically so. We know that our subjective experience telling us why we took certain actions can be factually incorrect, as in the famous studies of split brain patients.
I don’t think I take the idea of a subjective experience any less seriously then you do, but I also think that our subjective experience is an emergent property of complex processes in the brain, and see no reason to believe that 
meat is the only substrate on which these processes could ever exist.
Ok, this probably isn’t the place to go through this again. Suffice it to say that I’m in complete agreement with the bolded part: everything reduces to the fundamental laws. (Whether it is possible to derive everything from them is another matter entirely—indeed, the undecidability results I’ve been discussing with @Dr.Strangelove imply that it isn’t, but that’s of no concern here.) Strong emergence entails that the high-level phenomena are not metaphysically necessitated by the low level; they certainly are in the cases Anderson discusses (well, up to questions of determinism I suppose).
Great, I also think all that. Indeed, I’ve even gone and formulated a theory of how it does so.
I would state it as, "Everything must be compatible with fundamental laws. I would also add that it has to somehow be in the domain of the factors causing emergence. We wouldn’t expect an LLM to emerge a dolphin mind, because we fed it no infirmation that dolphins used to learn what they do. We wouldn’t expect a bunch of ants in a colony to write a book, but but we were surprised to discover that they maintain precise temperatures in the hive through precise application of fermenting plant matter, there was nothing in the domain of ‘ant-ness’ that would preclude that.
But noting that ants do this is a lot different than predicting they would do that by studying the behaviour of individual ants outside of an anthill situation. You can’t reduce the behaviour that way. Individual ants, or ants in small groups, behave pretty much randomly and erratically. Keep adding ants, and at some ant density suddently they start building bridges, coordinating food transport, fighting wars, taking prisoners, staging coups, and all kinds of things we would consider ‘intelligent’. But you can’t see any of that when you drill down to understand it. It’s not reducable. At the lowest level, ants are just state machines. They behave through simple rules. The complexity comes through emergence when thousands of them are put together. And there’s nothing in those simple rules that will tell you what will emerge, or at what scale.
Take love, for example. An emergent emotion created by a whole lot of activity in a complex brain. I think we are all in agreement that ‘love’ is a manifestation of a whole lot of phusical processes governed by fundamental laws. But you will utterly fail in trying to use reductionism to break down ‘love’ into smaller and smaller pieces down to its constituent neurons or whatever. Comolex systems are like that. The combination of non-linear responses and high sensitivity to initial conditions make them very opaque to scientific reductionism.
The analogy I like to use is a comparison of a watch with a puppy. A watch is complicated. In fact, their movements are literally called ‘complications’. A puppy is complex.
A complicated thing can be understood by reducing the complication and studying how it all goes together. Complex things are different. Try to ‘break down’ what makes a puppy a puppy, and you just get more complexity. Puppies have a brain. A brain is a complex system. If you drill down into the brain, you find more complex systems. Keep drilling and you find all kinds of complexity, right down to protein folding. And along the way, you completely lost whatever it is that makes a puppy a puppy.
A lot of what is going wrong in the world today stems from people not taking complexity seriously and pretending they can understand, plan, and change systems that are not amenable to understanding and planning. Scientific reductionism applied to complex systems doesn’t get you very far.
I think it would be productive to take a step back for a moment and clarify just what the question is that we’re trying to answer. This thread is about AI and ChatGPT, and in this context the questions about emergence center around questions of what novel qualities might yet emerge in these systems. More specifically, can a future AI – not necessarily using the GPT model, but in general, say, some advanced artificial neural net implemented on a digital computer – eventually exhibit emergent qualities like human or superhuman intelligence, understanding, and even consciousness?
From that perspective, obstacles to constructionist derivation of emergent properties from an examination of lower level components – obstacles like undecidability, non-determinism, non-linear dynamics, or what Anderson has called “the twin difficulties of scale and complexity”, are crucially important and can’t be dismissed as being of no concern here. Conversely, the premise that high-level phenomena must be metaphysically necessitated by the low level can be granted while still being able to answer a question like “can an AI develop human or superhuman intelligence, understanding, and even consciousness as an unpredictable emergent quality?” in the affirmative. That’s really the point I wanted to make.
I guess that sort of depends on what you mean by ‘compatible’. In a sense, every macroscopic behavior is compatible with the fundamental laws—those after all only concern the behavior of the fundamental entities, not that of, say, chairs. So it seems there ought to be some sort of metaphysical, if not logical, entailment from lower to higher levels that’s absent in the other direction.
But that’s in large part what I’ve been saying—we wouldn’t expect an LLM to emerge concepts, because we haven’t fed it any of the information that goes into concept formation.
Ok, I’ll bite. Why? That there’s things we can’t predict about the high level behavior seems rather a triviality, it’s true even of three interacting bodies. But that doesn’t mean we can’t find constraints on what could possibly happen. They won’t spontaneously dance the hokey pokey and time travel back to the year 1931, for instance. That there are certain aspects of a system’s phenomenology that are difficult or impossible to predict doesn’t mean that anything goes.
Indeed, finding out what doesn’t go may not even be hard—the converse of a difficult problem can sometimes be quite easy. Suppose we have a large number whose prime factors we can’t feasibly calculate; still it wouldn’t be the case that it could equally well be any, as for any proposed factorization, we can quickly check whether it works. So we can’t say which primes factor the number, but it’s easy to list which don’t.
The answer to “why” is that the class of potential emergent behaviours that we can rule out on principle is very specific and limited. Take, for instance, the random number sequence you keep bringing up. If we assume that our computational components are necessarily deterministic, and putting aside arguments about the difference between “random” and “pseudo-random”, then you have a point. But the point rests on the truism that computational systems defined as being deterministic cannot exhibit randomness, by that very definition. There aren’t many other computationally driven behaviours that are subject to such easy constraints and such easy dismissal. It tells us nothing about the possibility of interesting emergent phenomena like consciousness.
So, no discussion of the claim making the rounds that this thing has now taken and passed a bunch of AP tests (including AP calculus)? That claim, at least, is definitely false, because there hasn’t been an AP testing window since it was developed. Someone might, of course, have tried it on AP tests from previous years, but that’d be a completely meaningless test, because those previous tests were part of its training data.
Yes, we did. We fed a lot of human text in many languagues, and concept formation might be learned from that. We never told it how to do general addition, either. That emerged from the training data. I’m not sure why concept formation wouldn’t.
Let’s take a step back. The original claim was that all these models do is a form of statistical next-word lookup. The sentence may be grammatically correct, but the AI has no understanding of it. It’s just ‘adding’ or some other algorithm that just spits out words in order.
In my opinion, that can’t possibly be complete. For example, let’s consider word-in-context, an ability that emerged between 10^23 and 10^24 FLOPs. What is word in context? It means understanding a word based on the context around it when a word can have multiple meanings. For example:
“I went to the bank.” What kind of bank? A river bank? A financial institution? A choice to bank a ball on a pool table? How do we know which is which? Well, we understand the properties of all those things, so based on the context we can tell.
“I went to the bank and fell in.”
“I went to the bank, but it was closed.”
“I couldn’t make a straight-in shot, so I went to the bank.”
“You can bank on it.”
For GPT to give a coherent answer or continuation of those sentences, it needs the concept of ‘bank-ness’. It’s not enough to simply have a token for ‘bank’ along with vectors to its closest words
I pasted those four sentences into Bing Chat and asked for a definition of each ‘bank’:
I tried to think of something more subtle, so I asked it to identify what ‘bank’ referred to in the case of, “I’m going to run around the bank” vs “I’m going to run along the bank.”
To get that, do you think Bing Chat is just doing next work prediction? Did it know that you can’t run ‘around’ a river, but you can run around a financial institution? I asked Bing Chat:
Q: Can you solve word-in-context using stochastic next word prediction?
No, it’s not passing because the tests are in its training data. See this:
GPT-4’s training data was cut off in Sept 2021. So Caplan’s test would not be in its training data.
Here’s one of my favorite answers, since even PH.D economists often get this wrong:
Note that the words ‘comparative advantage’ are not spoken, and the concepts are not necessarily clear. Here’s what GPT-4 said:
It’s hard to imagine that answer being generated by ‘next word lookup’ from the terse question that left out a lot of nexessary concept and expected the reader to make the connections.
Here’s a link to a tweet listing all the exams GPT-4 has now passed:
I have no idea how it’s passing that dude’s Labor Economics tests. I make no claims about that. But if it’s taking an AP test right now, it’s taking a test that was available in the training data. Is it definitely just remembering the answers from when it saw those exact questions before? We can’t know (at least, not yet). But that’s a lot simpler hypothesis than assuming that it went from struggling with 2nd-grade math to acing calculus in a single revision.
Nor is that the only open question, for this claim. Who scored this test? There’s a lot of work that goes into grading AP tests. How was the information formatted, and presented to the AI? It’s nontrivial to present any calculus problem in plain text-- You can do things like LaTeX, but was that how it was presented? A lot of the material out there will just have the equations as an image-- Has its image-processing capabilities also advanced, to the point that it can “read” in a very seldom-used “language”?
That seems like a completely different point, though? And also one that’s pretty much opposite to the truth: the class of potential emergent behaviors that can be ruled out vastly exceeds that which might emerge, in the same way that the real numbers exceed the natural numbers—it’s a null set within the latter. Thus, if you put all potential behaviors into a hat, the likelihood that you draw one that can’t be ruled out is exactly zero.
I’ve been talking about randomness generation as a kind of prime exemplar of this sort of task, but I’ve also mentioned several others, where it’s anything but trivial to demonstrate that they can’t emerge, but where nevertheless a conclusive proof exists. Take the question of whether two 15 x 15 matrices can ever be multiplied together so as to yield the zero matrix: no LLM will ever emerge the ability to answer correctly in every case. Or take the question if, given the source code of a function, that function calculates the digits in the decimal expansion of pi: again, no LLM will ever emerge the ability to solve it exactly.
It’s far from obvious that these questions aren’t within the range of behaviors that an LLM can show. Indeed, if I didn’t know about them, I would hardly have batted an eye on encountering the claim that LLMs (or any sort of AI agent) can learn them. But they can’t.
Likewise, they can’t solve the general problem of inference—that is, finding a provably optimally parsimonious hypothesis to predict future data. This is at the heart of the AIXI agent, which I think is the best theoretical model of a truly general intelligence we have. And on my own model, the problem of finding a ‘safe’ self-modification to e.g. better adapt to the environment also turns out to be in that class. So here there’s two concrete behaviors that seem instrumental in bringing about a generally intelligent or even conscious agent which LLMs provably can’t emerge.
Because addition is an operation on the properties of symbols (the syntactic level), whereas concepts form the semantic level, of which we simply haven’t told the LLMs anything. What they do know, again, is just the following: for any word,
These are not data sensitive to the concepts the words refer to. They can be harvested and manipulated in complete absence of any understanding.
But it’s what we know to be the case—it doesn’t generate the next token based on a Markov chain model, true, but it absolutely does just generate the next token again and again, and absolutely all it uses to do so are the above mentioned three pieces of knowledge.
That doesn’t mean that it can’t learn structures present in that data—such as, for instance, addition. But what concepts words refer to isn’t present in that data—it’s that fact that makes language useful at all: that words can point beyond language into the world. If language were a closed system, it would just be self-referential, without any point of contact with, well, anything. Words would only tell us about words.
There simply is no credible story of how to get from the information which LLMs have access to, to the concepts words represent, because the representational role of language isn’t available to them any more than the particulars of dolphin minds. There’s just no ‘there’ there.
Of course, yes, that’s all it can do. That’s all it’s supposed to do. Yes, it is amazing, and in my opinion, a major insight, that this works so well. But we must be careful not to be taken in by the spectacle, otherwise, we’re making the same mistake as those who see intent in the design of biological entities do—argue from mere incredulity that because we can’t imagine this to be the product of ‘blind’ processes, it must not be. But what Darwin showed for biology, we’re now being shown for language production: that the appearance of intent does not suffice to conclude the presence of intent. There are mechanisms to produce the former in the absence of the latter—evolution in the biological case, whatever we should call what LLMs and their ilk do in the case of language.
If an algorithm is impossible to physically realize, then it may as well not exist.
This is a point on which I feel there is something of a difference between the scientific and philosophical mindset. Both use thought experiments all the time. But scientists reject thought experiments that violate the laws of physics, or at least use them to explore what may or may not be possible. Philosophers seem to take no issue with proposing Chinese Rooms that simply could not exist, because they would collapse into a black hole.
You need a computer program to determine if a sequence is random or not. If P!=NP, there exist one-way functions, and I can use that to generate a pseudorandom sequence that is arbitrarily hard to reverse-engineer, even with a computer the size of the universe. You will detect no correlation with anything else unless you discover the private data, which you can’t do in less than 2^N operations.
You say that there is still a non-zero carrying capacity. But there is a non-zero probability of lots of things happening, like the bits on the far end of a communication device spontaneously configuring themselves into a given message before it could actually be sent. There’s no need to worry about events with an infinitesimal chance of happening.
In all likelihood, properties like “causality” and “distance” are only emergent, macroscopic phenomena anyway. It wouldn’t bother me if they were violated some negligible amount of time due to random chance.
We’re also largely talking about capabilities that we already know are possible, like being able to multiply two numbers. These aren’t questions at the edge of algorithmic information theory. And mostly “emergence” has been taken to mean the process of generalization from memorization into a general-purpose algorithm. Or, put another way, at what point is it able to successfully extrapolate outside its training set?
It’s remarkable that this happens at all, even for simple cases.
No. This is just a matter of logic: to show that a thesis doesn’t hold, it suffices to show that a counterexample exists.
That’s not what a ‘nonzero capacity’ means. That the capacity doesn’t go to zero in the infinite limit implies, by Shannon’s noisy channel theorem, that we can use the channel to transmit information with an arbitrarily suppressed probability of errors.
Perhaps. But lobbying for the laws of physics to be revised just because you really want the world to be computable isn’t a sound strategy. The world is the way it is, and we’ll have to taylor our opinions to that, not the other way around.
So in the end, without appealing to hypothetical wholesale revisions of the laws of physics, what remains, as Yurtsever puts it in the paper already referenced, is:
The result presented in this paper shows that if violations of local causality are to be ruled out […] it is not possible to simulate quantum mechanics on a digital computer (i.e., a Turing machine); quantum randomness is “uncomputable” in this sense. This fundamental lack of computability of quantum phenomena may have certain far-reaching implications; for example, if quantum-mechanical processes play a significant role in the activities of biological neural systems, then brain activity cannot be simulated faithfully on a digital computer, no matter how elaborate the simulation.