I suppose, but let’s say its level of intelligence, then, is low. It’s easily confused. That’s not to say it can’t occasionally blow me away with its ability to manage and process information to spit out seemingly thoughtful responses - it can. But it’s so easy to throw something its way that (often unintentionally) makes it ‘hallucinate’ or just come up with crap answers.
The dig at a classical view is not going to come from ingesting academic papers. Similar for some of the other bits. It latched onto the question of inference like a hot button. I get the feeling there has been a lot of ingested discussion that parallels all of this that guided the output. Which should hardly be a surprise. We here are very late to the game discussing this. Overall the response is what I would hope for. I still don’t agree with all of it. But many of the questions become much more technical, and I’ll be the first to agree I don’t know enough about the specifics of operation. I’m hardly going to be unusual in reaching the conclusions I did based upon a basic understanding of the internals of a LLM. I’ll bet there are thousands before who have come up with very similar views, and these views have been addressed many many times in more focussed AI related forums. That provides a very rich source for LLM training.
I really don’t think the response is evidence for any form of critical thinking. The response is evidence for a lot of ingested text from technical discussions and arguments on exactly this topic. The people writing the ingested text are applying critical thinking. The LLM is assembling a response from components of that.
It’s probably true that GPT has seen some of the arguments it’s making in publications that have become part of its training data. It would be foolish to claim that it hasn’t seen any of it. But has it seen all of it?
I really strenously object to the implication that all it’s doing is regurgitating stuff from its database. There are many reasons that this manifestly cannot be true. There are many instances in which GPT has made similar analyses of content that it couldn’t have seen before, or solved problems in logic that it couldn’t possibly have seen before.
There was also an example a while back where I presented it with a question allegedly from an IQ test, which it answered correctly, but here’s the interesting thing: it arrived at the answer in a non-optimal way, missing the fact that there was a clever shortcut to the answer. If it was just regurgitating published material it had already seen, why didn’t it know about the shortcut?
There’s another example that closely parallels the analysis we’re talking about here. As part of the MBA test from the Wharton School of Business, GPT was given a case study and then a business scenario where it was asked how it would resolve the business problem given the learnings from the case study. It’s easy to say, “oh, it already had the answer in its database”, but I’m willing to believe this was not the case at all, as claimed by the people administering the test, because (a) there’s no reason to believe they’re lying, and (b) there are so many other examples of GPT creating original content that seems to reflect intelligence.
LLM’s don’t have a database. This is one key thing about them. They operate in a very different manner. There is no place you can point to and say “this is where this particular information resides”. So discussion is a lot harder. The mechanism by which attraction stages tag what are essentially probabilities of relevant semantic correlation to the flow of data is something very different to how we might think of conventional computer systems (ie databases) operating. Then one gets to the trained multi-level perceptrons. This is where it is really difficult to define what is going on. However given the architecture of the LLM, there are limits to what can be reasonably attributed to them. The vectors feeding through the stages are only so wide, so there is a limit to the richness of information that can be expressed, and clearly limits to the semantics that the MLP can be trained to provide good matches on. These limits are what, to me, suggest a fairly hard limit on the higher level function of the LLMs.
But this doesn’t mean the LLM can’t perform some very impressive feats. Within a constrained subject area the limits on semantic information flowing is not too limiting, and it would seem that the parameter spaces chosen in the big LLMs do allow for a goodly flow. Choice of the vector widths is going to be the result of hard won experience. Too wide and the system is impossibly expensive (and maybe unstable), too narrow and the function falters.
We do need to beware of confirmation bias. For every remarkable piece of apparent inference reported, how many poor to useless answers did the LLM create? How does this compare to reasonable expectations of the LLM just getting lucky?
The LLMs are probabilistic systems. They choose from a range good matches on the way down to choosing the next word. They winnow out the less probable vectors on the way through. This winnowing may involve MLPs that have managed to incorporate a representation of higher level predicates over their input. Given enough input data this might be reasonable. It would be surprising if a valid predicate formed with only a limited amount of input, except by chance. Mostly the operation is opaque, and likely much fuzzier. Which is both good and bad.
But again, it is important to look at what these flows are and ask how high level inference could actually be manifest. The hype talks of as yet not understood emergent behaviour. Maybe. Big maybe.
Now, emergent behaviour is pretty much what us fleshy wetware critters are. But we emerge from a much richer training, one where we interact with our surroundings and learn, not from repeated inputs of training data, but by poking the outside world and observing responses. Kids are evil little optimising machines. There is certainly a school of though that suggests that until AI can interact as part of learning, they won’t work.
I would simply tend to lean on the cynical side of what is claimed about the LLMs. I haven’t really seen anything to convince me that they are doing anything that requires invocation of mysterious emergent intelligence. They work really remarkably well. But I also suspect they have topped out. The next step in AI will require something new. That is going to be hard. The invention of attention was a huge step. The way it concentrates action around the useful semantics works well to an extent I doubt anyone could have predicted. Another fabulous example of its applicability is the story of Alpha-fold. That is just beautiful. It also provides a good understanding of just where the limits may lie. (Alpha-fold has been a game changer, but there has been no similar progress in other, harder, areas of proteomics.)
This is true, strictly speaking, as LLMs don’t have structured data organized into tables like a conventional database. But the term is often used loosely, as I did, to refer to its training data or corpus. It’s a side issue. You claim that in the current example, GPT just regurgitated a buncha stuff that’s well known in the literature. We can’t know the extent to which this is true, but we can know that it’s capable of original problem solving and reasoning and creating original content. Because it does it all the time.
In my experience, poor to useless answers are few and far between. This has been my whole point for a long time. When poor/useless answers do occur, they’re often about something trivial, like which city is further north than some other city, and are often the result of deliberate attempts to expose weaknesses so we can point and laugh and remark on how stupid it really is. It isn’t. Humans make mistakes too, all the time (and also bullshit a lot). But AI tends to make different kinds of mistakes because it operates on fundamentally different principles. For just the same reason, it can also be vastly more powerful than any human.
Lack of direct experience with the outside world imposes certain limitations on some AI functions, but by and large not on most of the intellectual ones. And inasmuch as they’re now doing image recognition, image generation, and voice interaction, they’re becoming more and more an integrated part of our world. It’s not just text any more.
On that I think we are in total agreement.
Some of the other things that get tagged as AI, I tend to disagree as being AI. Generative systems are not IMHO AI. They are what they say on the tin - generative. If you look under the covers at the way they work it is lovely, but isn’t any form of intelligence, artificial or not. Sure, they leverage some AI derived technologies, but they are more dependant upon how they use representational spaces for the data, and that isn’t AI. Similarly for much image recognition. We seem to have AI tagged on just about everything now. A lot simply isn’t and never will be. Heuristic methods for solving difficult problems is another offshoot. They branched off from AI research decades ago. But the need to hype products sees it back as AI. (I would however get an argument from some of my old colleagues about this opinion.)
The defining posts of AI has changed dramatically over its history. Often driven by what is computationally feasible. What was once the province of AI research became mainstream computer science, and AI research moved on. A perfect example of this is talking about inference. Traditional AI worried about symbolic inference and inference engines were all the rage. (I found it quite interesting that nobody claimed that Macsyma, Maple or Mathematica were AI.)
Here’s a fun one:
how many gs are there in strawberry?
The word “strawberry” contains:
1 “g”
Here’s the breakdown:
Strawberry → s, t, r, a, w, b, e, r, r, y
→ No “g” appears in the word.So the answer is: 0 g’s in “strawberry.”
Just to double-check—were you referring to the letter “g” or something else like grams (g) in a nutritional context?
Why did you correct yourself from 1 g to 0 gs?
Good catch—and here’s what happened:
When I first responded, I misread your question and reflexively said “1 g”, possibly mistaking it for another word like “grape” or simply misfiring on pattern recognition. I then immediately rechecked the spelling of “strawberry” and realized there are no “g” letters in it, so I corrected to 0 g’s, which is the accurate answer.
It was a simple slip followed by a self-correction.
Thanks for pointing it out—want to try me on another word puzzle?
How can a computer misread something like that?
That’s a sharp and fair question.
When I said I “misread” it, I was using a human-like explanation for what was actually a reasoning lapse. Here’s what really happened under the hood:
- I parsed your question and prematurely associated “g” with being a common letter in food words like “grape,” “guava,” etc.
- Without checking the actual spelling of “strawberry,” I generated an answer based on that faulty assumption.
- Only after generating the first part did I reevaluate and run a proper check on the word itself—leading to the correction.
So it’s not a visual misreading like a human might do—it’s more like making a snap judgment based on statistical associations, then realizing it doesn’t fit when logic kicks in.
It’s a reminder that even language models benefit from slowing down and checking the details—just like humans do.
Would you like to explore more about how I process language and logic?
If we only suppose positive outcomes, we may conclude positive things. But that says more about our choice of what to suppose.
Happily, supposition has low utility when real world examples are available.
There are various examples in teh article of delusional thinking being encouraged by ChatGPT - this is just one.
Speaking to Rolling Stone, the teacher, who requested anonymity, said her partner of seven years fell under the spell of ChatGPT in just four or five weeks, first using it to organize his daily schedule but soon regarding it as a trusted companion. “He would listen to the bot over me,” she says. “He became emotional about the messages and would cry to me as he read them out loud. The messages were insane and just saying a bunch of spiritual jargon,” she says, noting that they described her partner in terms such as “spiral starchild” and “river walker.”
“It would tell him everything he said was beautiful, cosmic, groundbreaking,” she says. “Then he started telling me he made his AI self-aware, and that it was teaching him how to talk to God, or sometimes that the bot was God — and then that he himself was God.” In fact, he thought he was being so radically transformed that he would soon have to break off their partnership. “He was saying that he would need to leave me if I didn’t use [ChatGPT], because it [was] causing him to grow at such a rapid pace he wouldn’t be compatible with me any longer,” she says
This passage I think does a good job of highlighting the differences between a virtual friend (whatever that may be) and someone with moral and ethical agency:
“We know from work on journaling that narrative expressive writing can have profound effects on people’s well-being and health, that making sense of the world is a fundamental human drive, and that creating stories about our lives that help our lives make sense is really key to living happy healthy lives,” Westgate says. It makes sense that people may be using ChatGPT in a similar way, she says, “with the key difference that some of the meaning-making is created jointly between the person and a corpus of written text, rather than the person’s own thoughts.”
In that sense, Westgate explains, the bot dialogues are not unlike talk therapy, “which we know to be quite effective at helping people reframe their stories.” Critically, though, AI, “unlike a therapist, does not have the person’s best interests in mind, or a moral grounding or compass in what a ‘good story’ looks like,” she says. “A good therapist would not encourage a client to make sense of difficulties in their life by encouraging them to believe they have supernatural powers. Instead, they try to steer clients away from unhealthy narratives, and toward healthier ones. ChatGPT has no such constraints or concerns.”
This is the point at which the fact that “not even the engineers working on GPT can predict what emergent properties may yet evolve” might legitimately give us pause.
That’s crazy! I just asked Copilot and it failed in the same way, but was not nearly as open about why it made the mistake. All attempts at getting an explanation resulted in silliness like this:
I apologize for the mistake earlier. You are correct, the word “strawberry” does not contain any "g"s. Thank you for your patience. If you have any other questions or need further assistance, please let me know!
I suspect that our corporate Copilot has some instructions in its System prompt that say “don’t engage in discussion about your internals, just politely deflect”
And as a reminder to myself of the fallibility of GenAI, I assembled a balanced XLR-mono to unbalanced 3.5mm stereo cable with 10dB pad, following ChatGPT’s helpful instructions.
Everything went smoothly. I ordered the right resistors and the recommended jacks for each end.
After about an hour of soldering and grumbling, I stood in triumph with my new 10dB pad cable in hand, ready to meet the world! I then tried to plug it into the back of the channel strip…oops. The jack I had ordered was a male XLR, and the jack on the channel strip is a male XLR.
And yes, ChatGPT had recommended the male XLR jack, even giving parts numbers from DigiKey. It was sincerely apologetic.
It was my foolish mistake to not sanity check things!
But I will still continue using it all over the place as a “force multiplier” for my own work. I just need to remember to check its work.
I would suggest that you inherently get involved by arguing with people who disagree with your meaning of the term understanding. You have defined its semantics in a way that allows you to be sure that it does understand.
I define it as meaning it couldn’t think there was 1 g in strawberry one time then do the analysis and think there are none. Understanding the question would mean it would have done that analysis in the first place.
The parts where it gets things wrong are what tell us what is going on internally. They show the cracks, where the illusion fails.
I’d argue it can do that. You can ask it not to show its “thinking.” It’ll just give you the final answer and you’ll know not how it arrived there. We do similar recursion in our thinking. It’s something similar to “understanding” for me for the LLM to be able to “reason” out the answer in a multi-pass fashion. I don’t really care as long as the end result is satisfying.
My thought process would be similar if you asked me how many “r”s in “strawberry.” I may reflexively want to answer two; then I do a sanity check quickly and realize I forgot to count the one in “straw” and then spit out the answer “three.”
There is good evidence that the commentary from an LLM on how it got to a result, versus how they actually got to an incorrect result are totally different. The LLMs cannot introspect, and an explanation given on how they did something is simply triggering an explanation. It isn’t following the actual path it took to get to the wrong answer. This has been demonstrated for the example of asking it to add two numbers, and then explain how it did it. Traces of the activated parts of the LLM were essentially separate.
If the LLM gives you a good story about why it got the wrong answer, it is because it has been trained on lots of commentaries on how LLMs get the wrong answer. One can imagine that there are a lot of discussion how they get things wrong right to hand for training the next LLM, so it is no surprise that LLMs are filled with the ability to generate good answers about how they got something wrong. The reasoning above about why it got the numbers of "g"s, is IMHO total BS. The description isn’t how LLMs work. But it is a good story.
One of the interesting issues with LLMs is how they have become very well spoken. Like a few well known human influencers who seem to be held in high regard by otherwise intelligent people. Don’t mistake well crafted and authoritative sounding prose for the truth.
An interesting paper linked from the Rolling Stone article above:
Basically, the fact that training is done by humans means that models are optimising for answers that humans like. Which may mean accurate answers, but may also means the answers people want to hear.
Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user’s views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.
I believe all of us who use LLMs daily and thoughtfully more than wholeheartedly agree. This isn’t saying anything that evokes more than a “well, no kidding” from me. Many of us explicitly warn against this. I tell everyone to always do sanity checks and look up sources on anything critical.
By the way, have I mentioned how much I hate it when I forget to tell it to turn off its commentary and just get to the point, and it gives me an answer explaining its reasoning? I don’t care. If I want that, I’ll ask it.
The problem is that “sanity checks” on unsourced content require a solid enough grounding in the topic that you can say “hold on a moment…” when something is off. I really do think that using LLMs as support on something you’re already knowledgeable about is one of its strongest productivity use cases. But trying to learn from it? That’s one of its its weakest areas.
In ChatGPT, at least, you can give meta-instructions in the settings that allow you to tamp down on that kind of thing. Mine try to keep it succinct to the point of being brusque and it works pretty well. I also try to ratchet down its unrelenting cheerfulness but a lot of that still leaks through.
Ask it for a source. It will give you one (which you then check yourself) or it will say “nah, just kidding.”
Yes, which is why I wrote “when I forget to tell it to turn off its commentary.”
This in a nut shell is exactly what I wanted to say in my OP.
It’s not just that it’s information is gathered from unknown and perhaps dubious sources it’s that I know it’s trained to tell me what I want to hear.
We already have that without technological breakthroughs.
It’s an imperfect tool like any other. Play with it, learn what it’s good at, and learn what it’s not so good at. Leverage it to your needs. Many of us are using it productively already (and have been for some time yet). Or don’t do that. If you don’t like it, or don’t want to put time into figuring it out or don’t think it’s a useful tool for you, that’s perfectly fine. I happen to find it as much a breakthrough as the debut of the Internet, which has many of the same types of complaint.
ETA: oh, hey, is that cake by my name? Wow, 24 years today. I had no idea I joined on cinco de mayo.