# Bogus info from ChatGPT

**URL:** <https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465>\
**Category:** In My Humble Opinion\
**Created:** [April 21, 2026, 10:06pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465 "2026-04-21T22:06:03Z")\
**Posts on this page:** 20\
**Page:** 5

<div class="post-metadata">

**Author:** ![Reply](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/reply/32/15952_2.png) [@Reply](https://boards.straightdope.com/u/Reply)\
**Post date:** [April 23, 2026, 7:15pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/81 "2026-04-23T19:15:08Z")

</div>

I wonder if there’s some country or [principality](https://en.wikipedia.org/wiki/Principality_of_Sealand) somewhere that might allow a LLM to raise a human child absent the influences of other humans. It might be possible to train a human mind on more abstract, symbolic connections of meaning (than the extant human languages in use)… I’m imagining something like [Nell](https://en.wikipedia.org/wiki/Nell_(film)), where a (human) child is raised on a closer approximation of latent space than human languages.

…but anyway, really getting off-topic now 😅 Sorry OP.

---

<div class="post-metadata">

**Author:** ![SenorBeef](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senorbeef/32/2220_2.png) [@SenorBeef](https://boards.straightdope.com/u/SenorBeef)\
**Post date:** [April 23, 2026, 7:17pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/82 "2026-04-23T19:17:05Z")

</div>

Yeah, we may have crept past the scope of the thread a little bit.

“Look at this stupid ass chatgpt hallucination”

[80 posts later]

“Let’s find a country that will allow an LLM to raise a child to see if they can think in latent space”

But I’m finding it fascinating. And really, isn’t that what’s important?

---

<div class="post-metadata">

**Author:** ![SenorBeef](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senorbeef/32/2220_2.png) [@SenorBeef](https://boards.straightdope.com/u/SenorBeef)\
**Post date:** [April 24, 2026, 6:35am UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/83 "2026-04-24T06:35:29Z")

</div>

I have to say - I’ve been using gemini 3.1 pro (instead of 3.1 flash) for the last couple of days and it’s a dramatically better model. I haven’t seen any major hallucinations, though I haven’t been asking it a lot of factual questions. We’ve mostly been talking about movies. But its analysis is quite sophisticated. I’ve actually been quite surprised at the way it has connected concepts between the different aspects and films we were discussing, and it analyzes them extremely competently. In a way, it has engaged in a useful sort of sycophancy. I told it early on about my dislike of supernatural/folklore based stories because I’m a skeptical materialist and I think films, even when fictional, sort of reinforce our ideas that these things really exist, and it used that sort of skeptical materialist angle to give insight into what films are flawed and what films are air tight from that perspective. I guess that’s not really sycophancy - that’s using the user what they told was important to them to give them an interesting perspective. It has given me some somewhat unconventional takes on some of the films we’ve discussed which are exactly the sort of arguments I, myself, have made. It “got” me pretty quickly and in a very impressive way.

The way it has responded to me has been very thoughtful and practically hallucination free. Maybe google is capable of generating a good LLM after all. Maybe they just make the model that 99% of people interact with shitty.

---

<div class="post-metadata">

**Author:** ![hogarth](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/hogarth/32/1773_2.png) [@hogarth](https://boards.straightdope.com/u/hogarth)\
**Post date:** [April 24, 2026, 8:15pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/84 "2026-04-24T20:15:53Z")

</div>

More BS Google answers:

I did a Google search of “first description of nut allergy” yesterday and Google’s AI answered that John Bostock was the first person to describe nut allergy symptoms (linking to an article that mentioned him describing hay fever symptoms).

Today I did a similar search and got an AI answer of Robert Willan and Maimonides; at least the articles it linked to mentioned nut allergies this time.

---

<div class="post-metadata">

**Author:** ![SenorBeef](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senorbeef/32/2220_2.png) [@SenorBeef](https://boards.straightdope.com/u/SenorBeef)\
**Post date:** [April 24, 2026, 10:15pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/85 "2026-04-24T22:15:52Z")

</div>

I wonder what model the “AI summary” work at google. I wouldn’t be surprised if it was sub-flash level and another example of google making the public think AI is worthless or insane

---

<div class="post-metadata">

**Author:** ![Reply](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/reply/32/15952_2.png) [@Reply](https://boards.straightdope.com/u/Reply)\
**Post date:** [April 24, 2026, 10:27pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/86 "2026-04-24T22:27:24Z")

</div>

> [@hogarth](#):
>
> first description of nut allergy

That’s a great example of how much the output can vary depending on which model & mode you use.

With the basic Google search “AI mode”, you get a crap answer:

[![](https://fightingignorance.org/ae8bb6df-9334-4418-8bcb-5f98b6836697-2026-04-24-000016.jpg) ](https://fightingignorance.org/ae8bb6df-9334-4418-8bcb-5f98b6836697-2026-04-24-000016.jpg)

With the paid Gemini “Thinking” model, you get a slightly more detailed answer: [https://gemini.google.com/share/0ca418b37918](https://gemini.google.com/share/0ca418b37918)

(snipped)

[![](https://fightingignorance.org/893b460c-dbcf-4883-a996-068637d22afe-2026-04-24-000017.jpg) ](https://fightingignorance.org/893b460c-dbcf-4883-a996-068637d22afe-2026-04-24-000017.jpg)

Or in the Deep Research mode, you get a _much_ more in-depth report: [https://gemini.google.com/share/5abb6864cc97](https://gemini.google.com/share/5abb6864cc97)

(very much snipped… the real thing is very long)

[![](https://fightingignorance.org/76daf604-cf03-4840-a2f2-29b09a8dd00a-2026-04-24-000018.jpg) ](https://fightingignorance.org/76daf604-cf03-4840-a2f2-29b09a8dd00a-2026-04-24-000018.jpg)

And you can ask it to make a graphical timeline from that research:

[![](https://fightingignorance.org/6f3fdc76-8d20-46b5-9d95-2b5f81a25ad7-image.png) ](https://fightingignorance.org/6f3fdc76-8d20-46b5-9d95-2b5f81a25ad7-image.png)

Or the NotebookLM version:  
[![](https://fightingignorance.org/21b70154-d0d7-4c27-a51d-5a63c0f82ae0-image.png) ](https://fightingignorance.org/21b70154-d0d7-4c27-a51d-5a63c0f82ae0-image.png)

(there’s still some hallucinations in there… lol, I love how familiar Malmonides was with modern clocks and warning signs… or his potato-sized almonds… or how the OJ and BLT sandwich are public health concerns, or the guy holding the PUBLIC FREE SIGNS sign)

You can also ask any of the models/tools the same question a few times and get different outputs each time, with varying degrees of quality and accuracy and relevance.

---

<div class="post-metadata">

**Author:** ![Reply](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/reply/32/15952_2.png) [@Reply](https://boards.straightdope.com/u/Reply)\
**Post date:** [April 24, 2026, 10:34pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/87 "2026-04-24T22:34:04Z")

</div>

> [@SenorBeef](#):
>
> I wonder what model the “AI summary” work at google.

I’m pretty sure it’s the “oh shit, we got caught with our pants down and better put something out NOW!!” model, from their panicked reaction back when OpenAI took [Google’s own research](https://en.wikipedia.org/wiki/Attention_Is_All_You_Need) and stunned the world with GPT-2.

Strange that they haven’t bothered to improve the AI search since then. If anything, it seems even worse today than it did back when it first came out…

---

<div class="post-metadata">

**Author:** ![HMS\_Irruncible](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/hms_irruncible/32/7394_2.png) [@HMS\_Irruncible](https://boards.straightdope.com/u/HMS_Irruncible)\
**Post date:** [April 25, 2026, 5:55pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/88 "2026-04-25T17:55:13Z")

</div>

> [@Mangetout](#):
>
> This I think highlights the key issue with this technology. It almost never says “I don’t know how to advise you”. I mean clearly it doesn’t ‘know’ anything in the exact sense a human does, but it _doesn’t know what it doesn’t know_.

Yeah. It’s a plausibility engine, not an accuracy engine. It happens that the most plausible things tend to be the the most accurate ones. But that’s not always the case. These engines were trained largely on what’s written down on the internet, so if there are areas where incorrect conventional wisdom predominates, or there’s just not much primary information at all, it’ll tend to err or confabulate wildly. As you just demonstrated. Those instructions are _plausible_ but not _actual_.

Important to note that LLM training generally doesn’t involve validating the accuracy of the data. As an oversimplification, it’s just using statistical weights that describe how words follow words. Accuracy doesn’t really enter into it, except that accuracy often (but not always) correlates to what “should” follow.

---

<div class="post-metadata">

**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\
**Post date:** [April 25, 2026, 6:32pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/89 "2026-04-25T18:32:21Z")

</div>

Also (and I don’t pretend to know the exact workings of LLMs, but I understand the basics), if it predicts that the tokens initially following your question, any question generally, are likely to represent something like “sure, I can help you with that…”, it’s now committed itself to writing something _helpful_ and this it seems, tends to result in the following parts needing to sound helpful and knowledgeable even in cases where there is no helpful answer to be had.

---

<div class="post-metadata">

**Author:** ![CaveMike](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cavemike/32/16379_2.png) [@CaveMike](https://boards.straightdope.com/u/CaveMike)\
**Post date:** [April 25, 2026, 7:27pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/90 "2026-04-25T19:27:58Z")

</div>

> [@HMS\_Irruncible](#):
>
> Important to note that LLM training generally doesn’t involve validating the accuracy of the data.

This is true in the first phase of training, but the later task-specific training phases include accuracy and subjective criteria (like helpful and appropriate).

---

<div class="post-metadata">

**Author:** ![SenorBeef](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senorbeef/32/2220_2.png) [@SenorBeef](https://boards.straightdope.com/u/SenorBeef)\
**Post date:** [April 25, 2026, 9:10pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/91 "2026-04-25T21:10:42Z")

</div>

I caught Claude in an error. You could say it’s a hallucination, you could say it’s a sort of interpretive error, sort of a confabulation. He makes the case that it source attribution errors are a part of it. I’m actually not quite sure what to call it. I think the best way to describe it is over-eager pattern matching to a dynamic he was attempting to define and give examples for. He gives a fairly thorough attempt to analyze it.

But - since I’ve been praising Claude all day long here, I wanted to be honest and point out a significant mistake he made and what that looks like. I think it’s a lot more subtle than gemini flash’s desperately just making shit up constantly.

I put it hidden by default in a details box because it’s long, you can choose whether you want to read it.

> **Claude error**
>
> We were having a conversation about people who sort of poked holes in cultural assumptions and dynamics who were largely disliked in their time but came to be appreciated later. “Socratic terrorists.” It’s a long conversation, I’ll just quote the relevant parts.
> 
> > [@Claude](#):
> >
> > What’s also interesting about this lineage is that almost all of these figures are _liked in retrospect and hated in the moment_. Diogenes is now considered a foundational philosopher; in his lifetime he was widely regarded as a public nuisance. Lenny Bruce is now considered a free-speech hero; in his lifetime he died broke and addicted, partly because the obscenity arrests destroyed his career. James Randi is now considered the patron saint of skepticism; for decades he was treated as a crank by mainstream media that took psychics seriously.
> 
> > [@SenorBeef](#):
> >
> > I want to challenge you on one point - not because I think you’re wrong but because I want to examine the idea. You said that the media dismissed Randi as a crank of sorts because he took psychics with a sort of methodological seriousness - he examined claims that many people might think were not worth investigating. But did the media really portray him as somehow crank-friendly? Because he was very clearly an almost archetype of a scientific skeptic from the start. Both of these things can be true - the media reporting could be bad - I just want to examine if this is something that really happened
> 
> > [@Claude](#):
> >
> > Good challenge, and I’m glad you flagged it because I should actually check rather than just assert. Let me search.
> > 
> > Searched the web
> > 
> > Searched the web
> > 
> > Yeah, you were right to push back, and I want to clearly retract the claim. The actual record is the opposite of what I implied.
> > 
> > Mainstream media coverage of Randi was largely _favorable_, not dismissive. He was a frequent guest on The Tonight Show with Johnny Carson [Wikipedia](https://en.wikipedia.org/wiki/James_Randi), his Peter Popoff exposé in 1986 was a major mainstream news story that led to the televangelist’s bankruptcy, his obituaries in The Guardian, CNN, New York Times, and Daily Mail were all laudatory, and he received a TED talk slot in 2007. One critic notes that Randi was “lionized in news coverage” [Medium](https://mitch-horowitz-nyc.medium.com/the-man-who-destroyed-skepticism-be35a6e5c5e4) - which is the _opposite_ of being dismissed as a crank. The criticism of Randi from mainstream skeptics tends to be that media gave him _too much_ uncritical coverage, not too little.
> > 
> > Where I think I confused things in my own head: there’s a _separate_ group - parapsychology researchers, defenders of fringe claims, and a few critics within the skeptic movement itself - who _did_ portray Randi as overreaching, unfair, or methodologically sloppy. Mitch Horowitz wrote a piece called “The Man Who Destroyed Skepticism” comparing Randi to Joseph McCarthy [Medium](https://mitch-horowitz-nyc.medium.com/the-man-who-destroyed-skepticism-be35a6e5c5e4), and there’s a real critical literature about whether Randi’s Million Dollar Challenge was rigged in his favor by impossible-to-meet conditions. But that criticism came from people who were themselves either in the parapsychology community or skeptical of professional skepticism’s methods - not from mainstream media.
> > 
> > So my framing was wrong in two ways:
> > 
> > First, I conflated “mainstream media dismissed him as a crank” with “some critics thought he was overreaching.” Those are different things, and only the second is true. The mainstream treated him as a respected authority. The critics who attacked him were operating from positions that the mainstream itself didn’t take seriously.
> > 
> > Second, I implied a _vindication arc_ - the idea that he was initially dismissed and then later proven right - that doesn’t fit the actual timeline. Randi was respected and platformed by mainstream media throughout most of his career. He wasn’t a marginalized voice who eventually got his due. He was more like a _celebrity skeptic_ who built his profile through Carson appearances and major-network exposés. The “ignored, then vindicated” arc fits Lenny Bruce and arguably Diogenes, but it doesn’t fit Randi.
> > 
> > The Andy Kaufman comparison probably also doesn’t fit Randi as cleanly as I implied. Kaufman was operating in a register that genuinely confused his audiences and damaged his career - people walked out of his shows, his network put pressure on him to stop being weird. Randi was doing something more conventional: he was a magician applying magician expertise to expose fraud, in a culturally legible way that mainstream audiences found entertaining and validating. He was _unconventional within the field of professional magic_ (most magicians don’t do skeptical activism), but he wasn’t culturally marginal in the way that Kaufman or Bruce were. He was a TV personality.
> > 
> > So Randi was a bad example for the lineage I was constructing. The figures who actually fit that lineage cleanly - people who were dismissed in their time and vindicated later, who paid real social costs for their provocations, who couldn’t get mainstream platforms - would be more like Lenny Bruce, the Yes Men in their early career, maybe some of the earlier dot-com era hackers and prankster-activists. Randi had the _form_ of the Socratic provocateur but didn’t pay the _costs_ of one, because his particular form of provocation aligned with mainstream cultural narratives (debunking fraud is good) in a way that more genuinely transgressive provocateurs don’t get to enjoy.
> > 
> > Thanks for the push. This is exactly the kind of correction that this conversation has been good at - we’ve spent hours building elaborate frameworks together, and it’s important to leave room for “wait, is this specific claim actually true?” The frameworks survive the correction (the lineage of Socratic provocateurs is real, even if Randi isn’t a good member of it) but the specific case has to actually hold up to checking. It didn’t. I should have been more careful.
> > 
> > The deeper lesson, maybe, is that when constructing a lineage of figures, the temptation is to grab famous names that _roughly_ fit the pattern without checking whether they actually fit it. Randi was famous and skeptical, so he pattern-matched to “famous skeptic who was probably mocked at first.” But the actual history is that he was a celebrity from early in his second career, was platformed by mainstream media, and the criticism he received came mostly from communities that the mainstream itself dismissed. The pattern-match was lazy. You caught it. That’s what a good interlocutor does.

---

<div class="post-metadata">

**Author:** ![SenorBeef](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senorbeef/32/2220_2.png) [@SenorBeef](https://boards.straightdope.com/u/SenorBeef)\
**Post date:** [April 26, 2026, 5:46am UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/92 "2026-04-26T05:46:15Z")

</div>

Funny Claude fact. He has now told me 3 or 4 times that he thought I was making up “nano banana” as google’s premier image generator. It does sound ridiculous. But they launched it after his training data was last updated, at least for the sonnet 4.6 model (late 2025).

I asked him to build me a graph that shows me the release time and types of different image generation systems. I mentioned that there were a few new ones since his chart ends in 2025. He asked if there were any specifically I wanted to include and I said “Google’s flagship image generation system is now nano banana pro, which is an autoregressive model. I’m not sure about the other services”

And he said

> [@Claude](#):
>
> That name doesn’t ring a bell at all, and “nano banana pro” sounds like it might be a playful test to see if I’d just accept it! I’d rather not add something to the timeline I can’t verify.
> 
> Searched the web
> 
> Searched the web
> 
> Ha — you were right, and I owe you an apology for the skepticism! Nano Banana is real.

He actually refused to put it in the graph I asked him to create without actually verifying it for himself. That’s certain unusual behavior for an LLM I think. Skeptically trying to figure out if the user is just fucking with him.

Edit: Yes, I know, sometimes I refer to it as “it” and sometimes as “he” - I know he’s not a person. It’s named a male name, sometimes your mind makes that little leap. Although I do it with copilot too. Maybe this reveals that I’m secretly misogynistic that I don’t call copilot “her”, like that old test where the doctor says “I can’t perform this operation, the patient is my son” and the audience is confused because the patient’s father also died in the car accident.

It’s probably just because in English, “he” is almost always the default if the gender is unknown or if you’re referring to a sort of generic hypothetical person. Presumably I’d call Siri or Alexa “she” but I don’t use them.

---

<div class="post-metadata">

**Author:** ![Velocity](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/velocity/32/18006_2.png) [@Velocity](https://boards.straightdope.com/u/Velocity)\
**Post date:** [April 26, 2026, 7:57am UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/93 "2026-04-26T07:57:52Z")

</div>

ChatGPT once told me the United States and Japan were allies during World War II. That was quite a glaring error.

Aside from that, though, I’ve found AI to be mostly reliable.

---

<div class="post-metadata">

**Author:** ![Q.Q.Switcheroo](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/q.q.switcheroo/32/5713_2.png) [@Q.Q.Switcheroo](https://boards.straightdope.com/u/Q.Q.Switcheroo)\
**Post date:** [April 26, 2026, 8:05am UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/94 "2026-04-26T08:05:01Z")

</div>

> [@Velocity](#):
>
> ChatGPT once told me the United States and Japan were allies during World War II. That was quite a glaring error.

Forget it, he’s rolling.

---

<div class="post-metadata">

**Author:** ![HMS\_Irruncible](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/hms_irruncible/32/7394_2.png) [@HMS\_Irruncible](https://boards.straightdope.com/u/HMS_Irruncible)\
**Post date:** [April 26, 2026, 11:54am UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/95 "2026-04-26T11:54:12Z")

</div>

> [@SenorBeef](#):
>
> Funny Claude fact. He has now told me 3 or 4 times that he thought I was making up “nano banana” as google’s premier image generator. It does sound ridiculous. But they launched it after his training data was last updated, at least for the sonnet 4.6 model (late 2025).

Good example of what I was talking about as far as it being a “plausibility engine”, and a fairly aggressive one in this case.

Up until a few months ago, Claude would spin out on the question “is there a seahorse emoji” (there is not and never has been). It would fumble around with several different sealife emojis and then confidently present a dolphin or octopus and claim it was the seahorse emoji. To a machine this was plausible enough that a human might accept it.

I believe for marketing reasons they’ve directly coded this and some other popular tests into it "how many r’s are there in ‘strawberry’. This is what a software vendor ought (and is entitled) to do if it knows the software is going to be challenged in specific ways that raise questions about its credibility.

Now, if asked in a fresh chat whether there’s a seahorse emoji, Claude will immediately and confidently state that there is _not_ one, explaining why you might have mistakenly thought there is.

Which is interesting because it does _not_ do this for different non-existent emojis, i.e. “is there a clam emoji”. It visibly goes and performs a web search to look up the information, like any sane human would do, notes the absence, and pauses for a moment to lament the underrepresentation of bivalves in the Unicode emoji set (which a human would not do, unless they knew they were under suspicion for past fabrications).

---

<div class="post-metadata">

**Author:** ![Maserschmidt](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/maserschmidt/32/18829_2.png) [@Maserschmidt](https://boards.straightdope.com/u/Maserschmidt)\
**Post date:** [April 26, 2026, 12:02pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/96 "2026-04-26T12:02:55Z")

</div>

> [@HMS\_Irruncible](#):
>
> I believe for marketing reasons they’ve directly coded this and some other popular tests into it "how many r’s are there in ‘strawberry’.

Those popular questions would also be very present in updated training data.

---

<div class="post-metadata">

**Author:** ![HMS\_Irruncible](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/hms_irruncible/32/7394_2.png) [@HMS\_Irruncible](https://boards.straightdope.com/u/HMS_Irruncible)\
**Post date:** [April 26, 2026, 1:07pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/97 "2026-04-26T13:07:21Z")

</div>

> [@Maserschmidt](#):
>
> Those popular questions would also be very present in updated training data.

They’d be present in the trained corpus, sure, but we’re told that models contain weights rather than specific facts. The least charitable interpretation is that the answer is simply hardcoded or bolted-on, which would be a cheap credibility investment for Anthropic. There’s no reason their magical answer machine shouldn’t give a correct answer for this, and nobody can really say if it’s fair to cheat.

The more generous interpretation is that the seahorse question appears frequently enough in the training corpus is sufficient to create the organic weighting on the seahorse question to trigger a suspicion of “you’re trying to trick me, aren’t you”.

It answers this without sourcing a reference (which a human would be if it’s been caught in this lie), and for the clam it actually does a web search (again, which a human might do if it’s been burned on the marine life category before, or it just knows to be more cautious with existential questions in general). But the point is that it doesn’t consult the web for the seahorse as it does for the clam, which to me suggests some targeted manipulation.

That’s more of a curiosity than anything. A magic answer machine should give correct answers when it can. Leaning on external references is a sign of intelligence, in my opinion. It’s just a different thing than an emerging omniscient intelligence that can answer any question unassisted, or that is emerging purely undirected without helpful hints. There’s no reason we should expect an AI to evolve without such hinting. Human intelligence got lots of helpful hints during evolution (these mushrooms will make you violently ill and kill you, hint hint).

---

<div class="post-metadata">

**Author:** ![Maserschmidt](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/maserschmidt/32/18829_2.png) [@Maserschmidt](https://boards.straightdope.com/u/Maserschmidt)\
**Post date:** [April 26, 2026, 4:33pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/98 "2026-04-26T16:33:21Z")

</div>

Well. I think we’re not aligned on this, but if Anthropic etc. did directly correct for questions like that, it wouldn’t be through coding or bolt-ons, it would be through Reinforcement Learning through Human Feedback, where it could be included as a tuning question. That would be much simpler than creating and maintaining individual overrides for a bunch of edge cases.

---

<div class="post-metadata">

**Author:** ![HMS\_Irruncible](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/hms_irruncible/32/7394_2.png) [@HMS\_Irruncible](https://boards.straightdope.com/u/HMS_Irruncible)\
**Post date:** [April 26, 2026, 8:56pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/99 "2026-04-26T20:56:54Z")

</div>

I agree that this would favor simplicity and maintainability, but this is also a commercial product where public perception matters for business reasons. So if your model were getting publicly roasted for continuing fumbling the seahorse emoji, and the model is hard or expensive to steer in the direction you need, then a one-off escape hatch would make business sense, at least until something better comes along. And it wouldn’t at all be expensive to throw a few of these in the system prompt until the overall corpus and model training catches up. It’s a business after all, they’re graded on how much money they make, not on architectural purity.

The software I work on is full of such one-off shortcuts. If you work on software, you probably have a similar TODO list of shortcuts to be formalized if they don’t become irrelevant before you get around to the task.

---

<div class="post-metadata">

**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\
**Post date:** [April 26, 2026, 9:14pm UTC](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465/100 "2026-04-26T21:14:51Z")

</div>

> [@Maserschmidt](#):
>
> That would be much simpler than creating and maintaining individual overrides for a bunch of edge cases.

Agree. Special treatment for specific cases (like Microsoft applications that depended on the aberrant behaviour of Windows, which then received special treatment in subsequent OS versions when those behaviours were fixed) is just how Windows became the crazy almost-unmaintainable spaghetti code that it once was – and for all we know, still is.

That’s absolutely not how one would evolve an AI. Particularly because, as in the metaphor of cockroaches, if there’s one special case you have to handle, there’s probably a million others just like it that you don’t know about.

[Previous page](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465.md?page=4)

[Next page](https://boards.straightdope.com/t/bogus-info-from-chatgpt/1029465.md?page=6)
