AI search results can be crazy stupid (cars named after plants)

[wolfpup]

@Stranger_On_A_Train

@Sherrerd

(Also, I’m really curious about how AI developed to be a routine fabricator of results. What human choices led to this outcome?—that one is likely beyond the scope of the thread, but I remain interested.)

It isn’t a ‘choice’ per se; it is a consequence that large language models (LLMs) which power chatbots are ‘just’ really complex word completion systems. They have enough parameters and a vast pool of training data to produce fairly sophisticated textual responses to a prompt by dint of the metasemantics of being trained on so much text, but while they often appear to ‘understand’ the context of a question they don’t actually know whether a response that the produce is factually correct or not. So, they can (and often do) generate complete nonsense and fabricate seemingly factual information expressed in an authoritative manner, which convinces casual users and devotees that they are somehow reliable general knowledge systems.

Your observation here is what I would call “biased” rather than something like “wrong”. It continues your drumbeat of denying the reality of how extremely useful advanced LLMs like GPT 5 can be, especially with their new HealthBench criteria. See this, for instance:

GPT-5 surpasses human doctors in medical diagnosis tests - International...

I acknowledge your skepticism and know that you will continue onward on that path, but I do have a relevant anecdote. WIthout going into details, I have medical symptoms that may be due to a variety of causes, some minor, some potentially very serious. I have been engaged in a long discussion with ChatGPT about the symptoms and the results of preliminary medical tests like ultrasound and what can be concluded from them, and the most promising next steps.

According to you, I am a complete idiot relying on a “text completion engine” to give me advice. According to both my GP and my referred specialist, I am a surprisingly knowledgeable patient and am being directed to the diagnostic resources that they and I – based on my best information including GPT – agree are best.