# AI search results can be crazy stupid (cars named after plants) **URL:** \ **Category:** Miscellaneous and Personal Stuff I Must Share\ **Tags:** ai\ **Created:** [October 2, 2025, 9:00pm UTC](https://boards.straightdope.com/t/ai-search-results-can-be-crazy-stupid-cars-named-after-plants/1023619 "2025-10-02T21:00:06Z")\ **Posts on this page:** 1\ **Showing post:** 20
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [October 3, 2025, 12:50am UTC](https://boards.straightdope.com/t/ai-search-results-can-be-crazy-stupid-cars-named-after-plants/1023619/20 "2025-10-03T00:50:02Z")
> [@wolfpup](#): > > Your observation here is what I would call “biased” rather than something like “wrong”. It continues your drumbeat of denying the reality of how extremely useful advanced LLMs like GPT 5 can be, especially with their new HealthBench criteria. How useful is a system that often provides incorrect factual information? > [@wolfpup](#): > > See this, for instance: > > [GPT-5 surpasses human doctors in medical diagnosis tests - International Hospital](https://interhospi.com/gpt-5-surpasses-human-doctors-in-medical-diagnosis-tests/) From your cite: > _The researchers emphasise that these findings represent performance in controlled testing environments rather than real-world clinical practice. “It is important to recognize that these evaluations occur within idealized, standardized testing environments that do not fully encompass the complexity, uncertainty, and ethical considerations inherent in real-world medical practice,” they cautioned in their discussion._ In short, it is good at taking a standardized test which includes some multi-modal elements. That is an impressive feat of replication but doesn’t mean that it has any actual understanding of the complex interactions of a human patient in the physical environment. > [@wolfpup](#): > > I acknowledge your skepticism and know that you will continue onward on that path, but I do have a relevant anecdote. WIthout going into details, I have medical symptoms that may be due to a variety of causes, some minor, some potentially very serious. I have been engaged in a long discussion with ChatGPT about the symptoms and the results of preliminary medical tests like ultrasound and what can be concluded from them, and the most promising next steps. > > According to you, I am a complete idiot relying on a “text completion engine” to give me advice. According to both my GP and my referred specialist, I am a surprisingly knowledgeable patient and am being directed to the diagnostic resources that they and I – based on my best information including GPT – agree are best. Sure, your interactions with ChatGPT gave you the nomenclature and jargon to sound like an informed patient, and even some cursory information about appropriate diagnostics (information you probably could have gotten by reading the same online sources that ChatGPT was doubtlessly trained upon) but that doesn’t make it an expert or reliable system for medical diagnostics. It is repeating the use of language found int he structure of sources of training data, and provided those are credible sources it is providing cromulent-sounding guidance. But it has no actual knowledge of medical diagnostics or pathology of disease as applied in a clinical setting; it just has the text and image date from which to synthesize a statistically appropriate response. Given a sufficiently large base of training data, enough parameters to generate a complicated response, and a “Chain-of-Thought” recursive model to enable it to break the parsing of the prompt into manageable segments such that it doesn’t immediately spiral off topic and ‘hallucinate’ a completely inappropriate answer, it can produce a plausible-seeming response that reads like what the first pass of an attending physician might write in their notes. That doesn’t mean that it is actually making a good diagnosis, or that it would recognize an obvious anomaly or error, or that it could formulate an appropriate treatment plan. Stranger --- _[View the full topic](https://boards.straightdope.com/t/ai-search-results-can-be-crazy-stupid-cars-named-after-plants/1023619)._