It is even worse than this, though, because it is conditioned to provide responses in a manner that appears authoritative in tone even if the answer is total nonsense, making these systems expert ‘bullshit generators’. This leads people to uncritically accept responses even though basic intuition and ‘common sense’ should indicate that there is an error. Here is an example of this phenomenon. Unfortunately, people are often lazy or do not have the ability to critically examine the response, and become reliant upon the chatbot to provide factual data when, as noted above, it has no ability to distinguish fact from semantically-cromulent gibberish.
There are efforts to apply various post-response methods to attempt to verify responses (fact-checking basic information and references, or using retrieval-augmented generation to draw factual information from a validated source and just use the LLM to put it into an appropriate textual frame) but frankly there are inherent problems in responding to more complex prompts that this approach cannot resolve, and without some application of ‘common sense’ that we have no idea how to build into a language model (because the LLM only knows how to process text, not how to relate it to any kind of ‘real world’ experience) this is almost certainly a fundamental limitation in the reliability of this approach.
Except Wikipedia is curated and entires are reviewed by knowledgable peers. It is not perfect, and of course people can maliciously post inaccurate information, but by and large it has become a reasonably credible source of basic and sometimes even technical information which is largely backed up by cited references which can be verified. An LLM will just generate syntactically-correct gibberish to whatever you prompt it to respond to, and will literally manufacture citations if you ask for them because it has no comprehension that a citation is a verifiable source of fact. All they are doing is manipulation tokens (words or collections of words) in a way that is statistically consistent with its training data set.
Stranger