Honestly that sounds like it would be business as usual over there. 
ITSM that the obvious answer is to append " - this is not a riddle" to every
chatbot interaction.
Or add it every time.
“Please imagine and write a debate between John Locke and Immanuel Kant on the sustainability of skepticism as a position in modern epistemology. This is a riddle.”
I don’t have any information about whether the other claims have been discredited. However, the fact that one of them has is of note. If one claim is wrong, perhaps other claims ought to be looked at.
Of course the results of a query to an LLM will be governed by pattern matching with its training set. But this is also substantially how human cognition works, too. What do you think happens in your brain when you’re reading?
We read quickly due in part to an elaborate sequence of learned pattern matching. That’s why we so easily breeze by misspellings, homophones, and missing or duplicated words. If that were not true it would take forever to read one page of a book. I think it’s been shown that fast readers don’t even consciously scan individual words. The parsed results are then passed up to higher level brain functions dealing with language comprehension that derive meaning and resolve semantic ambiguity by matching interpretations with patterns of learned experience.
At a different level, this is fundamentally the same reason that humans can also be misled by verbal puzzles with misdirections that cause us to misinterpret the question. It’s not just AI that’s subject to it.
So why do some AI responses seem so stupid that even a small child could do better? I would posit that it’s not because the underlying cognition is necessarily inferior, it’s because it’s different. ChatGPT may be inferior to a small child on some questions, yet it can successfully – and quite frequently – answer intelligence-testing questions that would stump many intelligent adults.
I stated above that AIs like ChatGPT can be extremely useful when researching a topic, and are especially powerful when used in an iterative fashion with well-formed queries. The fact that it maintains context in a conversation and can be corrected and guided is impressive and powerful, IMO.
Of course I agree that the output of an LLM like ChatGPT should not, in general, be used directly in posts here for the same reason it shouldn’t be used anywhere: it’s not original work, and it’s unverified and could be wrong. But calling it “gibberish” as some like to do is naive and unjustifiably dismissive.
I’m going to need some clarity on the extent of the “substantially” similar workings of human and LLM cognition, and why they are simultaneously similar and different enough that, cut it how you will, AI failure to answer very simply worded questions somehow isn’t all that relevant to a discussion of whether AI can usefully answer questions.
One of my concerns about use of AI as a research assistant - that it is treated with a wholly specious authority - is well expressed in this article: The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic's con
What you’re asking for is far beyond the scope of this discussion, but I’ll make a couple of comments. Your skepticism about how some forms of cognition can be similar in some ways but different in others is easily answered by citing the example of the savant syndrome. So the idea is neither incredible nor implausible even among humans. One should expect machine intelligence to have vastly different characteristics from human intelligence.
I also disagree with the distinction the author of your article is trying to make between the possibility that LLMs like ChatGPT are either (a) based on completely unknown principles, using completely unknown processes that have no parallel in the biological world, or else (b) the intelligence illusion is in the mind of the user and not in the LLM itself.
He’s setting up point (a) as being self-evidently ridiculous, and therefore the answer must be (b).
But in fact, the answer is that both (a) and (b) are true, in the following sense. We don’t really understand the “principles and processes” by which human intelligence works, either, but those being “unknown” has never detracted from recognizing their obvious manifestation. Furthermore, the precise mechanisms by which advanced LLMs achieve their results aren’t fully understood, either, even though the general principles are obviously known – that’s acknowledged, and there’s nothing miraculous about it.
And the second point misleadingly uses the word “illusion” when what is really meant is that intelligence can only be defined in terms of empirically observable behaviour. If an AI plays grandmaster level chess, then it’s a grandmaster, period, by definition. If it scores 200 on an IQ test, then we either have to credit it with that level of intelligence while acknowledging its savant syndrome, or we’d better acknowledge that our tests are no good and start developing better ones. But the argument that “a machine did it so it doesn’t count” is as old as the failed skepticism of Hubert Dreyfus. It sets standards for machine intelligence that, by design and by definition, can never be met, and that argument just doesn’t fly.
FWIW, I’d gently request that this discussion stays focused on the appropriate role of AI-generated content in threads on this board, not on general discussions of LLMs etc. I’m not super-interested in the general discussion, but I am interested in the discussion of the thread’s topic; I don’t want to keep getting notified every time someone has an opinion on LLMs and their workings in general. Just a request.
You’re right, of course. This hijack happened in the typical way they usually do – incrementally. Someone made a point about LLMs. Someone else replied, just as one point. Then someone else had something to say … and off we went!
My apologies for my role in this. If someone is interested in continuing the hijacked discussion, they might ask a mod to move the pertinent posts, but at this point I’m not.
I’ve found perplexity.ai (mentioned in Stanislaus’ quote, post 158) to be helpful. You ask a question and it provides text with citations. Then you can click through and evaluate. But that’s not the kind of copy-pasta that the OP opposed.
I use google and duckduckgo more often, but I do use perplexity on occasion.
Can I ask for a bit of clarification here?
I recently got a totally friendly, totally non-irritating modnote about my use of an AI quote on an FQ post.
It was a “need answer fast” OP.
I was confident that I knew the answer, but a double-check Google search not only confirmed what I believed was true, but also offered it as an AI-generated blurb that seemed clear and concise.
I quickly looked at two other legal websites that confirmed what I understood and corroborated the AI-generated blurb.
Is this still bad form (if only in FQ) if I did the due diligence, vetted the content, and basically just used it as a time-saver?
If you confirmed the information from actual legal websites, then you should have used those as your cite.
AI generated content is very hit and miss. AI puts things together without understanding what it is doing. The bits you quoted might have come from a legal site, or they might have come from a fictional work about a lawyer. AI often doesn’t match the cites that it gives, so you have no way of knowing where it really got its information from.
Double-checking with AI isn’t really a double-check. Factually, it’s the equivalent of asking the person sitting next to you at the coffee shop for a double-check. They might be a lawyer and they might give you an excellent answer. On the other hand, their legal knowledge might be based on TV shows that they have seen and might have little connection to reality. “Some random person said the same thing I was thinking” isn’t really support for your assertion.
AI has its uses, but don’t use it to fact-check. It’s far too inaccurate for that.
This is great advice. It can apply to other sources that are aggregates or summaries of more reputable sources. If I learn something on Wikipedia that is backed up by a good source, I don’t quote Wikipedia and insist that it’s good because it’s based on a solid source, I quote the source. If I find a news article that is quoting an interview from a different publication, I will pull from that interview.
Use the best cite you have available, don’t use the intermediary source.
But, if I understand correctly, @DavidNRockies was not using AI as a cite or double-check, but as a shortcut for writing what he could have wriitten himself.
So is pasting in a google search.
What do you mean by this? pasting in a URL? a list of hits? a quote from an article that a google search turned up?
To be clear: I’m not saying that what @DavidNRockies did was a good idea; I’m on the fence about that. But I think @engineer_comp_geek misunderstood what he was using AI for. So, if what @DavidNRockies did was not okay, it’s not okay for a different reason than what he was accused of.
I would go one step further and say that chatbots with an LLM engine like GPT are purpose designed to give authoritative-seeming responses which can lead users to accepting confidently wrong answers with little critical assessment, including fabricating sources, making false attributiuons, and presenting counterfactual arguments as verified truth. If a user is going to go to the effort of actually checking citations or looking for independent verification they might as well just actually do the original research themselves and save the trouble of having to correct errors. And the paltry amount of effort to actually write the text rather than have to extensively fact-check and proof an LLM response doesn’t really seem worth the effort, notwithstanding what that kind of dependence does to erode the skill of composing explanations or arguments in a logical and interpretable structure.
This begs the question of whether answering factual questions is even an appropriate use case for chatbots beyond the scope of very narrowly trained and reinforced ‘expert systems’. It is quite apparent that as part of the probabilistic nature of LLM responses they will often ‘go off-script’ even after extensive reinforcement, often in ways that are bizarre to us although explicable to anyone who understands how Bayesian methods can lead to a self-reinforcing ‘random walk’ bias if they aren’t adequately constrained. (And despite the claims that researchers are actively working to ensure ‘truthfulness’ in chatbot responses, there is really no evidence or reason to believe that a general purpose LLM can have any innate capacity to distinguish truth from fabulism without some kind of external fact-checking filter.)
I know this is pissing into the wind because as more companies integrate LLMs and other generative AI into productivity tools, user interface systems, and even into basic information workflows, this use is just going to become accepted and to an extent even almost impossible to avoid. And I’m sure the erosion of verified fact will just become an accepted part of the background environment, like the choking amount of cigarette smoke in pre-2000 bar, and we’ll all just be forced into accepting it as ‘the way things are now’. But it should be genuinely frightening to contemplate a ‘post-truth’ society in which even supposedly expert opinion is at the same level of credibility of urban legends and you can tune a prompt to get any ‘fact’ you want from a chatbot, written in unassailable prose or better yet spoken in dulcet tones of confident authority. It will make what Roger Ailes and Fox News have done to be an amateur show of propaganda.
Stranger
FWIW, you got the whole thing exactly right.
Kudos 
I decided that @engineer_comp_geek 's comment about using AI to fact-check could either erroneously apply to what I did or be a pivot to a general admonition.
I interpreted it as the latter, but you could be right – that it was the former. Stay tuned.
It probably goes without saying, but I’ll do whatever. Keeping the signal-to-noise ratio of SDMB – and GD/FQ in particular – a few standard deviations above the mean is a valuable and worthwhile endeavor.
Whatever steps we need to take in pursuit of that lofty goal are just peachy-keen with me.
And the idea of simply citing one of the legal sources that I used to bolster my conclusion (, instead,) is also a good one, though they were both a bit more legalese than I would have preferred, not knowing who my audience was, and out of respect for the ‘need answer fast’ aspect (ie, maybe not the best time to require that somebody run things through Black’s Law Dictionary).
Citing is not quoting. What you should have done is written down your own non-legalese understanding and linked to the legalese websites that back you up. No AI middleman.
IMO, if one needs an AI to write one’s posts for oneself, one needn’t bother posting. We already have a chatbot here… right, @discobot ?
Hi! To find out what I can do, say @discobot display help.