I see a kernel of merit in what the OP is saying, but I vehemently disagree with the reasons given. Saying that AIs are “notoriously unreliable” and that “they string words together real plausibly, but they’re not relying on research; rather, they’re relying on the probability that certain verbs, nouns, and conjunctions (etc) would go together in a particular format” shows a level of disdain that isn’t supported by AIs’ real-world performance. As I noted here, GPT-4 scored in the 90th percentile on the Uniform Bar Exam; it aced all sections of the SAT, which among other things tests for reading comprehension and math and logic skills, and it scored far higher across the board than the average human; it passed the Wharton MBA exam on operations management, which requires the student to make operational decisions from an analysis of business case studies; and it did equally well on many other tests. They’re certainly not infallible, but OpenAI and Google Research, among others, are betting big money that large language models and other AI technologies will have enormous commercial utility. There’s clearly a lot more going on here than simple token prediction, and a big part of that is the unexpected skills that emerge on very large model scales (“emergent properties”) and after extensive machine learning.
Maybe the disdain originates from the trajectory that the public perception of GPT has been following since it became widely publicized: first, it was hailed as amazing and revolutionary, but as some of the failings of the earlier models became known, there was a tendency to dismiss it as “just” a stochastic parrot driven by token prediction. The implication is that the demonstrated competence is purely an illusion, but as shown in the link above to some of the tests GPT-4 has passed, this implication is false.
In the 1960s the AI researcher Joseph Weizenbaum developed a conversational program called Eliza that was modeled on the style of a non-directive therapist; it never had anything original to say, but it would respond in a human-like way to comments that the user typed. It clearly had no understanding of anything and was completely predictable. There was a story that Weizenbaum’s secretary was enthralled with it and loved conversing with it, which subjected her to a certain amount of ridicule in the AI community. I have a feeling that much of the disdain for GPT stems from the notion that if it runs on a computer, then it’s just some stupid algorithm just like Eliza. The unfortunate use of the prefix “chat” in ChatGPT 3.5 reinforces the impression that it’s just a mindless chatbot that strings words together, further reinforced by a simplistic but totally unrealistic impression of how it works.
Sorry for the long preamble but this is at least the third time that the OP’s disdain for GPT has come up on this board and I’m perplexed by the reason for it. But on the practical question of what our protocols should be on this board with respect to AIs like that, I don’t see any problem with occasional use of GPT as a cite, provided that (a) it’s attributed and not passed off as the poster’s own writing, otherwise it’s like any other kind of plagiarism, (b) vetted by poster by cross-referencing with one or more reliable sources, and (c) sparingly used, otherwise it becomes a substitute for the poster’s own knowledge and reasoning.
In particular, I don’t see a problem with the post that @Crane made that spawned this complaint. It was attributed, it’s not something the poster habitually does, and as far as I could tell from some casual Googling it’s substantially correct. So it adds useful information to the discussion. I agree that the absence of cited sources is problematic, but if the information is suspect it can easily be refuted. If, instead, we insist that we should never do this, then we’re depriving ourselves of a potentially very useful tool – one that some important companies are betting will be worth a fortune in the commercial marketplace. It’s not just an information source; among its skills is the ability to summarize potentially opaque prose, such as sections of scientific papers, and the ability to analyze and organize unstructured data. We should just use it honestly and with proper attribution.
