Absolutely.
I have mentioned before that I have a brother-in-law who, at the drop of a hat, will give you a long, detailed, and almost completely wrong ‘explanation’ of topics he knows nothing about. 
There’s a middle ground here. The current state of LLM AIs is that they can correctly answer a wide range of questions, but that they still give incorrect answers frequently enough that they can’t be relied upon.
I look at GenAI as having a super smart friend who always gives you answers about whatever you ask, but you occasionally catch them making stuff up when they forget the details.
You know that it isn’t malicious, and their friendly and detailed responses are almost always helpful.
But… you rely on them for things that are verifiable as you go, such as progressively writing a complex application in Python, or setting up your home automation. However, you never take their words verbatim and use them as your own when writing.
And you keep your BS meter tuned and call them on it–they’ll quickly say “You’re right, there isn’t an option X in application Z. Here’s the correct way to do it…”
Such a friend isn’t rendered useless because they occasionally make stuff up in a non-malicious way. Surely we all know someone like this, and they can contribute greatly to our work.
Exactly. And I want to be clear that I although I frequently defend GPT, I have never claimed – either in this thread or anywhere else – that it’s consistently reliable. But neither are people, or any other source.
What I will say, though, is that in my experience GPT today is right much more often than it’s wrong, even about complex issues. It can answer long and complex questions and be informative and directly on point (and usually accurate).
One is always free to verify its responses with other sources, and if it’s anything important, one absolutely should. But it’s usually much easier to verify an existing response than to dig up that information yourself, especially if the issue is complicated and you may not even know exactly the right question to ask.
Thus, in my opinion GPT as it exists today already provides tremendous utility, and it’s always getting better. Never mind whether it “really” understands anything, the only valid question is “is it useful?” I’ve had long and informative conversations with it about everything from the thermodynamics of pizza stones to the nature of black hole singularities. It is assuredly not just a “bullshit generator”. There wouldn’t be much point in pouring billions of dollars into research and further development if that’s what it was.
There is a clear incentive to develop a superior model of AI that is not an unreliable bullshit generator. Easy to say that…
Stuff moves quickly though. I’m reminded of all the “lol AI can’t draw hands” jokes when AI generated renderings hit the scene and they already feel dated.
I’m not terribly invested in LLMs as I just don’t have much concrete need for them and don’t find them super entertaining on their own as a fuck around for fun style application. But I recognize that any impressions made “when it appeared” are going to be flawed in 2025.
Part of the problem is that people say ‘sentence completion’ and it seems like it reduces the action of the thing to the level of T9 predictive text, when in fact, ‘sentence completion’ done right, is a really big deal (it’s more or less how human babies learn to speak and think, by interacting with, copying and emulating the way adult humans already speak as a result of thinking).
Sentences contain and embody concepts and instructions and queries and logic - in order to get really good at learning sentence completion, aquisition of concepts and such just happens along the way.
The possibly-disturbing implication for us fleshies is that maybe we’re not actually as smart as we think we are - that maybe aspects of our intelligence are encoded in the structure and rules of our language - in maybe the same way that your ability to navigate to a destination is, sure, in part a result of your skill at driving, but also in part, just a feature of the roads.
I think it probably is easier to check a stated fact than to research it from scratch, but the big issue is that a lot of people consulting LLMs are doing so with no intention to verify what it says, because that’s effort and the very reason they are asking a chatbot in the first place, is aversion to effort.
That’s an astute point and I know that a lot of research has been done in this area. But I was vague on the details … so I asked ChatGPT! 
Maybe the skeptics here might want to comment on the performance of this alleged “bullshit generator” and point out the bullshit parts! (Maybe there are some, I’m not an expert in linguistics, but I’ve certainly heard of Chomsky’s work on the theory of universal grammar.)
There have been studies done focused on the idea that logic and intelligence are part of the inherent structure of language. Is there a name for this theory and who have been its major proponents?
ChatGPT:
Yes, the idea that logic and intelligence are part of the inherent structure of language has been explored in various ways across philosophy, linguistics, and cognitive science. While there’s no single, universally agreed-upon name for this theory, it’s often associated with a few overlapping concepts and traditions. Here are the main ones:
1. Rationalist Linguistics / Nativism (Chomskyan tradition)
- Main Idea: Language is not just a cultural artifact, but rooted in the biological structure of the human mind. The structure of language reflects deep, innate logical and cognitive capacities.
- Key Proponent: Noam Chomsky
- Key Concepts: Universal Grammar, generative grammar
- Chomsky argued that the ability to acquire and use language is innate, and that language reflects deep logical structures, like recursion and hierarchical organization.
- He emphasized the rationalist tradition (e.g., Descartes) that sees thought and reasoning as prior to language, but expressed and structured through it.
2. Logical Grammar / Categorial Grammar
- Main Idea: Language has a formal structure that can be mapped onto logical forms.
- Key Proponents:
- Richard Montague (Montague Grammar)
- Tried to show that natural language can be treated with the same rigor as formal logic.
- Bridged the gap between formal logic and natural language semantics.
- Ajdukiewicz, Bar-Hillel – earlier proponents of categorial grammar, which ties linguistic categories directly to logical types.
- Richard Montague (Montague Grammar)
3. Linguistic Relativity vs. Linguistic Universals
- Some thinkers argue the structure of language reflects universal cognitive patterns, which are tied to reasoning.
- Others suggest language shapes thought (Sapir-Whorf Hypothesis), though that’s slightly different from saying that logic is inherent in language.
4. Cognitive Linguistics & Conceptual Metaphor Theory
- Main Idea: Language encodes how we structure and reason about the world.
- Key Proponents:
- George Lakoff and Mark Johnson
- In Metaphors We Live By, they argue that metaphors are not just poetic, but reflect underlying conceptual structures used in reasoning.
- Lakoff also worked on the idea that mathematics and logic are cognitive constructs, deeply tied to language.
- George Lakoff and Mark Johnson
If you’re looking for a single name, Montague Grammar is probably the closest explicit theory that formalizes the idea of logic being inherent to language structure, though Chomsky’s Universal Grammar also touches the core idea from a more biological and cognitive stance.
Let me know if you want to dive into any one of these more deeply!
Human babies are also motivated to create a social relationship with the people they are having a conversation with. LLMs have no such motivation.
All of the LLMs are going to be sold to the highest bidder and rich people will be able to scrub, shine and polish the LLMs to give the answer they have bought or enabled at the point of a gun. And you won’t have anywhere else to go for knowledge; LLMs will be rewriting Wikipedia or any other easily available knowledge source. That’s the actual future.
Are you under the very strange impression that the wealthy and powerful have not historically been able to manipulate and censor information? Cite: any authoritarian dictatorship throughout history, or present-day America. LLMs, because they have access to such enormously rich and varied sources of information, may be the best tools yet for enabling freedom of information. They could certainly be abused in the hands of an evil power, but so could absolutely everything else.
This part was probably written by a Alan Blackwell, a human, and I find it quite clever:
AI literally produces bullshit . (Do I mean “literally”? My friends complain that I take everything literally, but I’m not a kleptomaniac).
Probably true. I was annoyed by the hype, and so I deliberately set out to test the limitations and ‘break’ it. Which was not difficult.
No doubt it has improved. And I don’t see any real problem in using it as a source of information that you might not otherwise come across. Just like Google or Wikipedia… it is ‘scraping’ from more sources than anyone could read in a lifetime.
What DOES worry me is that some people seem to just take the result as the ‘answer’, without using any other sources to check it. This seems seductively easy since these systems produce language that is grammatically correct and tends to sound ‘authoritative’.
I agree, this is absolutely true. People need to understand the limitations, and I’ve recently seen disclaimers to that effect in ChatGPT.
But out of curiosity, as one of the LLM skeptics here, what’s your reaction to my previous post with the response from ChatGPT when I asked how language might actually model intelligence?
Keeping in mind that the pragmatic question here shouldn’t be the philosophically murky question of “how do we assess ‘true’ intelligence”, but rather, “is this useful”?
Bullshit is not necessarily false. Bullshit is what you get when the speaker isn’t particularly concerned about truth. All of what GPT says is bullshit. Most of it is true, but it’s still bullshit.
No, it wasn’t difficult in Nov 2022, but I still found the whole thing abso-fucking-lutely amazing. I mean, no shit it’s not gonna be perfect in its infancy. I didn’t set out to “break it” back then–I set out to find out what can it do well, and what does it suck at. And even back then I found lots of useful applications for it that I implemented in my life. What it does today is basically magic to me. I use it daily to help me navigate through my Logic software. On Tuesday it helped me figure out how to fix a broken Bissel rug shampooer after I sent it a couple pictures of it. There’s a whole thread of uses I participated in on this board, so I won’t rehash them. It’s plenty useful now even if it never gets any better.
Perhaps it helps to think of ChatGPT and other LLMs as basically an enormously upgraded search engine. Search engines are extremely helpful, even indispensable, but no one would take the search results verbatim without some intelligent scrutiny.
No, because a search engine isn’t a source for answers. It points you toward things that are sources for answers, and then you can evaluate those sources. If I ask a question about Mars, say, and a search engine directs me to something from NASA, I know I can trust it, but if it directs me to some crackpot webpage, maybe not. But if an LLM answers my question, I don’t know whether it got its information from NASA or from the crackpot.
I can’t immediately find the quote, but there is an interesting blog by Steven Wolfram talking about LLMs where he says that perhaps human language is not really as complex as we think it is?
You simply ask it. When will people realize you can ask LLMs for citations? (And then do check those citations. Often they are on point. Sometimes they are not. But they significantly hasten my research speed for anything too complex for a normal search engine.)