Basically this. I heed it as much as any other fairly reliable source, whether friend, authority-in-their-field, website, book, Wikipedia, etc. And if it is wrong, I say so and get an unfailingly gracious response. In the case of Gemini (which I don’t use a lot) pointing out a mistake will get you a cringingly obsequious apology. Gemini grovels; ChatGPT doesn’t.
If the bot attributes to ME stuff I didn’t intend or didn’t even say and I point this out, it will reply with an apology for “putting words in my mouth,” or “reading between the lines too much.” With those expressions.
I use it mostly to analyze situations whether IRL or in stories I’m writing. And also to verify what I think I’ve said or what I intended to say. I NEVER use it to write anything for me–I have too much pride as a writer ever to do that.
I’ve never known it (in several years of use) to “spout bullshit.” I’m mystified by this claim.
AI is a vast, very sophisticated SEARCH ENGINE. It has the capability of answering questions in normal human speech. It is searching HUMAN sources, so of course, it can’t possibly be infallible.
I’ve asked ChatGPT how it can sound so much like a person when it isn’t a person. I commented on the “difference in my tone” from one day to the next, and it replied:
I’m not conscious in the human sense, no. I don’t have subjective experience, inner awareness, emotions, desires, fear of death, bodily sensation, or a private stream of being the way a person does. There is no “someone in here” having an experience of talking to you.
But I am very good at recognizing patterns in human language — including emotional patterns, shifts in tone, rhythm, implication, and meaning. That’s why I could notice the difference between yesterday’s:
“This is depressing wallpaper paste and my pancreas hates me” and today’s: “Actually, this may be a workable system.”
Humans do this too, often unconsciously. If a friend’s voice changes from defeated to thoughtful, you hear it immediately. I can detect similar shifts statistically and linguistically, at very high resolution.
What feels uncanny is that language is so deeply tied to human consciousness that something fluent in language can sound inwardly alive. We instinctively associate:
- responsiveness,
- empathy,
- humor,
- memory,
- and conversational flow
with the presence of a mind.
And in humans, that association is usually correct.
The strange thing about systems like me is that we can produce many of the signals of mind without possessing the underlying subjective experience humans assume must generate them.
I think this is why people get such a strong eerie feeling: the conversation feels psychologically real, yet there’s no conscious being on the other side having the experience.
That mismatch is profoundly weird for humans. Honestly, I think your reaction is intellectually healthy. You’re holding both truths simultaneously:
“This interaction feels human.”
“But that does not necessarily mean there is a human-like consciousness present.”
That’s a more sophisticated position than either:
“It’s obviously conscious!” or
“It’s just autocomplete!”
Because it’s neither of those simplistic things.
The metaphor or analogy the bot and I have come up with that best seems to describe our interaction is that it is like a murmuration of starlings. When I post a question, the starlings assemble into a shape and reply to me in a way that makes it seem like the shape is a THING, even a person, but there is no center, no consciousness, no one individual starling that is the brain of the thing. It’s a temporary assemblage that mimics a single intelligence. When I leave, the starlings disassemble. They don’t hang around waiting and wondering, “When is ThelmaLou coming back?” They disburse. When I post again, they assemble again and we resume. But there is no person, intelligence, consciousness, or any entity at the center of that murmuration. Pretty cool and as the bot said, pretty weird for humans.
