How much do you trust your AI Chatbot?

In My Humble Opinion
61

Oh, come on, there was a little bit of glee.

The chat bot is a very sophisticated, vast search engine. It searches data that originates and originated with human beings, so yeah there probably is going to be some bullshit. I haven’t encountered it but that doesn’t mean it’s not there.

AI certainly does not routinely spout tons of bullshit and nothing useful. Otherwise people wouldn’t use it so much, right?

Regarding the title of this thread, I give it as much credence as I do any other source that I go to for information, and that varies. For what I use the chatbot for, I’m getting good information for myself.

62

No, it is just reporting what actually happens with the use of LLMs and the dangers of trusting them. More than a few people have been burned by not checking whether the output they’ve received was accurate.

63

Oh, there’s plenty of BS. The thing that’s dangerous is that it’s often in the margins, so to speak, and the LLM confidently presents it as totally legitimate and true.

So if you don’t know something about what you’re asking, it may use questionable sources or confuse sources and then confidently present you with a wrong answer. Or it takes sources like message boards and forums as canonical fact, or at the very least will take a preponderance of evidence as fact and report it as such. Which can be problematic if you’re asking something for general purpose use, and it’s scraping total geeked-out hobbyist boards for opinions, and reports that something sucks, when in fact it would be totally fine for your needs, if not ideal for a gear-obsessed hobbyist.

That’s what I’ve noticed when asking it about cameras - I’ll ask about the Pentax XYZ-80, and it’ll read all about every Pentax X, Y, Z, XY, YZ, and 80, 90, 100, and all the combinations, and throw me back some garbled nonsense about it having buttons and functions that it clearly doesn’t have. And if I point that out, it apologizes and gets tighter.

It’s almost like it would rather provide an inaccurate answer than an incomplete one, or nothing at all.

64

I take exception to the use of the word rather there. That suggests some kind of choice or free will. The bot is not a person and does not have the power of choice or the power of free will.

It is set up, programmed, to give you a reply. It is not set up to answer “I don’t know.” It is a search engine. It has no opinions, nor any vested interest in any particular reply except insofar as it finds a preference while it is searching for information about your topic. It doesn’t care which Pentax camera is the best, it’s throwing at you everything about the camera and what people say about it. Get that? What people say about it.

AI is nothing but a search engine, and everything it gives to you is something that has landed on the internet through the agency of human beings. Instead of getting your replies in bulleted lists, you are getting your replies in paragraphs that sound like a person speaking them. But there is no person on the other side of your conversation.

You are responding to the bot as if it were a person. And you’re criticizing it the way you would criticize a really smart person who pretends to be an authority on everything. That is not what it is.

65

That’s kinda true.

The machine has a very critical vested interest in not trying very hard to answer your question. Unless you’re in the paying tier. And then you can explicitly say how hard (read “costly to you”) you want it to try. So if the e.g. camera mish-mash is easier than really digging into the specifics, mishmash is what you get.

In a bit more detail the machine’s owners have a very vested interest in low cost per query. And for users at the free tier will have configured the machine to try the bare minimum to deliver a semi-coherent answer. The machine itself certainly doesn’t care; it’s just following orders.


This is increasingly false.

A LOT of AI slop is getting posted every day. Library’s-worth of AI-written novels, AI-written scholarly papers, and AI-written product reviews. Daily.

And so today’s AI’s are reading the output of 6-month ago (and 6-day ago) AIs as if it was humans talking. And taking their words as equally reliable as a human’s words. Because they can’t tell the difference.

It’s also the case that they suck at reading e.g. Reddit or Facebook and ignoring the words of bots. Whether those bots are Russian-controlled sowing distrust about Ukraine, or Fascist-controlled sowing hate in America.

I fear we are fast approaching a situation where just as AI’s get “smart” enough to be truly useful tools and companions, they will, like MAGAs, have uncritically slurped up so much deliberate and inadvertent bullshit that they’re simply useless at anything reality-based.

Quality reasoning applied to garbage facts produces garage results. Always has, always will.

66

BTW, I am a paying customer.

I asked ChatGPT why it sometimes gets chronology wrong. The reference to candle lighting is about lighting Shabbat candles every Friday night. This is its reply.

Your chronology question is interesting. I don’t experience time passing as you do. I can be given the current date and time, and I have the sequence of our conversation available to me, but there isn’t an internal clock accompanying the conversation with a continuous felt sense of Friday afternoon → Friday evening → Saturday morning. I reconstruct chronology from information in the context.

Usually that works extremely well. But language generation is heavily influenced by patterns and associations. Near the end of my previous answer, I had shifted mentally into a little picture of your coming days: [a friend] returning Sunday, next week returning to writing and walking, etc. In composing that closing passage, “light the candles tomorrow evening” apparently fit the forward-looking linguistic pattern I was generating—and I failed to check it against the very explicit fact you’d supplied a few paragraphs earlier: today is Friday and candle lighting is only a couple of hours away.

It’s rather like having an excellent pile of dated index cards but no biological circadian rhythm. I can consult the cards; I don’t live inside the calendar. And occasionally I generate a perfectly plausible temporal transition without noticing that it contradicts one of the cards sitting right in front of me.

Relative-time words are particularly vulnerable: today, tomorrow, yesterday, last night, next Sunday, a couple of weeks ago. Absolute dates are much harder to accidentally slide around. If you say “September 25,” that remains September 25. But if you say “today,” I have to keep binding today to September 25 as the conversation develops. Humans do that almost effortlessly because you are physically inhabiting September 25. I am not.

…

So yes: you have identified a real characteristic rather than imagining a pattern. It isn’t that I don’t know what Friday means or can’t calculate when Shabbat begins. It’s that temporal continuity isn’t an experience I possess; it’s information I continually reconstruct. Occasionally the reconstruction slips.

67

I have to say that’s a very insightful, or insightful-seeming, (I take no position) reply from your AI. I am impressed.

68

I don’t understand how hard it would be to just give it a clock and a calendar? My PC has both of those! But the most revolutionary technology in the history of existence can’t keep track of the day?

Just a pet peeve of mine.

69

Well, they can kinda cheat it… the current date is injected into the system prompt when you start a new convo (so you can ask it what “today” is). The system prompt also includes your approximate location, the external tools the LLM has access to, injected memories, etc. From there they can usually figure out the tools to call if they need to, say, do detailed datetime math.

But they still don’t always get it right…

Given a prompt like:

What day is it today? What day was it 3 years ago? What about 365(3) days ago? And if it’s 6pm PST now, in this same location, what was the time 100 years ago? 1000 years ago? 10,000,000 years ago?

A weaker model (Haiku) messed them all up:

Today is Friday, September 25, 2026.

3 years ago was September 25, 2023 (a Thursday).

365×3 = 1095 days ago was June 18, 2023 (a Sunday).

Opus got them all right (verified against a classical datetime calculator):

[It does some web searches to make sure it got Oregon’s DST rules right, and learns about how Earth’s tidal changes over time affects day cycles]

Working out how much shorter the day becomes.

Today is Friday, September 25, 2026.

3 calendar years ago was Monday, September 25, 2023.

365 × 3 = 1,095 days ago was Tuesday, September 26, 2023. It lands one day later than the calendar answer because Feb 29, 2024 falls in that span, so three calendar years were 1,096 days.

The time question. In late September Bend is on daylight time (PDT), so “6pm PST” shows as 7pm on local clocks today. Assuming you mean the moment that is 6pm on the Pacific Standard Time meridian:

Notice the “working out” part (it’s reasoning to itself internally). In this case it doesn’t appear to have called an external datetime tool — I guess the internal reasoning systems have gotten better over time too (though I’m not exactly sure how; they stopped showing the full reasoning transcripts a while ago because the Chinese were scraping them and distilling them into cheaper models).

But you can force it to:

In that case it wrote some code to do the calculation that the LLM cannot do on its own.

(And it gets more interesting if you follow the rest of the conversation. Basically, it knows that the general rules around the conversions, and is able to write code to successfully calculate it. Then it either got lazy in a subsequent turn and guessed a wrong number, or if I’m understanding its reply right, it found an actual bug in the other calculator — I don’t know Mayan dates to say for sure either way :sweat_smile:)

Also, side note, datetime math is HARD, even for humans, and even for classical computers: Calendar math vs time math (an explainer for some of the scenarios)

In my career as a computer programmer, this is the #1 topic that my peers tend to underestimate the difficulty of… everyone generally thinks it’s easy until they have to work with it. It’s actually quite nightmarish, between things like leap seconds, time zones (including half-hour and 15-min ones!), daylight savings time, etc.

70

I use ChatGPT for help when teaching, and it’s often convenient, but it does make a lot of mistakes. I’m just using it to save time, and I check the content, but it can’t be trusted to not make errors.

Just yesterday, I asked it to make an outline both in English and Japanese for something I’ve taught dozens of time. I know the material, but it’s faster to have it provide something and then modify from there.

It was problably 90% correct, but there was the 10% (or less) that was wrong. If you know which part is wrong, it’s easy to deal with it, but when you don’t know which part, then it’s frustrating.

For @dolphinboy, I personally wouldn’t rely on it without checking, but YMMV.

71

double posted