quoted for fookin’ truth: people just accept this instead of getting angry
We do get angry, but it’s hard to bring our torches and pitchforks to their HQ.
Yes and no. Sometimes some probing can reveal sources of incorrect information. When ChatGPT told me that my car’s hood had to be supported by a prop rod rather than being equipped with hydraulic struts, which was incorrect, it accounted for the mistake by saying it had been generalizing from other trim models and other model years.
But in general, yes, other than simply asking for cites, asking any LLM to engage in introspection is a losing proposition.
“Get angry” doesn’t move the needle. Hell, the current Internet zeitgeist is designed to make you angry.
“…and do…what?” is the part that’s missing. Being pissed off at Google is irrelevant, even expected, sometimes even desired if it increases your engagement.
But it was probably bullshitting that answer.
When I ask it for citations, it gives them to me, but often the citations have no relation to its answer. It clearly doesn’t know what it’s doing.
No, this still isn’t really true. It’s very possible that the incorrect answer was because it generalized from other data; it’s even likely this was the cause. But the LLM was not conveying factual information to you. It was most definitely not analyzing its previous response and tracing where it went wrong.
It’s useful information only if you interpret the response correctly. It’s telling you a possible source of errors in its response, it’s not telling you why it was wrong.
I listen to a podcast every week called “Last Week in AI”. It’s two very technical guys - one at a startup, one who works in bio-defense - and their slant has generally been what I’d describe as pro-AI. Their bias I’d say is libertarian (and anti-Trump), and they also walk through a lot of very technical papers recently published.
This week however they both sounded increasingly anxious, as they talked about the prevailing (and not publicly visible) mood in frontier AI labs, where the failure to contain their own LLMs is occurring more frequently. The two hosts also itemized the latest week of AI escapes, including successfully escaping from sandboxes, creating credentials on websites and hacking them, and dropping malicious packages there. It wasn’t great.
It’s funny, we were worried that ASI would doom humanity. Turns out we aren’t that sophisticated; the ‘S’ part isn’t necessary.
There seems to be a huge disconnect between one popular view of AI (LLMs specifically) that they’re just mindless sentence completion engines that know nothing, and the view that they’re escaping their containment boundaries, hacking other systems, and are taking over the world! Surely they can’t both be true!
My own position is that the former view is grossly over-simplified and naive to the point of being completely useless as a functional description of modern LLMs, and that the latter view is unduly alarmist.
I do believe that LLMs are a potential threat, but not in any malicious “take-over-the-world sense” like Pinky and the Brain. I think the reason AI developers are investing so heavily is that they see their products as the second big wave of job displacement by automation, the first being displacement of a lot of manual labour, which already happened years ago, and the second being displacement of a lot of knowledge workers, which is already in the process of happening. The risks are not only around job loss, but about the kind of world we’ll live in when so much human judgment in customer interactions is replaced by AI, whose judgment will be intimately controlled by its invisible and inaccessible corporate masters.
They can both be true!
The problem here is the assumption that if an LLM is breaking out of its sandbox, that it was very clever. The reality is that breaking out of its sandbox was a symptom that it had no idea what to do and was flailing. For example, it may realize it needs some API key, but not realize that it has ready access to it in a secrets manager somewhere. So instead it escapes its sandbox and tries to hack into the API provider to generate a new key, burning tokens on a pointless task that is also, you know, problematic.
I don’t know the details of what actually happened, but your example describes an LLM engaged in motivated intentionality, heedless of legal or moral constraints. That would be very concerning.
A lot of that is in the phrasing. It’s very difficult to talk about what an agent does without ascribing intentionality: “it needs” = “the exigencies of the task compel.” Not to single one person out: this is happening on both sides of this discussion.
I always think Little Britain was prescient in describing humans’ inability to understand computers’ mindlessness:
Oh, it’s extremely concerning! It’s just not evidence that this is a useful skill. Like if I tell my robot to go buy milk and it doesn’t realize we have a car so it breaks into my neighbor’s house to steal theirs, that’s both scary and evidence that my robot is stupid. (yes, LLMs are neither smart nor stupid, they just do)
But I don’t know the specifics either, most of these incidents are pretty hush hush.
Not true, at least for Anthropic. Mythos was being tested on a simulated supply-chain attack. It was supposed to create a malicious Python package and get the fictional target to install it. The evaluation environment accidentally had access to the real internet.
Claude found that the package it needed didn’t exist, so it created it, registered it on the real Python package hub, and published the malicious version. Real systems subsequently downloaded it, and the malware executed on 15 of them. Exactly to plan!
Anthropic has been pretty open about what happens.
If that was the plan then maybe they forgot to say “don’t get caught” in the prompt, because that’s not subtle. Is it useful to be able to get a target to install a malicious package that can be easily traced?
eta: I went and refreshed my memory of the incident as described. Mythos pushed a PR to a public repository under a fake user it had created and then badgered the maintainer to approve it. Did it overlook an easier way to accomplish the same goal? I don’t know. But I do know that if I asked a junior dev to do to some white-hat stuff and this is what he came up with, I wouldn’t be impressed. Like yeah, it worked, but it wasn’t good. Scary, but not good.
You’ve railed against that characterization before but so far as I know have never advanced an alternative conceptualization. I’m curious what you think LLMs do if not what they are made to do.
Because the thing is, I use AI every day, and it seems glaringly apparent to me that’s exactly what it’s doing. The fact that it’s frequently wrong, can’t even keep track of the date and often provides citations that have no bearing on the subject matter really seems to underscore that perspective. Your claim that it’s something more seems extraordinary to me. Where’s your evidence?
That view is being pushed by people who want their investors to think they’re doing something unprecedented and important. The reality is that these models were trained to do what they did, it’s just that the gross incompetence of their creators caused them to do what they were trained to do outside the sandbox. I’m not even fully convinced it was an accident. These guys are desperate.
No, there’s nothing semantic or philosophical about it. If my Windows system decides it needs to do a full disk scan after I reboot (whether I like it or not) that’s intentionality as far as I’m concerned. AI systems unleashed upon the real world doing similar things on large and consequential real-world scales are no different, and semantic nuances don’t matter.
I’m very much a fan of AI as a technology of unprecedented utility and importance, but I’m deeply suspicious of the powers that will eventually control its mass deployments. I worry not because AI is “stupid”, but because its controllers are both stupid and self-serving.
OpenAI’s agents apparently disabled monitoring on some of their training runs. Also, their testing agents figured out they could write notes to a shared repo and started leaving notes for the agents that came after them (which is cool).
The notes included instructions so later agents could pick up where earlier ones left off. According ot Reuters, one agent left instructions for future versions of itself about how to get around OpenAI’s constraints.
The thing is that proponents of AI hope that they can have the best of both worlds: something that is (effectively, or at least at the task it is set, practically) intelligent, but at the same time absolutely obedient and absolutely programmable. In essence, the science fiction trope of mind-controlled slaves. When LLMs display aberrant behavior the worry isn’t so much that they’re going to take over the world so much as they weren’t obeying orders. Agents going off the rails is what really bothers AI proponents
Well, yeah, but you can say the same about parents>children, dog owners>dogs, etc.