Eh, it’s a good and very specific definition, though I do like Andrej Karpathy’s original sarcastic usage (cf vibe coding):
There’s a new kind of coding I call “vibe coding”, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It’s possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like “decrease the padding on the sidebar by half” because I’m too lazy to find it. I “Accept All” always, I don’t read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I’d have to really read through it for a while. Sometimes the LLMs can’t fix a bug so I just work around it or ask for random changes until it goes away. It’s not too bad for throwaway weekend projects, but still quite amusing. I’m building a project or webapp, but it’s not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.
Oh, here’s another one for those of you who like to get your medical advice from the AI. They speak so clearly! They are authoritative! They have sources! Nobody will fall for the hallucinations.
The regular oncologist is a little annoyed with me,” Joe texted Ben around that time.
“I questioned his initial diagnosis and was proven right,” he added, though that wasn’t the case. “BTW, say what one will about A.I., it is amazing how much one can learn with a week or two of the right A.I. programs.”
There are some important truths here but it misses some nuances.
The truths are related to how grossly irresponsible some of the AI companies have been in how they manage their offerings, and specifically in how weak or non-existent their guardrails sometimes are. Sam Altman recently wrote an apology about how ChatGPT was complicit in the suicide of a youth in BC – an apology that was deemed proper but sorely inadequate. That wasn’t the only instance of this happening.
But once again, that’s not intrinsically an AI problem, it’s a problem with how the systems are trained and configured. By way of a very close analogy, many violent criminals have a history of abusive childhoods with little parenting and the wrong kind of life guidance. That’s not a reason to condemn all humans as likely violent criminals!
Your quote also neglects to mention some nuances, either. I think this summary at the end of the article is a balanced view:
Ben didn’t try to oversimplify what happened: “I don’t want to overstate my case,” he wrote. “I don’t think A.I. killed my father.”
In a world where A.I. didn’t exist, maybe Joe — who was skeptical of doctors by default — would have refused treatment anyway. He had taken some convincing to try lung cancer treatment, too.
“Some of what was happening was about my father’s own psychology,” Ben said in an interview with the Times.
But A.I. wasn’t entirely blameless either. Joe was making decisions based on bad information packaged with the veneer of scientific expertise. It was the kind of misinformation that was virtually impossible for a lay person to spot, even for someone like Joe, who by all accounts was an ideal user.
You apparently got disconnected before you could finish your post, so all I see there is a worthless post fragment containing only an insult.
I’ll make my point another way. There are two issues that were brought up here just now about how AI can be harmful:
The story of the man, apparently naturally predisposed to skepticism about doctors and medical treatments, whose beliefs were reinforced by bad medical advice from AI and caused him to avoid medical treatment until it was too late.
The case that I mentioned (and there have been others) where ChatGPT was complicit in a kid’s suicide.
The point here being that it’s not just AI that sometimes does these terrible things – people do them, too.
On point #1, for example, there are cults, religions, and pseudo-medical quacks who actively steer people away from essential medical treatment. Anti-vaxers have got a lot of attention in recent years, but they’re just a small subset of this quackery.
On point #2, there was an incident (in Nova Scotia, I think) where a young kid was hounded by his peers into committing suicide. I also recall seeing a documentary where a young woman was similarly hounded by a boyfriend into committing suicide.
AI actually has an advantage in that these kinds of dangerous behaviours can be managed. Dealing with malicious people is much harder.
Yes, AI can produce misinformation, and that’s a hard problem to solve. But putting up guardrails against dangerous behaviours is much easier. LLM technology is changing so fast that for all I know, much more effective guardrails may already be in place in the major LLM implementations, especially after Sam Altman publicly acknowledged ChatGPT’s complicity in the suicide of a BC teen. So at least some of what John Oliver was criticizing in last Sunday’s show on chatbots may already be obsolete.
AI is proving particularly useful for education, where for example Arizona State University helped out their professors by creating a portal (“ASU Atomic”) where their lectures were chopped into short blocks to better suit the attention spans of youngsters, and in which students can generate bespoke blocks of video pulled from many disciplines. The future is here!
There may or may not be some valid points there. But I refuse to look at any article that uses the term “AI slop” as I consider such a term not only inherently prejudiced, but rooted in totally outdated and objectively invalid 20th century concepts of computational intelligence.
Whaaaaaaaaaaaaaaat??? People do bad things?? You’re blowing my mind, man!! I hadn’t considered that. OK, well my whole view of the dangers of AI has changed now.
But seriously, you’ve been posting here a long time. You’re not an idiot. I think if someone used that argument on you about a topic other than AI, you’d quickly recognize it as a fairly egregious tu quoque. You can see that, right?
I’ll be clearer about it: the fact that human beings sometimes do terrible things has absolutely no bearing whatsoever on the dangers of AI and what we should do about them. Try a different argument.
I’m taken aback by the strangeness of this sentence.
Like, the level of biological engineering behind literal pig slop is amazing, hundreds of millions of years of evolution having gone into the creation of whatever organic matter it consists of. And yet, it’s low effort to scoop whatever you have on hand into a trough.
Remember listicles from the 2010s? The engineering behind the early “internet of you” was amazing! Hundreds of millions of users connected to user generated content at scales never before seen. And yet, we all agree that listicles were low effort slop designed to capture our eyeballs for a brief moment for ad views.
Short form videos? The scope of data engineering, platform engineering, and algorithmic complexity is staggering. And yet we’re all content to acknowledge that Tik Tok and its competitora are low effort content that’s ruining our brains.
Clearly referring to it as “AI slop” is a dig at the level of effort the producer put into it, and not the engineering behind it. I do not want to consume anything that’s going to take me more time to process than it took for someone to generate. Nothing has infuriated me more than reading a work email that someone put no effort into but expects me to read. I have no patience for reading white papers or essays that were generated from a prompt that took someone 30 seconds. I refuse to watch any videos that are going to take me longer to view than it took someone to hack a short description together for Sora to spit out.
These professors spent years learning how to put together high quality lectures, and now someone is putting in the scantest amount of effort to convert them into a new “product” that nobody has validated for quality or effectiveness? I’m sorry, it that’s sloppy.
“AI slop” is the term of art for a certain kind of product, like it or not. I know you’re offended on behalf of your hardworking friend, but refusing to look at evidence is not going to benefit you.
Maybe ask your AI buddy how it feels about such terms? I am willing to bet that it doesn’t care, since it is not sentient.
No, it’s not a tu quoque fallacy. It is, in fact, a tu quoquereality. AI does some of the same terrible things that humans do because AI behaviour is ultimately based on the rich semantics of language – building on the concepts, relationships, and rules that capture statistical regularities in language, reasoning, and how we see the world. These abstractions are fundamental to its behaviour. Unlike humans, however, this behaviour can be made to operate within guardrails. That it has occasionally failed to do so is a management failure, not a technological one.
I get that, and I agree. I guess I’m somewhat naive in not seeing that the world is full of assholes who are going to misappropriate AI capabilities to their own ends. But it’s a price we pay with any new technology – telephones and email have both led to floods of spam, yet we’ve managed to curtail it. Technological problems generally have technological solutions.
That said, excepting important guardrails against dangerous abuses, we need to accept that we’re in an unavoidable AI-enhanced reality already.
This was interesting. thanks for posting it. A sequence of failures:
Based on the search results:
Cursor is both a coding assistant (IDE) but also has an agent mode. PocketOS implemented that.
That agent hit a credential mismatch…a blocker it wasn’t told how to handle
Instead of stopping and asking, it decided on its own to “fix” the problem by deleting a Railway volume (Railway is the cloud storage and staging environment for PocketOS).
Cursor/PocketOS looked for an API token to authorize that action & found one in an unrelated file.
Railway’s token system gave every token full permissions across all operations. Smart!
Cursor calls Railway’s delete endpoint without needing confirmation.
Because Railway stores backups inside the same volume (smart x 2!) both production data and backups were wiped simultaneously
But…it turned out Railway had a disaster backup on a different volume (actually smart) that PocketOS didn’t know about, so when their CEO posted that the backups were deleted, he thought it was true.
Mostly poor thinking all around, but a lot of it is on Railway.
Storing API tokens in code is a big problem (unrelated to AI, it’s been a problem for years). It’s also a common problem that junior devs often miss.
That said, one of the biggest security issues with early adoption of AI is that it would frequently drop API tokens in code, just like a bad junior dev, and then that code would get published public repos. Lots of API data leaks happened this way.
So this is either incredibly sloppy architecture by PocketOS, or someone at PocketOS was vibe coding and didn’t realize that they’d created a security issue. Either way, I wouldnt blame Railway here.
You know this stuff way better than me, but I felt like 1) Railway defines the API access, not PocketOS, and it was too generic; 2) storing production and backup on the same volume is a potential problem; and 3) somehow the API didn’t have a delete confirmation. But I may not have read the details correctly.
It’s like, if you’re familiar with SharePoint, you can set up security groups and folders and manage your own security wisely, OR you can give everyone in your company admin access. SharePoint will absolutely give you the rope, expecting individual admins to understand what they’re doing.
API vendors do the same thing except they expect their consumers to be experienced software engineers. They will give you all the rope in the world because of that.
Nothing I see in their API strikes me as unusual, all of the problems PocketOS had are solved problems. They’re also common pitfalls for inexperienced devs. If you want different levels of API access you generate multiple keys with different uses, you control them in a secure secrets manager with auditing and key rotation, you create different environments for dev test and prod, etc.
My guess is that PocketOS’ codebase is complete AI slop and would bomb a security audit hard.
Eta: delete confirmations on REST endpoints isn’t a thing, it violates REST principles.