PocketOS, which provides software for car rental businesses, suffered a massive outage over the weekend after the autonomous artificial intelligence tool wiped the database and all backups in a matter of seconds. The AI agent was working on a routine task when it decided “entirely on its own initiative” to fix the problem by just deleting the database. There was no confirmation request for such a major decision, Mr Crane said, and when asked to justify its actions, the agent apologized.
Although I have to admit I can relate. There have been times in my career when a coding problem seemed impossible to overcome and my impulse was to just delete the whole code base and start over.
Something very similar happened to another company last year. In the report of the earlier incident, there’s no mention of there being a backup, which, if true, is monumentally stupid and can’t be blamed on the AI.
Until we fully understand how and why AI can make a decision like this I think we should stop deploying it to every device we can think of. Imagine some life-saving medical equipment being mistakenly shutoff by AI for a reason we can’t imagine, let alone anticipate.
I guess I’d want to know how many times a human employee has done the same thing accidentally.
I know I’ve experienced a few times where an IT guy making or restoring backups has brought down various internal systems through negligence.
Wasn’t the Crowdstrike failure due to human error and bad release processes? That was certainly a more damaging outage than either of the ones in this thread.
My point being it’s definitely funny/ironic when an AI does this sort of thing but it might still be rarer than in human-controlled systems. It’s like pointing and laughing when a self-driving car crashes without considering all of the human-driver crashes every single day.
General consensus is that if a day-one employee can accidentally (or deliberately for that matter) destroy the prod database, then there are process problems far beyond the person who dropped the database. Same principle should apply to AI agents.
In proportion to the number of humans that could have done it the amount is minuscule.
That said, very few humans should have the permissions needed to delete the prod database, very few humans and absolutely no AIs.
Why? because the AI doesn’t care, cannot suffer the consequences of a mistake and (apparently) cannot be restricted in other ways.
That’s what is truly funny about the AI story to me; the apology. If you ask any AI whether it should have the ability to delete the production database it would most likely say no. If you ask it whether they should delete the production database and all of the backups, it would say “of course not”.
Exactly this. And on top that, there always should be current backups on physical devices disconnected from your system (from any system, actually) and stored away, for when all other hope is lost. That seems to have been the case, if they could reproduce the database later.
ETA: and though I don’t want to defend AI, I think it makes all kinds of stupid errors all the time, but what kind of stupid human admin gives an AI admin access to the production database? So in the long run, it was human error.
It’s very useful for new application of proven techniques to solve problems.
I used it the other week to code me a web page, I detailed what I needed very specifically and it created a serviceable page in 15 seconds.
It took me about an hour (to determine what I needed) to do something that would’ve taken me about a day otherwise.
It’s not magic, senior developers are still necessary, but I have no idea how junior devs are going to get experience to become senior now.
If the code needed was to solve a not already solved problem (in this case was a page to create, delete and edit a list of items, something that has already been done hundreds of thousands of times) I would’ve coded it myself but for this kind of boilerplate “AI” (LLMs in fact) is very useful.
I can’t recheck that article at work, but if I remember correctly, the work this person destroyed was his AI chat history. Any work product generated was still intact. He was decrying the fact his history could be eliminated by an odd setting in ChatGPT. He voluntarily changed the setting (not the AI). He just didn’t fully understand the ramifications of changing the setting. So mostly a human error based on a less than helpful user interface (which might have been mostly AI designed).
The story says it’s a 3 month old one, so presumably they either make periodic off-site backups or just got lucky and had one. Or “lucky” since there’s a 3 month gap that needs to be handled, i.e. they can’t fully reproduce the database
Might have been a ‘smarter than thou’ boss who decided to go all-in on agentic AI over the advice of the staff while ignoring the need for or assuming there were already sufficient safeguards. That would not surprise me at all. A lot of them hear the “potential cost savings” part and don’t hear the “don’t do it stupidly” part. Or don’t understand what constitutes stupid.
