People using AI to code software that they have no idea how it works or whether it has security holes or scaling problems. Sloppy work like this tends not to survive long in the wild, but some will slip through the cracks, and in a year or two there’ll be a booming business in rewriting them (either that or screaming for MORE AI to fix it).
Entrenchment of ignorant middle management. Currently MM is a greasy layer of bullshitters that infest every business. AI will help them appear smarter, and there’ll be an arms-race of middle-management bullshitters leaning on AI to support their dumb decisions. “Hey VP, AI says we should do XYZ, you’re not going to argue with the multi-million smartypants machine you just bought, are you?
Reduction of standards of work. If I can accept that AI cuts the quality of my service in half, but cuts its cost by 10x, then mediocrity will become the new standard.
Companies are spending so much on AI, expecting it will be a “new way of working”, that the use of AI itself will become a rateable performance task, moreso than what you actually create.
I’ll lean into that last point. My current employer actually has a rubric of AI use that’s used for performance evaluations for the past 2 years. We are to use AI for everything we possibly can. We are to use AI to find efficiencies. We are to surface new ways of using AI in our work and to integrate it into our products. We are to evangelize AI use internally. We are measured on our use of AI tools. We are penalized for expressing skepticism about AI or disparaging it in any way.
There’s an unwritten rule that we’re not to complain about AI for layoffs, because we build an AI product that loudly promises to increase productivity. Our customers look to our own press releases as proof, so layoffs are now effectively part of our product offering. And indeed we’ve laid off around 6% of our workforce per year for the past 3 years, keeping our pre-existing core products in a maintenance pause, supported by incredibly overworked skeleton crews, facing more and more outages, while our stock is at an all-time high, as is our exec compensation.
AI will be an enabler for horrible social relations that already existed, but will now become even more intense. I am a lot less worried about real hyper-intelligence than the weaponization of pseudo-intelligence in the hands of idiots and bullshitters.
I would challenge point 2 slightly. There is a tendency to lump all middle management as a useless series of layers of bureaucratic bullshit. And much of it is. But some degree of middle management is necessary for the functioning of a company for the same reason layers of officers and NCOs is necessary for the functioning of an army. To your point, AI poses a risk of greater enshitification of the middle management layer by replacing human (and thus accountable) interactions.
But other than that I think everything you said is spot on. Really I could sum it all up in that the main risk of AI is that it is an unsustainable bubble of enshitification. I think AGI is becoming like self driving cars and cold fusion. Promising technology that continues to show remarkable breakthroughs but other than a few specific use cases under controlled conditions, always remains “just 5 years away” from being reliable and scalable to actually change society.
Some would say we are in a recession already and it’s only the irrational spending on all things AI related that is masking that. But for all that spending, I don’t think we are seeing any results. Companies don’t seem to be producing better products or services thank to AI. In many cases, they are much worse.
And as I understand it OpenAI and some of these other companies aren’t turning a profit and the costs to build and power all these data centers hasn’t been passed on to the customer yet.
Maybe if all costs are included? Meaning not just the cost of producing that widget (be it a good or service) but the cost of poor quality of that widget to the company over time. Replacement of product, loss of business, lawsuit …
The importance of widget quality is nonlinear. If use of AI keeps it over a “good enough” threshold then decreased quality may be worth it for increased throughput, there is a huge market for good enough that costs less, but, and this varies by widget, the cost of being below a threshold can increase exponentially.
I work for a nonprofit with about 50 employees. We recently had a work presentation on, of all things, the terrible book Who Moved my Cheese? It turned into a sort of pressure campaign to start using AI. The presenter also revealed that, thanks to AI, she spent only 15 minutes on the presentation she was giving us - and let me tell you, it showed. Because I’ve listened to a hilarious in-depth pillorying of that book and her presentation barely mentioned the book at all (which I suppose I should be grateful for.)
I tried using Microsoft co-pilot yesterday - my first time using anything like that. All I wanted to do was re-draw a table, but for some reason Excel wouldn’t let me clear the formatting to redraw the table. Notably, you can’t command copilot to just change stuff around in Excel - it has to give you instructions and work-arounds. All in all, Copilot took 20 minutes of my time to ruin my Excel spreadsheet. It couldn’t have done a worse job.
I’m planning to try out Chat GPT soon, to develop a few SOPs, but I’m not optimistic.
I think people are slowly starting to realize what a con this all is. All the companies had to say was, “This product can cut down on your stupid administrative work,” and people might have had a sensible response, but they went and promised the moon.
The big problem here is that if AI starts doing tasks we can’t possibly comprehend, we have no idea whether it’s correct or not. And we have no reason based on existing models to believe that it will be trustworthy-- or even competent.
Real AI would be much faster than institutions. Unlike a corporation or bureaucracy, it could screw up massively faster than a human overseer could react.
An AI replacement for such things would theoretically have no human qualms or restraint. Tell a human organization to kill 90% of the population and there’s a good chance they’ll balk; AI wouldn’t care.
Related to the above; an AI replacement for corporations and bureaucracy would lack human judgement. Even if they theoretically could, the people going for them want perfect servants, not something that’ll tell them “no”. So when told to do something stupid or insane, they’ll just do it without question.
As for markets; once such AI was perfected there wouldn’t be a market anymore. Just God-Kings and their AI armies and servants. The rest of the population would be a “resource drain” and exterminated as such. Or at least that would be the ideal outcome as far as the techbros & fascists pushing so hard for AI are concerned.
The answer to this is “just build another AI to review it”. As in, that’s the approach that’s already been decided and is being built.
To me this is the aspect of AI that actually has the potential to suck up all the energy and boil the ocean, not whether someone’s creating a .gif of Taylor Swift with 4 boobs. The latter sort of thing has a cost that can be directly imposed on the requestor.
But once we get into this doom loop of pushing out massive numbers of AI slop apps, and finding they have tons of security holes, and then building another to check it, and then building another AI to check the checker, this is where we may get stuck in a rut of having no choice but to add more compute, because all the risks will already be in the wild, at a scale where simply undeploying the apps won’t be an option. I think that’s the thing that’s going to pull every last flake of coal out of the ground.
Again this is a consequence of mediocrity becoming the norm, because pushing out something that seems high-impact and low-cost will usually override concerns about whether it provides enduring security or value.
And then you realize that actual GI runs pretty well on a couple of Twinkies, Coca Cola, and possibly a little cocaine, mushrooms, or some other drug of choice if you want a bit more randomness in the output.
Unless you kill lots of people and get on the news, the cost of lawsuits is much less than your increased profit due to low quality. I was on a call with the head of quality for the AT&T computer division. (My boss used to have that job.) He stated, right out in the open, that their business model was poor quality and high prices because people (right after the first AT&T breakup) associated AT&T with high quality.
Companies are definitely going to accept mistakes when you can get rid of lots of people.
Man is worried about AI taking their job.
Mankind is worried about AGI taking theirs – the job of consuming the Earth’s resources until it is destroyed by climate change.
Again depends on the widget. AI doesn’t change the basic market dynamic here.
Which ties in to this comment -
And the referenced article.
The article makes the point that for their academic definition of “governance”, markets, bureaucracies, democracies, and LLM AIs, all qualify. Not that they are all artificial.
And well duh. Of course they are all means by which aggregates process information and make decisions, not by necessity with that level having sentient awareness of the choices being made, not by any single individual having control. “The invisible hand” making the decision regarding costs vs quality not with any sentient awareness, so on. AIs of course are similar in that.
As I read the article the idea is both that these LLM AI systems are forms of governance and tools for governance:
By treating AI as a technology of governance, we can ask how it changes the markets, bureaucratic processes, and forms of democratic representation that reshaped the world during the Industrial Revolution. By surmising, alternatively, that it might become a form of governance, we can ask whether it may have transformative consequences in its own right.
We are painfully aware I think that all forms can be examples of good governance, and of bad governance. As tools of the forms of governance that are markets, bureaucracies, and democracies, they can and will be exploited for both, potentially amplifiers and accelerators of current processes and distortions.
My point was that corporate America (and the corporate world) will do things that reduce quality and increase risk if they think it will make money. That definitely includes rolling out crap AI that will often give wrong or even dangerous answers.
As for the rest, if you look at things in terms of governance, I’m sure you can cast AI as a form of governance. I’m not seeing it, but that’s from lack of trying. As a computer scientist I think of things in terms of search space heuristic, where you make decisions based on some data or training and come up with a locally optimal solution. LLMs can be seen that way. So can evolution.
No less and no more true with AI making the widgets than a person. The governance of the mythical free market being whether or not it makes money with the invisible hand taking it from there. Below a threshold of quality sales go down nonlinearly and profit decreases even at a very low per widget cost.
The linked article references such explicitly, how much slop is tolerable, and the answer being, as you’d guess, lots.
Slop—prediction and categorization errors—is inevitable. AI algorithms are imperfect means for fitting curves to the complex processes they look to summarize. The owners of platform companies often care most about exploiting economies of scale and are relatively insensitive to local mistakes. Amazon uses AI to hire and fire its fleet of self-employed drivers, on the basis of classification decisions that may appear arbitrary to humans. As one of the engineers who designed the system explained in vulgar terms to Bloomberg (Soper 2021), “Executives knew this was gonna shit the bed. That’s actually how they put it in meetings. The only question was how much poo we wanted there to be.” More generally, engineers often view themselves as part of a “we” designing and implementing the systems rather than the “they” whom these systems are deployed to categorize, manage, and predict (Eliassi-Rad 2024).
And the article was written from the perspective of a political scientist! And the thesis is that such a perspective is more useful (say to discussions like this one) than is generally appreciated:
Political scientists can usefully think of AI in terms of its relationship to governance arrangements, the various large-scale collective means through which humans process information and make decisions, such as markets, bureaucracy, and democracy. Governance arrangements do not think (Arkoudas 2023, Mitchell 2023, Nezhurina et al. 2024), but they usefully reorganize human knowledge and activity. So too does AI (Yiu et al. 2024).
In addition to asking how AI ought to be governed (Stanger et al. 2024), we can think of AI as a technology of governance. AI provides new means for implementing the basic tasks (such as classification or categorization) that markets, bureaucracy, and democracy rely on. Some kinds of AI, such as LLMs, may become sufficiently distinctive and important to be a form of governance in their own right, an independent collective means of information processing and coordination, equivalent to markets, bureaucracy, and democracy, rather than a mere technological adjunct.
The concerns of AI being misused as a tool of governance, of it supplanting other forms of governance, are perhaps more in the realm of political science than of computer science?
The difference is if people assume that information coming from a computer is accurate - kind of like how they assumes products coming from AT&T were high quality. In the ‘50s and early ‘60s there was a trope about the computer person saying the computer was producing perfect results, but then it screwed up in obvious ways. HAL9000 and “The Ultimate Computer” episode of Star Trek are two late examples. Perhaps we need more obvious AI screw ups. You’d think the ones with lawyers would do it, but obviously not.
The problem with LLMs is not that they aren’t perfect, they can’t be. It is that their users think they are perfect and the management decides not to spend money on checking their results. It is much harder to justify expenses to prevent problems rather than ones to fix problems. I’ve done this, the budget of my group depended on it.
As for governance, I guess if all you have is a hammer … I wonder how far they would apply it. Automated HR systems reject or pass through resumes based on keywords for years now. (That’s not firing, true.) Is that governance?
I don’t think better quality being better has ever been true. There are plenty of other factors. Perceived quality might be more profitable than real quality in the short run. Brands that were great 40 years ago are crap today. So, AI quality is just like other quality.
My experience and knowledge is of course limited, but my impression is that most users are very aware that the output is imperfect. I heard one analysis of programmers who were high use users of AI to help with their programming and who were assessing that it was making them significantly more productive - but when productivity was actually measured it was actually significantly less, because they were doublechecking the AI’s work that much. Overall I think people expect generic from AI, but efficient generic.
A big question is not if it can be perfect, but can it in specific use cases be better than generic, better than average. Using it for military applications for example, giving a kill order to a drone … expecting it to be perfect is impossible … humans aren’t … the issue is if it can be less imperfect than human operators are but faster and cheaper. The scary part is if not, how much additional imperfection will be tolerated for speed and cost, for power amplification?
A military example makes the process being paid for poorer quality clear. And some poorer quality would be tolerated for much greater … efficiency … but there would be a threshold of harm that efficiency would not justify.
To the writer of that article, definitely. Bureaucracy is governance. And that is using automation as a tool of bureaucracy.
I totally accept that people “know” that AI results are imperfect. The LLM will even tell you that. But when management (governance) decides that the speed of LLMs allows them to pull in deadlines and expect more output. When that happens, people will stop checking. And they will usually get away with it. Until they don’t.
Here’s an example from the Times today. At MIT a professor split some students into three groups, and asked them to write a 500 - 1,000 word essay. One group could use ChatGPT, one could use Google, and one had to write it from their stored knowledge. The ChatGPT essays were far worse than then other two. Then, just a minute after the students turned in the essays, they were asked to quote a sentence or two from it. The ChatGPT users could not remember even one sentence they wrote. The Google users could remember a few, but the non-computer users could quote far more, in one case quoting the entire essay.
I bet the first group finished faster (the article didn’t say.) I suspect the programmers you mention had a better idea of the potential problems than most people, which is why they checked. But how much are they going to check when the deadline is looming. Especially if there is not a lot of new code, and the LLM usually gets it right.
I used ChatGPT for the first time today to develop a couple internal policy documents for the federal grant application process. I’ve been pressured to use AI at work so this seemed like a low stakes way to start. I’ve been avoiding this task mostly because I didn’t know how to structure it.
I have a moderate degree of knowledge in the field of federal grants management. What it gave me was logically structured, and it didn’t miss anything that I thought was really critical. The only modification I requested it make was a process for identifying and vetting subrecipients.
It was really helpful to have a structure to use - this was kind of what I needed to get over the hump. My biggest issue is that the word document it generated was a formatting mess and Word wouldn’t let me correct it the way I wanted to.
But of course now I have to spend a good deal of time formatting the document and fixing the formatting problems, re-organizing and tweaking and making damned sure it remembered every critical piece of the process. This wouldn’t work at all without my knowledge.
Oh, I also liked that it gave me ideas for other documents I need to have on file, like a pre-award risk assessment for subrecipients. It got me thinking a little creatively.
I’m not entirely sure if it’s saving time or not. Arguably it’s better, but not like, life-changingly better. It still can’t do my job for me. The primary benefit I got from it was eliminating the friction between thinking about the task and actually doing it. This has been on my to-do list for years.
Also, I know people are going be impressed with this and it feels kinda shitty to feel like I can’t take much credit for it.
It’s a mixed bag, really. And not anywhere close to ready to replace my job.