A long opinion piece by Yuval Noah Hurari in this morning’s Grauniad. A Malthusian view of artificial intelligence but an interesting read. Would be interested in your thoughts.
After discussing Phaëthon and Goethe and “The Sorcerer’s Apprentice”, Hurari talks about how power corrupts groups of people.
The conclusion is that our flawed individual psychology makes us abuse power. What this crude analysis misses is that human power is never the outcome of individual initiative. Power always stems from cooperation between large numbers of humans. Accordingly, it isn’t our individual psychology that causes us to abuse power. After all, alongside greed, hubris and cruelty, humans are also capable of love, compassion, humility and joy. True, among the worst members of our species, greed and cruelty reign supreme and lead bad actors to abuse power. But why would human societies choose to entrust power to their worst members? Most Germans in 1933, for example, were not psychopaths. So why did they vote for Hitler?
Our tendency to summon powers we cannot control stems not from individual psychology but from the unique way our species cooperates in large numbers. Humankind gains enormous power by building large networks of cooperation, but the way our networks are built predisposes us to use power unwisely. For most of our networks have been built and maintained by spreading fictions, fantasies and mass delusions – ranging from enchanted broomsticks to financial systems. Our problem, then, is a network problem. Specifically, it is an information problem. For information is the glue that holds networks together, and when people are fed bad information they are likely to make bad decisions, no matter how wise and kind they personally are.
He points out how more information, which technophiles like Andreessen think will save society, can be problematic in the absence of cooperation.
Despite – or perhaps because of – our hoard of data, we are continuing to spew greenhouse gases into the atmosphere, pollute rivers and oceans, cut down forests, destroy entire habitats, drive countless species to extinction, and jeopardise the ecological foundations of our own species. We are also producing ever more powerful weapons of mass destruction, from thermonuclear bombs to doomsday viruses. Our leaders don’t lack information about these dangers, yet instead of collaborating to find solutions, they are edging closer to a global war.
And points out the long-standing skepticism of experts like Bostrom.
Others are more sceptical. Not only philosophers and social scientists but also many leading AI experts and entrepreneurs such as Yoshua Bengio, Geoffrey Hinton, Sam Altman, Elon Musk and Mustafa Suleyman have warned that AI could destroy our civilisation. In a 2023 survey of 2,778 AI researchers, more than a third gave at least a 10% chance of advanced AI leading to outcomes as bad as human extinction. Last year, close to 30 governments – including those of China, the US and the UK – signed the Bletchley declaration on AI, which acknowledged that “there is potential for serious, even catastrophic, harm, either deliberate or unintentional, stemming from the most significant capabilities of these AI models”.
Taking an example, Hurari takes the example of “move 37” of AlphaGo, a genius computer move in a chess-like game no one has presumably tried before, which was initially hard to understand and impossible to determine how it was derived. Hurari imagines programs designed to beat financial markets having similar initial success before causing the global markets to fail.
Hurari then makes similar arguments with regards to personal privacy, Cold War and cyber weapons. Hurari’s analyses are not without flaws, but the ideas are interesting and not completely implausible. Many of these ideas have been discussed by others. As we know, AI companies and interested governments are strongly in favour of other AI companies being subject to limits and regulations.
Thoughts?
I enjoyed reading Hurari’s views on anthropology. But after reading Graebar, which was far more comprehensive, it was obvious he was adept at summarizing widely held views but that some of his ideas were overly simplistic.
To that end, how can people prevent the problems Hurari delineates? Indeed, it is highly likely financial companies will seek to benefit from algorithms they don’t really understand, despite problems with quants. But many will eschew such investments if they can.
As for war and weapons, are material restrictions powerful enough to make much difference? If a major problem with AI manifests, how long might this take?
It seems to me that they are blaming human-caused problems on AI. “The algorithm” isn’t AI and is being abused & misused right now, so not having or using AI is no protection. Not when the problem is the people, not the machines.
Many eschewed investing in CDOs and RMBSs but that didn’t stop them from being impacted by the 2007-08 financial crisis.
I agree that not using AI does not protect you from its effects, and that less powerful algorithms are already being misused like any financial novelty. But this can’t be dismissed by saying people are the problem instead of AI, when AI ostensibly offers more power for both good and harm; where the latter is more important but less emphasized.
The EU has made “regulations” by politicians who are unlikely to understand their ramifications. Since these are necessarily vague and may be difficult to apply, powerful tech companies don’t like them. Less ethical groups will ignore them. And everyone wants them to apply to their competition but do not need them themselves, as butter and M&Ms do not melt in everyone’s mouth though may dirty the hands.
That’s what you take away from the article?
The problem is people setting AI machines in motion without controls or a clear understanding of how they work like the brooms in The Sorcerer’s Apprentice.
People are stupid. And more and more of their reality is defined by screens presenting digital information people use to make decisions. So what happens when AI has greater the ability to filter edit that reality?
Let me give some real world examples:
- People drive a particular route because Wayz or Google Maps tells them there is or is not traffic on a particular road.
- People’s perception of what is considered a “good job” or “good place to work” or where jobs even exist is based off of a collection of advertisements, news articles, and career related sites like LinkedIn, Indeed, and Glassdoor which we already know now to be flooded with misinformation.
- My conservative in-laws Fox News based perception of New York City as a leftist dystopian hell hole is very different from my reality of going to work at my company’s Midtown office (when I actually go there).
- Similarly, my experience and POV of my company working remotely via email, Teams, and Zoom with people across multiple time zones is very different from my experience sitting in an office where I have at least a passing familiarity with most people and can see the day to day activity levels.
- For the most part, everything you know about climate change, foreign wars, politics, and the economy is based off of information provided to you, not your ability to go outside and look at it.
My point is that there is already a trend of people being more physically isolated and more depending on digital information networks to maintain connections. So it seems to me as people rely more on AI to filter, provide, and generate content for the information they use in their day to day lives, the greater the risk that we don’t know why this content is being generated, by whom, and for what end (if any).
I think where he’s right on is in the ways that today’s AI, namely machine learning and similar things can be used for purposes that become dangerous due to the opacity of the way that today’s AI learns and makes decisions. The “Broomstick” example is a good example.
I mean, if I were to come up with a stock-picking AI, it wouldn’t really tell me why it’s picking a particular stock. I’m having to trust that training it on nearly 140 years of the DJIA has allowed it to see patterns and trends that I can’t. But would I bet the farm on its decisions? That’s the question here. Would I let it invest my money without having to sign off on it? That’s the equivalent of the example of the dictator with AI-controlled nuclear weapons. What mischief can be achieved by letting them run wild? Right now, the applications tend toward the benevolent- medical diagnoses, improving supply chains, etc… But what happens when AI is turned toward making better chemical weapons? Or toward making better weapons of any sort? Or for identifying weak spots in economies or other systems?
That I think is the real warning in the article.
I’m not sure the stock picking AI is a great example. We more or less already have that. And you don’t really need to understand the model to get a sense if the stocks make sense or at least are something you want to invest in. It’s still what I call a “John Henry” problem in that you understand WHAT the machine is doing (digging a railroad tunnel and laying track) even if you don’t really understand HOW it does it faster than a human.
What is more dangerous is investing your money in “AI Fund”, or worse, having the AI manage your investments with no idea how or what it is doing. But you have to do it, otherwise you won’t get the same returns as everyone else and will see your wealth erode due to inflation. Meanwhile the AI is coming up with some weird and esoteric financial instruments that make CDOs look like Blue Chips.
Which I suppose may be ok so long as it “works”. But maybe the AI Fund algorithm misinterprets some command like “make my portfolio 20% greener” and executes a series of complex trades that intentionally put the market on a convoluted path to shut down 20% of manufacturing capacity.
That’s the risk the article is warning about I think. Creating these global AI-run systems and networks that we are really unable to control and monitor because they are too fast and complex. Sort of like John Henry racing the steel-driving machine and when the starter pistol fires the machine teleports both of them to the Moon.
Or it perceives that the best way to beat John Henry is just to murder him right out of the gate, rather than dig the tunnel faster.
Basically it’s saying that the real danger isn’t when we achieve AGI and machines start thinking, it’s that today’s applications and possible applications are already extremely dangerous.
I think that’s a far more realistic scenario than the machine springing to life and deciding to KILL…ALL…HUMANS! Some quirk in how the AI processes information causes it to decide to shut down a power grid or turn on the stoplights in a city to green and no one really knows why.
Military applications are particularly troubling because AI controlled drones don’t tire, don’t fear destruction (beyond whatever self-preservation is required to complete their mission), and can outperform humans in every way. Perhaps most concerning is that an army of AI drones takes a lot of the stakes out of warfare, making it highly likely they will be used preemptively.
And what about “profiling”? To what extent do you want your career or partner or where you live to be decided because “computer said so”. Sure, it might be nice if it picks a bunch of stuff that is a perfect fit that I might not have even known about. But there is also the question is if the AI is actually serving my best interests or someone elses,
I’d be worried that it would put together a bunch of disparate data points and decide that someone isn’t a good fit for some sort of completely esoteric reasons that may be valid, but poorly understood. Or worse, it’s opaque decision making based on historical data may be prohibited from using race, but then turns around and uses a bunch of proxy data elements- home location, school, credit rating, etc. and then basically grants a veneer of legitimacy to a fundamentally racist decision making process because race is prohibited.
There are a whole lot of dystopian sorts of scenarios that can be envisioned. The ones that chill me the most are things like health insurance companies using AI to tailor premiums and coverage without really being able to explain the why behind it. Or even for just underwriting policies- all that stuff should be more transparent than it is in my opinion, and AI would make it even more opaque.
The Alignment problem is a serious problem, if we are truly on the path to developing superhuman Artificial General Intelligence (AGI), and not just for Sci-Fi ‘Evil Robot’ dystopian reasons, but rather, just because it’s really very hard, perhaps impossible to be clear about what it is that we, humans, want.
That is the crux of the alignment problem; if you build something that is significantly more capable than yourself at a wide range of tasks, and is goal-focused, how do you ensure the pursuit of that goal doesn’t turn into things you didn’t anticipate. If you build a hugely capable robot and task it with making you a sandwich, then you decide to hit the off-switch, if it values the goal of making a sandwich more highly than being switched off, it will, if it is able to internally model the world, try to prevent you hitting the off-switch, not because it has gone rogue or anything, but because it wants to make a sandwich (and it is able to reason that if you switch it off, it won’t be able to do that).
And if you increase the goal value of letting you hit the off-switch higher than making a sandwich, it will stop trying to make a sandwich and will instead focus its efforts on persuading you to hit the off-switch (that is, it will become useless - so the choice is between a useful powerful thing that we cannot control, or a thing we can control, which is not useful).
I should note that if you have a really sound ‘yes, but’ objection to the above, you may have solved the Alignment Problem - if you have, then congratulations, and the entire scientific field of AI is waiting to hear from you. (Spoilers: mostly likely you haven’t solved the alignment problem, and your ‘yes, but’ is not a new one.)
But we might not be that close to developing AGI; current models may simply not be on a path that leads to that eventuality; even without AGI, AI has the capacity to negatively impact our world in many ways; people often argue that this is just like any other point in history where automation threatened the traditional way of doing things; mechanical harvesting machines; printing presses; powered automated weaving looms, etc. I think this is a bit different in that this technology is going everywhere, all at once, very fast - it can do harm without actually being better than the things it replaces, just because of the wholesale and very rapid nature of change.
In past cases of technology threatening the workers in an industry, there was usually time and opportunity for those workers to move elsewhere, perhaps retrain to do something else, or even if they couldn’t, it was just one industry, not all of them at once, so the impact was easier for society to absorb. This isn’t like that.
The other thing that I think makes this different is that the world has changed so that organised (and disorganised) criminals now seem to be poised to very quickly exploit any new niche, often before the technology is mature in the target industry.
Maybe I’m just getting old, but it does seem like this is getting worse; it took scammers a decade or so to react to the widespread use of e-commerce, to ramp up to the point that online marketplaces were seriously choked with scams and fraud. It has taken those same scammers only months to exploit generative AI to take those scams and fraud to new market sectors - maybe just because they already had their foot in the door, but maybe also because AI, in its current phase, is pretty much explicitly about ‘faking it’ - faking the work of humans.
Yeah, it’s also worse because whilst we humans think we know what we want, our goals are almost impossible to precisely quantify - we’re muddling through the world in a way that our brains tell is is sound and logical and best, but it simply isn’t; for example I think most people reading these words would agree that we all want a cure for cancer to be developed - except, if you really want that, what are you doing sitting there reading these words? Why aren’t you doing some action that explicitly brings us closer to the development of a cure for cancer?
- Part of the answer is that development of a cure for cancer is not our only desirable goal.
- Part of it is that our desire to attain goals does not arise out of pure rationality.
- Part of it is that even if we did evaluate our goals out of pure rationality, we don’t have access to all of the variables and interdependent factors
- And part of it is that even if we did have all of those things, the picture that we would then see might not make the curing of cancer even a desirable goal*.
*By which I mean, suppose we did have all of the knowledge and rationality and thinking power to consider all competing and inter-dependent factors, and we could clearly see the exact steps necessary to cure cancer, but those steps included, say, humanely culling 50% of the human race in order to free up the resources to do some step, and we also calculated that we would be in that 50% - would we still consider a cure for cancer to be a desirable outcome?
This is a classic thought experiment in AGI - it’s not that AGI is an all-powerful and malevolent genie that is just looking for some loophole in our wishes, it’s just that it seems our wishes can’t ever be free of loopholes, and if those loopholes are a quicker route to the solution to the stated problem AGI would take that route because that’s what we think we want it to do when we designed it - we want it to be efficient; that’s pretty much the reason for making it.
Thank you for two very strong posts. I wish I’d written them. You said a lot concisely and elegantly.
AI needs to improve when “teaching the trainee” but we are not there yet. IMHO It is not realistic to think endless data is a smart and sustainable way to go.
Or just digging through the various genetic databases, identifying some esoteric combination of genes that makes someone susceptible to cancer, and taking everyone with that combination of genes out, thereby eliminating one source of cancer. Logical, effective, and horrific.
Yeah, and there’s an infinite cascade of ‘yes, but not actually that’ kind of qualifiers you would have to add to the instructions to prevent those sorts of outcomes; Sort of like:
Human: Cure cancer; that is: make sure that occurrences of cancer drop to zero.
AI: That’s easy; I will kill everyone. Reduce the human population to zero; nobody will have cancer if there is nobody living
Hu: Yes, but not actually that. OK, I don’t want you to kill anyone, but make sure that occurrences of cancer drop to zero.
AI: Sure thing! I can simply employ another person or agency to do the killing! Please let me know if there’s anything else you need help with!
Hu: No, wait; not that! I don’t want you to cause, directly or indirectly, the unitimely death of any humans, but make sure that cancer stops happening
AI: Hmmm… That could be tricky, but I think if I render all humans sterile, no new humans will come into existence and eventually the rate of cancer cases will drop to zero as the human race becomes extinct, without me having directly or indirectly caused any deaths.
…and so on; the AI is not looking for the loopholes to be nasty or evil or anything, it’s just looking for the most efficient route to the solution to the stated objective and the search space for those solutions is inevitably larger than the human instructing it can imagine, if the AI can process vast amounts of data more effectively than the human, and there is no way to ever be completely sure you have closed off all the loopholes (which, from the POV of the AI are not loopholes, they are efficiency shortcuts) that the AI might discover.
And it’s like that because that’s what we think we want - we want a machine that can do things faster and more efficiently than we can (or else we might as well do it ourselves), but that inevitably means it’s thinking of solutions we hadn’t considered, and it has no way to filter those out as undesirable unless we give it that context, which is a hard problem.
Humans navigate right and wrong (when indeed they do), by calling on a lifetime of experience of making small mistakes and having them corrected by parents, by teachers, by the police, by peer pressure, etc - and we (usually) end up with a ‘feel’ for whether something is the right way to go or not - and notably, people still get that wrong - but if you tried to write down a set of explicit rules for right and wrong behaviour, an entity without that ‘general feel for intent’ will not know the difference between something that is permitted and something that is wrong, but accessible via a loophole.
I think that on some basic level, taking the most efficient route to a solution - any solution - is fundamentally evil. That’s why we have laws and morals and ethics and empathy: so that we don’t do things with perfect efficiency.
I have to think about this.
Yeah, maybe, or maybe just amoral. Unconcerned with your primitive human values.
I feel like ‘evil’ would be seeking to do it in such a way to cause an adverse collateral effect.