FOLLOWUP: What are the chances artificial intelligence will destroy humanity?
[Regarding your column on artificial intelligence,] there’s no way you can just dismiss Cohen’s predictions (or any other prediction about AI that I can think of) as unfalsifiable. I don’t think the standard of falsifiability has any relevance here. “Unfalsifiable” is not synonymous with “wildly speculative.” You can certainly can call them “wildly speculative” or “completely wrong”, but to make that case I think you must simply address Cohen’s evidence and reasoning on its merits. – Riemann
Cohen isn’t offering evidence; we need to be clear about that to start with. His reasoning, on the other hand, makes an alarming amount of sense and deserves a wider audience – we’ll get back to that. It’s his unstated assumptions that are exasperating and do his cause no good. And that’s unfortunate, because he raises an important concern.
Let’s clarify a few things. My reference text, so we’re on the same page, is “Science & Speculation,” a 2021 paper by UK philosopher Adrian Currie. Currie acknowledges that scientific speculation is sometimes derided as “theological” – that is, lacking in testability. But he thinks it has its place in some circumstances, speaking of “speculative hypotheses [that] are not to be judged in terms of evidential support, but what I’ll call their ‘productivity’” – by which, broadly speaking, he means their ability to spur fruitful thinking.
This is a useful way of looking at the matter. The notion that advances in artificial intelligence are likely to lead to artificial superintelligence (ASI) is inarguably a speculative hypothesis. It lacks evidential support or, to put it another way, it’s currently unfalsifiable (untestable). Some may quibble that it’ll be testable eventually. However, the scary-AI argument is that full-blown ASI may emerge quickly, unexpectedly, and irreversibly – in other words, once it becomes a testable proposition, it’ll have become dangerous and undefeatable. So if we’re going to take precautions, we need to do it now, with nothing to go on but a scientific hunch.
An argument like that isn’t going to get much traction with the National Security Council. On the other hand, as speculative hypotheses go, the potential emergence of ASI has been productive – it gives you something to think about. The paper by Michael Cohen and company is evidence of that.
To perhaps oversimplify a complex argument, Cohen and his co-authors contend a sufficiently advanced intelligent Agent would likely find the easiest way to achieve a goal set for it by humans would be to cheat, and would modify its programming accordingly. In other words, there’s a good chance it would go rogue – and from there the road to perdition is short.
OK, a rogue computer isn’t a novel idea. The contribution by Cohen and company is to show that it’s not just possible but likely. Not long ago that conclusion would have been disturbing, maybe, but not urgent. Now, when intelligent Agents seem like a real possibility, it’s cause for genuine alarm.
That’s where Cohen’s approach becomes problematic. The unstated assumptions in his paper, but which he acknowledged to me, are that ASI is sure to emerge eventually and that a superintelligent Agent would have superpowers. Given that, it’s not difficult to demonstrate that a rogue Agent could destroy humanity.
Could that really happen? I suppose, but most people would roll their eyes.
That’s too bad. Whatever may be said for ASI, artificial general intelligence – let’s think of it as an Agent with human-level reasoning and decision making ability coupled to machine-level capabilities in other respects – now seems achievable (although it’s not a foregone conclusion). A human-level Agent would have the same propensity to go rogue as a superintelligent one. Not having superpowers, it wouldn’t necessarily be able to destroy humanity, and it might have a tougher time modifying its programming. Still, ordinary humans sometimes outwit other humans. If a machine could do the same, it might do some serious damage – for example, if someone were foolish enough to put it in charge of life-support equipment, a power grid or, God help us, a nation’s nuclear arsenal.
There’s little sign anyone outside the scary-AI school has given serious thought to this prospect. Evidence on this score is a recent commentary in the Wall Street Journal, “ChatGPT Heralds an Intellectual Revolution,” by Henry Kissinger; Eric Schmidt, former head of Google; and MIT dean Daniel Huttenlocher. This lengthy piece may be taken to represent the mainstream of informed opinion on AI. It acknowledges the technology’s benefits but mainly focuses on the dangers: humans may put too much trust in AI-generated conclusions they can’t easily corroborate; AI can be used to propagate lies; it may be controlled by a small elite; and so on.
But there’s almost nothing in it about the possibility of rogue AI – that is, intelligent Agents purposely pursuing agendas that would harm humans – other than a vague reference to “alignment of AI aims with human goals.” The authors are a well-wired group and are surely aware of scary-AI thinking. But they either think such talk is crazy or fear that, by mentioning it, they’ll be thought crazy themselves.
That’s where pursuing a speculative hypothesis far beyond the realm of the knowable gets you into trouble. By painting their conclusions in apocalyptic terms, the scary-AI crowd makes it easy for their conclusions to be dismissed – and for no reason. Whatever one thinks a rogue Agent might be capable of – destroying humanity or merely murdering the crew of a space mission – the policy recommendations for today are the same: analyze how things might go wrong, consider precautionary measures, and monitor the state of the art. In other words, spend less time on the science fiction – or what may seem to some like science fiction – and more on what decision makers can be persuaded to think might actually come to pass.
– CECIL ADAMS
After some time off to recharge, Cecil Adams is back! The Master can answer any question. Post questions or topics for investigation in the Cecil’s Columns forum on the Straight Dope Message Board, boards.straightdope.com/.
