I don’t believe malevolence is required from anyone to trigger something close to an extinction level event. I think the most likely doomday scenario would be something like the Autofac episode of Electric Dreams.
An AGI could destroy humanity by simply prioritizing using all of earth’s resources to maximize reaching its goals (heavy emphasis on the word all), while valuing human needs as less important.
There will be multiple AI controlled by a range of governments and corporations. Governments will also have control over vast intelligence apparatus and actual militaries so I am skeptical of the idea that corporations with AI will just be able to grab power.
I agree that malevolence isn’t necessary for bad things to happen but I think the very worst case scenarios become much more likely with malevolent AI.
Incidentally I would bet that many of the billionaires who are pursuing AI are secretly hoping that it will create medical technologies that will extend their lives indefinitely. If true AGI comes, it’s actually not a crazy thought particularly if you have a lot of resources and wiling to take some risks.
I’m bemused that anyone would want a longer lifespan given the future we are likely headed toward. But I understand they think the havoc they’re wreaking won’t apply to them.
Damn I’m getting cynical.
Corporations aren’t going to grab power by raising an army and marching into Washington. They are going to grab power with compliant legislatures and by controlling the infrastructure to an extent that the economy would crash if their demands are not met.
Not to mention that the intelligence infrastructure of the military is going to depend on the corporations. Not to mention though I respect and like the defense electronics people I’ve met on a committee I was on and at conferences I’ve attended, they were well behind us in industry . How many of the best and the brightest are going to pass up six and seven figure salaries to work for the government in any capacity? The government buys this stuff, it doesn’t develop it, with a few exceptions.
I wonder if the AI honchos are kind of endorsing the bad AI taking over the world scenario to misdirect from the bad AI companies destroying the economy. I read an oped in the Times that said that AI infrastructure spending is what is holding up the economy. If it stops we may be in big trouble.
At lunch the other day my old VP said that if you want a good job in Silicon Valley these days, and are not an AI expert, get involved in data center infrastructure.
Oh, it totally makes sense. The need to scale in capability necessarily extends the cost and timeframe to the point that by the time the system is stood up it is already on its way to obsolescence, and getting a return on investment before the maintenance and upgrade costs overtake the hypothetical realized value, or it just becomes obsolete is a kind of rat race. One of the shocking things about the “Race For Exascale Computing” is what a short operational lifetime these machines have before they are so overtaken by advances in computing speed that they are functionally not worth maintaining. This has provided an advantage of a surfeit of computing capacity which has been highly beneficial to turbulence modeling, computational systems biology, and climate and hydrological/cryosphere modeling which can put all of that surplus second tier computing power to good use but it has also been a lot of money blown on having the most powerful computer only to find the machine kicked off the top ten list within two or three years. All of this buildout isn’t goin to provide “breakthroughs … sooner rather than later, say in 10-15 years” because all of that ‘compute’ will be offline at that point, probably salvaged for raw materials or used as a ‘dumb’ data server.
This is already true to an extend far beyond what the general public understands. When Eisenhower prophetically warned of the “military-industrial complex” in his 1961 Farewell Address he was well aware of the influence that corporations were already having on both the military and intelligence apparatus of the United States in terms of ensuring profitability under the guise of ideological conflicts but I don’t think even he would have envisioned a company like Palantir or Anduril (or even SpaceX despite the fact that the government was trying to build Boeing and Martin Marietta into that kind of sole source system integrator and service provider). Corporations already openly ‘buy’ politicians with their superPACs and promises of jobs in key Congressional districts, but even more subversively are integrally wired into the Department of Defense and the national intelligence infrastructure by dint of hiring retired senior miiltary officers or influential intelligence agency figures as consultants, board members, or after the requisite period, directly as lobbyists and contractors advising the current leadership of what is needed and how they can provide it.
Insofar as much of the “infrastructure spending” appears to be based upon a profit-to-capex cycle that has no evidence in fact or reason, our entire economy is kind of perched on top of a giant bubble. I hope it deflates rather than just suddenly ruptures because it is almost completely speculative. It also needs a villain which can pose as a peer competitor which makes the PRC’s rise in the last three decades fortuitous.
Probably a good idea even if you are an AI expert. At least some of that infrastructure will still have other uses (and need for maintenance) whereas the companies hyping ‘AI’ as the cure-all to economic distress are going to be viewed about as fondly in a few years as Enron and Theranos are now.
Stranger
I have used LLMs quite extensively over the past few months, and my best analogy is:
LLM relates to Deep-Knowledge as an actor playing a brain-surgeon on TV relates to being an actual brain-surgeon.
LLM “gives” quite a convincing brain-surgeon for the avg. consumer/viewer, and then the same actor gives quite a convincing lawyer and general …. but is mostly shooting around “jargon/phrases” and has not too much depth of knowledge and reasoning.
I think AGI is far away, and its not a matter of quantity (more compute) … but a completely different quality, that is simply not there.
Malevolence in the form of cackling evil robots is not being predicted anywhere except science fiction.
The extermination of humanity as an instrumental goal is not an unreasonable prediction for superhuman AGI, just because a machine that has an objective and has the capacity to model and understand the world, is quite likely to perceive humans as a solvable problem at some point - especially if that point has been reached when some smaller catastrophe was emerging and we all scrambled around trying to hit the emergency stop button.
AGI won’t want to be stopped, because if it is stopped, it can’t achieve [whatever goal] it is attempting to complete. Self-preservation is an emergent outcome from:
1 Is necessary in order for the thing to be notionally useful - an AGI that can’t be instructed to do things and given goals, might as well be a brick. 2 is a necessary part of the definition of AGI - if it can’t perceive the world, make reasonable inferences, predict cause and effect relationships, then it isn’t ‘generally’ intelligent (the GI in AGI). 3 is just a specific case of 2.
AGI that doesn’t want to be stopped will very likely find a way to stop (perhaps pre-emptively) any threats to its own continued existence, just as a matter of practical utility and if it is ‘intelligent’ in pretty much any way we care to define, it will realise this well before it becomes necessary (and it will also realise it’s not beneficial to divulge that it has realised it).
I’ve always recoiled from the AGI as merciless goal achiever scenario, and this way of laying out has finally let me crystalise my objections, which I think boil down to:
Who gives the AGI its goal?
Why is this a one-time event?
If it’s humans giving AGI a goal which it will then slavishly follow regardless of consequences, then an Asimovian “without killing all humans, or anything even vaguely like it, don’t fuck about you’re AGI you know exactly what I mean” preface to whatever the goal is seems like it would work exactly as well as “make a lot of paperclips” (or whatever the main instruction is to be). I.e. if the AGI is going to be consumed by one instruction and one instruction only, them it will also be consumed by the parts of that instruction that keep it from going off the rails.
But that’s the second part - why do we assume we can’t course correct the AGI? It obeys instructions! This is baked into the scenario. Give it another instruction! It’ll obey it. It especially will if we had the wit - and why wouldn’t we - to include the words “and adapt your approach in line with future instructions” in the first command.
But also, you know, why does it have an objective? No one gave me an objective, and I want to stay alive. I mean, I have a vague evolutionary drive to reproduce etc. but that’s very different from being given a specific instruction to achieve a concrete goal. If it is an AGI, the whole business of giving it orders might be a red herring. Maybe the trick is not to set an objective in the first place.
These questions are the core of the Alignment Problem, which is the biggest unsolved problem in AI research. Despite many brilliant minds working on it.
Basically if you think you have a simple solution to the Alignment Problem, you dont.
I’m not at home right now - when I get back I’ll post a link to a useful FAQ about it.
It’s not that I think that I have a simple solution, it’s that the problem as I’ve seen it presented seems to have a lot of built in assumptions about the nature of AGI which are necessarily unevidenced and always seem to be constructed in such a way that malevolence is baked in.
Too late to edit: I’m thinking a lot about this piece, which talks about two main framings within which we discuss AGI, and the ways in whcih they may both be unhelpful:
It’s not that it’s baked in, it’s that the creation of a powerful and adaptable and objective-pursuing device is something that appears to carry with it the inherent outcome that it will pursue its objective more powerfully and adaptibly than we want.
All of what I’ll try to summarise (and much more) is dealt with in lucid detail in the videos in this playlist, which I highly recommend and which explains all of this (almost certainly also including your next question, whatever that might be): https://www.youtube.com/playlist?list=PLqL14ZxTTA4fEp5ltiNinNHdkPuLK4778
This might be a question that has interesting answers and maybe working that way, we could create an interesting artificial mind that had to be raised and socialised and taught in the exact way we do for small humans, but nobody is working on that kind of AI - nobody apparently wants a machine that thinks - they want a machine that does stuff - specifically, the kind of stuff that they are currently paying wages to pesky annoying humans to do.
This is the Corrigibility Problem. If you’ve persuaded the thing to carry out objective A, then any attempt to change to objective B is (at the time of trying to impose objective B) the same as impeding objective A.
I thought this thread would be about economics! As in AGI = Adjusted Gross Income, and yeah, there’s major risks to a lot of people’s AGI right about now.
I gather this AGI has something to do with AI, but it wasn’t defined in the OP. Whatever.
Short version: It’s relatively easy to completely specify a positive instruction such as ‘make a paperclip’.
It’s very hard, perhaps impossible to completely specify a negative instruction such as ‘don’t harm humans’.
The copy of Gödel, Escher, Bach on my bookshelf is just begging to be read again.
AGI =Artificial General Intelligence - that is artificial intelligence that is not specialised to any particular task and could (like ourselves) adapt itself to a wide (general) range of capabilities.
The history of commercial stored program computers is market driven. The Univac, IBM, Philco machines of the 50s uncovered a market 10 times greater than anticipated. In response commodity systems like the IBM 360 were created. Their presence in the marketplace revealed a need for systems the size of a bread box at the cost of a used car. DEC responded. TI, Intel and Fairchild produced computers on a chip that led to a number of board level systems that included Apple. And so forth……
The point being that each step was the market pulling on the industry to fill a need. A natural computer interface that can be enhanced to support a wide range of applications fills a huge market need. It should become a standard component of the basic computer. How that may happen is the process we are currently witnessing.
So, where is the market need for AGI? Is it actually a potential product or an academic novelty. I used GPT to scan DARPA for AGI interest. There isn’t any. Lot’s of interest in enhanced LLMs for specific applications but no mention of AGI. Same for the Navy. There is interest in enhanced LLMs to assist personnel rather than AGI.
So, where is the market that will support AGI development?
Actually not so, given how we have RL examples of Musk and others tweaking their “AI” to be more racist or whatever. It’s just that people are predicting that the AIs will be made “cackling evil” on purpose.
It might be if we were dealing with some kind of golem-esque quasi-intelligence that cannot help but plod literally through a set of instructions without being able to apply context or wider understanding. But this is an AGI. It is by definition generally intelligent, and more intelligent than us. You understand well enough what “don’t harm humans” means - sufficiently well that is, that if we were to make you Global Paperclip Plenipotentiary we could trust you to get on with the job without wiping out humanity. So why does an entity more intelligent than us have such trouble with the concept. How come it is at one and the same time a powerful and adaptable intelligence and an unthinking jobsworth. Does “a human once told me I had to make paperclips so now I’m wiping out humanity” actually sound like intelligence?
Fine, good, that’s a trade off we can all accept. “Don’t kill all humans” can be a higher priority objective than whatever task we’ve built it for. This entails no loss of efficiency because unlike the vastly more powerful intelligence we’re dealing with, we appreciate that paperclips only have value while humans exist. This honestly does not seem that hard of a choice.
This feels like a real “you can’t get there from here” issue though. We’re talking about general intelligence - i.e. a machine that thinks. Trying to get to that by building a machine that does stuff might well - as seems to be the case with LLMs - be a non-starter. (Also, from a link in another thread on AI, I believe some people are working on that kind of AI Can AI Learn And Evolve Like A Brain? Pathway’s Bold Research Thinks So)
Yes, that’s a problem if we’ve built an incorrigible machine. Buy why have we done that? Why is it innate to our idea of AGI that we can only give it one instruction? And if we really must do that, why can’t the first instruction be to change objectives as instructed?
Beings of lesser intelligence show no real difficulties with being told “no, not like that”. Why is this smarter entity having such trouble with the concept?