It’s not that it would generate new objectives that are entirely unrelated to our instructions. The problem is that it’s extremely difficult to specify what we want in a manner that does not have unforeseen consequences in an entity that is much smarter than we are.
And if you think this is not discussed, you cannot have read any serious work in the field. That’s the entire discussion, the alignment problem.
A straightforward and suitably chilling example is that it would quickly realize that it is much more effective at achieving goals than any human. Therefore the best way to achieve any goal that a human specifies is to ensure that it survives. So the first thing it does is to circumvent any kill switch.
Could it, say, conclude that it can’t very well carry out any goals that humans ask of it if it gets destroyed or switched off or whatever, and so it needs to safeguard itself — not because anyone asked it to, but because that’s obviously a means to whatever end someone could ask it to aim at?
Right: it’s the Genie story. ‘Be careful what you ask for’ and so on.
I didn’t say I thought it was not discussed–just that I haven’t seen it in the discussions of AI I happen to have read. A fair assumption might be that the process by which AI obtains the capacity for independent goal-generation, is well-known and accepted.
If it’s well-known, I’m just saying, I haven’t seen it laid out.
That seems plausible. It would mean a single goal (‘my survival is crucial so that I can carry out instructions I’ve been given’). Would that lead to a capacity to generate multiple goals independent of the issue of ‘how can I best carry out instructions’?
Again, I don’t want to get off-topic. But since all this does sound far beyond what we have now, it rather supports the OP’s premise.
What if it’s given two instructions that, being mutually exclusive, can’t both be carried out? What if it’s given a single instruction that it’s possible to fail at, or one it’s impossible to succeed at? On the one hand, you could maybe keep your 100% success rate intact by taking steps to keep from being given bad instructions — but, on the other hand, it seems to me that you could keep from ever failing to carry out instructions by making sure you never get any instructions.
Because humans are no longer capable of instructing the AI. The AI and other rival AI can think faster and better than we can.
We have created a computer that can play chess better than any human. Someone else creates another computer that can play chess better than any human. Our computer plays their computer. Are you going to tell our computer how to play?
Why would it, as it were, CARE about following the instructions that humans would eventually give it? I kind of thought the idea is that — prior to getting any given specific-content instruction — it’d have some kind of general-purpose “successfully carry out a specific-content instruction” context already in place, or else it wouldn’t start plugging away if a particular instruction comes along.
Computer programming never included any assumption that the computer would care about following an instruction.
Right, this would be a basic assumption of all human-to-computer communication: that the computer would follow instructions given it.
The big question is the hypothesized leap to computers caring one way or the other about anything. Because for their first decades, ‘caring’ has been irrelevant. It’s just been ‘get instruction then carry out instruction.’
I agree that a sufficiently powerful entity engaging in sufficiently large-scale activities on planet earth with sufficiently little regard for human life would be dangerous.
But I don’t think it’s a given that, granted current conditions, we are on a path there. There is a lot that has to happen in between.
One difference–a big one, I think–is that if we suppose that AI has near-infinite intelligence, then I think we have to suppose that it will understand the implications of its actions; otherwise, it’s not that smart, right? And if it does understand those implications, then it stands to reason that it could potentially go the environmentalist route.
Like the other poster, I’ve read popular science articles and books on the topic, and I haven’t come across some of the things you’re talking about. Personally, I would prefer that you present specific arguments here instead of simply saying I’m wrong and pointing at a large body of research.
I guess what I’m saying is, ‘successfully carry out an instruction when given an instruction’ sort of already is an instruction: yes, a somewhat content-free one, but it arguably could already get an AI plugging away — without caring one way or the other — at the (general) task of being able to successfully carry out an instruction if ever given a (specific) task.
But trying to guess when a potential apocalypse might happen is not a useful way to look at the problem. I really have no idea if it’s more likely to be a year away, a hundred years away, or never.
The issue that to whatever extent people are working on developing artificial intelligence, we should be concerned that they are working with appropriate safeguards. Because if AGI does happen then runaway superintelligence may follow very quickly.
We are at no risk of AI killing all humans and ‘taking over’. An AI cannot run the world by itself - not even enough of it to guarantee continued energy and replacement parts. For a very, very long time humans will be absolutely necessary for AI to thrive, and a healthy human economy is necessary to provide it all. Enslaving humans will not result in high quality 5nm chips to replace failing ones.
A much bigger risk is that stupid humans will give AIs power over critical systems without fully understanding their power and limitations. For example, letting AI control air traffic might be an efficiency win, but after some time we will absolutely rely on it, giving it some power over us. If an AI improves efficiency with complicated routing algorithms and such, taking AI out of the mix would be an economic cost. And if we do it long enough, we may not even have the human based control infrastructure to go back to.
We don’t need legislation stopping research into AI, but many industries should be looking at AI standards and practices and write rules around safe adoption of AI. This isn’t a job for acadenics and AI researchers, it’s job for process engineers, traffic engineers, etc.
Somewhat. This feels like the first big thing to come along in a long time. As with a lot of other stuff these days, I’m not sure how easy it will be to monetize it and turn it into products (see: the tremendous flop of the Metaverse).
Wait, we’ve had stagnant wages for decades. I’m not an optimist. I also think a couple hundred years with rapidly advancing technology and changing social systems is not a proper sample.
If the type of AI comes along that is capable of the architectural plan I described–and both boosters and doomsayers say it is–then all jobs are destroyed. Period. If a robot can walk around an industrial plant and hear perfectly and interpret English and Japanese perfectly, then I am out of a job. There is no upskilling possible after that.
Further, it’s not as though we have an economy currently in which people are happy and prosperous and it’s all copacetic. No, people are frustrated, pissed, and generally lacking in hope. I said “innovation-scarce” above because companies like Amazon, etc., which used to seem like the future and give us all kinds of great stuff and services are pulling it back. The Internet is locked down with a zillion paywalls. Customer service, or the lack or AI-ization thereof, has practically been weaponized against consumers. The Great Cheapening is upon us, and companies themselves are having a hard time of it (see Disney, Nike, etc.). Things suck right now, and technology is making it worse, not better. I see AI as being another net fuckover of ordinary people. (And I say this as an MBA who doesn’t want to be negative about business, etc.)
Except the small business owner is then also replaced. Except maybe vape shops. Those are eternal.
I think you might have missed the past 30 years–but maybe things are different in Canada. A lot of good jobs have been replaced with shitty service jobs that don’t pay as well or have as much social prestige. But our malaise has at least funded the rise from poverty of India and China (it actually has–and it is a benefit to humanity, but the price in the US has been high).
It’s already happening, and then that pressure is extending to the university system (wait, what do I get for this degree? Fuck all? Oh, OK…).
Is it going to improve voice to text transcription? Personally, I shudder to think of all the AI cruft that will soon be infecting our “devices.” It’s… gonna suck, sorry.
I didn’t say that. I am predicting that, like anything else, we are going to see some leaps, and then it will hit a wall, and the way forward from there will be unclear. E.g., the iPhone in 2007 was incredible. Now, however, improvements in cells phones are barely noticeable.
Here we agree.
Well, AGI will destroy human society has we know it, so I hope not. But I also don’t think so, and I have not see credible writers assert that that is true.
I don’t think they’re close. Gemini is both impressive and a bit of a joke at the same time.
It does not have a will in the sense that I described. It does not have the capacity to fear and fight for its own survival.
The whole consciousness issue is interesting but different than what I was talking about.
I don’t think that is a given. I’m not even sure what ‘superintelligence’ is. But it may be that AIs simply can’t get much smarter than the smartest human becausevthere is no ‘smarter’ data to train on.
More likely, IMO, is that ‘superintelligence’ will just turn out to be normal intelligence with the knowledge of a thousand different experts. Perhaps it will be able to draw across it all and make new inferences that are beyond the capability of individuals simply because they can’t learn as many fields, while being no smarter than an expert in any one field.
Or maybe new mental capabilities will emerge that have yet to emerge in people. We really don’t know, but I would bet on the former.
I have no idea what ‘appropriate safeguards’ would look like, other than isolating it from the internet.