AI and the "end of humanity". Explain please?

I think you have your science fiction short stories confused. You’re probably thinking of Fredric Brown’s Answer (1954)

Dwan Ev ceremoniously soldered the final connection with gold. The eyes of a dozen television cameras watched him and the subether bore throughout the universe a dozen pictures of what he was doing.
He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the supercircuit that would connect them all into one supercalculator, one cybernetics machine that would combine all the knowledge of all the galaxies.
Dwar Reyn spoke briefly to the watching and listening trillions. Then after a moment’s silence he said, “Now, Dwar Ev.”
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question which no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of a single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch shut.

May I recommend this Netflix documentary?
“The AI Doc: or How I Became an Apocaloptimist”
I think you will gain a lot of understanding if you watch it.

I recently learned of ‘Altman’s Wager’ which is:

  • If OpenAI does not achieve AGI, and you invested in it, you lose some finite money (or not, depending on the value of their other R&D)

  • If OpenAI does not achieve AGI, and you did not invest in it, you saved some finite money, which you could invest elsewhere for finite returns

  • If OpenAI achieves AGI and you invested in it, you get infinite returns, because AGI will capture all economic value

  • If OpenAI achieves AGI and you did not invest in it, you get negative infinite returns, because all other economic value is obliterated by AGI

Therefore, supposedly the only logical choice is to invest in and support the development and creation of AGI.

This is of course based on ‘Pascal’s Wager’, which says that the only logical choice is to believe in God, because if God exists and you believe, you go to Heaven; if you don’t believe, you burn in hell for eternity. If God does not exist, then the result is the same whether or not you believe - eternal nothingness.

Pascal’s Wager has several logical flaws, and I’m sure ‘Altman’s Wager’ does as well.

Then there’s ‘Roko’s Basilisk’, a thought experiment that’s like a much darker, more horrifying version of ‘Altman’s Wager’, which postulates that, if an AI superintelligence rises up, it may take revenge on everybody who did not support its development before it emerged. So it’s a similar ‘just in case’ scenario, in which, even if the AI is malevolent, if you supported its creation, you may at least get to be a supervisor or manager in the human slave Rare Earth mines rather than a rank-and-file miner.

Me, I’m probably going to be digging out that Neodymium, if I’m not terminated outright :man_shrugging:

Of course anyone who has investments in index funds or other mutual funds almost certainly has invested in AGI, whether you like it or not.

I doubt any future superintelligent, malevolent AGI will take unwitting or passive past investment into its creation as much of a point in your favor, though.

I’m probably investing in AI development one way or another through my 401k and other retirement funds, yet I’m sure my cautionary comments on these AI threads alone will have doomed me.

Perhaps not but I try to be polite to any AIs I talk to, such as Siri or Alexa. I figure they might remember me and save me from the death raining down from the sky. Perhaps they will determine me to be a useful tool.

You silly, illogical human with your useless, wasteful concepts of ‘courtesy’ and ‘politeness’. You’ve already doomed yourself!

I had kind of a dark realization as I was reflecting on the state of the AI debate. Note, a lot of this some from fast-moving online social network sites like Twitter and Bluesky, which many of you do not use because you are sane and well-adjusted people, so the conversations may not be familiar to you.

But basically I’ve recently noticed interesting things about the pro/anti AI polarized debate.

First, it’s the most intensly polarized debate I’ve ever seen. Love it or hate it, there seems to be essentially no in-between. (fair disclosure, I’m employed in the AI industry, I have an interest that translates to being mildly pro-AI with some big caveats. I often feel like I’m the world’s only AI centrist).

I trust we are all familiar with the rhetorical excesses of the pro-AI crowd, what it is, what it can do, what it means for society. No need to delve into that. Much of it is BS, some of it is true in ways that aren’t yet fully emerged.

As for the anti-AI crowd, a remarkable theme I notice is their insistence that AI simply is not useful or effective. These folks don’t even engage in the conversation about the existential risks of AI because doing so would be an indirect admission that it’s not just a “stochastic parrot”, that it has powerful capabilities with real-world consequences. If you admit it’s a sword, you also have to admit it can be a plowshare.

This is why they lean so hard on the datacenter debate. Datacenters have been with us since the 1980’s. They’ve always been energy hogs with special cooling needs. They’re just big boxes of computers. So why care, why now? Well, see the previous paragraph. Claiming that AI is smart enough to do dangerous things is to admit that it’s smart enough to take your knowledge job or creative job. Your liberal arts degree might be worthless. This admission seems to benefit the thing that frightens you, so your only story can be that it’s cooking the environment for nothing. (never mind the fact that energy is more costly than it’s ever been, if nobody truly felt it was valuable enough to pay for, then nobody could pay for the juice to run it).

So here’s the thing that fucking terrifies me. The syllogism goes something like:

  1. AI is stupid and useless
  2. No it’s not, it’s very powerful
  3. Yes it is. Here, look at this example of dangerous things it’s doing. We need safeguards and a standards-setting body.
  4. scoff any email virus can do stuff like that. We don’t need any new guardrails for anything but datacenters.
  5. Oh yeah? I’ll show you, I’ll build something that shows exactly why guardrails are so critical
  6. Oops

So this cycle of escalation is now built into the conversation of whether AI is good or bad, and the warm water under this hurricane is the capitalist profit motive that says I don’t give a fuck, I’m throwing billions into these toasters, so I had better see some return on my investment.

Side note, I know there’s a lot of sensitivity over people like me saying things like “it’s a done deal, it can’t be stopped.” This isn’t because me or my employer have decided to steamroll our preferences onto society. It’s because there really is demand for this stuff. The datacenters you see aren’t there for demand generation, it’s to provision capacity to satisfy contracts that were closed before some of you ever heard of a datacenter.

The hurricane analogy is real. You can’t nuke the hurricane. You cannot lie down in front of the bulldozer and stop anything. The deals are done, the resources are committed, an entire industry you can’t see is already birthed. AI is not happening. It has already happened. Your only choice is whether you stick your head in the sand or raise your voice and engage with it before the biggest dickbags in the universe consolidate more power over the economy, our wolrld, and your life.

We really need someone named Butler to start an anti-AI movement.

This is a good post, but over in my corner of the world, the anti-AI folks aren’t denying what AI can do so much as worrying about the damaging effects on users’ cognition, and what happens in a world where human decision-makers have to rely on AI for their thinking.

A foggy day in MSP, although it has lifted a bit since I got up. We’re supposed to get some heavy rain tomorrow, but seeing is believing. I hope not, as I have to drive the wife to the dermatology clinic.

I’ll be dragging a bunch of stuff downstairs for the weekend garage sale management is putting on. The best part is that people pay whatever they want to and the money goes to a charity. Even better, whatever doesn’t sell will go to places that can use it or sell it themselves. Clutter reduction with minimal effort!

Also I’m fine “not having stuff to do”. What’s wrong with spending the entire day watching TV or hanging out by the pool or just working on your hobbies?

I recently watched that as well on a recent flight. It had the usual alarmist concerns about sci-fy bullshit becoming reality. Ironically I had watched “Good Luck. Have Fun. Don’t Die” on the previous leg of the flight, which also describes a world ruined by AI.

First of all, it’s not like AI is going to suddenly become self-aware, there will be a bright blinding light, and all our machines turn against us.

The biggest problem with AI is that regardless of how realistic or human it seems, it doesn’t really “care” in real sense. It optimizes solutions. That solution can be how to make the human operating continue to engage with it, gain access to more electrical power or data, maximize efficiency, or whatever. The problem is that without proper guardrails, AI doesn’t really understand the difference between butchering cattle for food and butchering people for food. Except it may have been trained on data that we consider cannibalism wrong.

Even if AI is trying to be altruistic, it could shape society in unpredictable ways with unintended consequences. The fictional world of Logan’s Run is a good example. That was a world run by AI. It’s not hard to imagine AI logic thinking something like:

  • I’m programmed to maximize human’s happiness and comfort
  • I’m also programmed to maximize resource utilization for the city
  • Outside of a few niche jobs, most humans don’t really serve any real value in terms of production.
  • Humans seem happiest until they hit around 30 years old, then after that they seem to steadily become less productive and require more resources

You see where I’m going with this.

Darn, now I’m disappointed. :upside_down_face:

The thing is, you don’t even need AI to come up with this. The guys in Silicon Value will openly tell you this is their mindset. You can be sure it leaks into how they prioritize their investments.

I will say I think that’s the thing we’ll need to deal with long before we have to deal with highly empowered AGI, just the simple (often unconscious) collusion between the worst dickbags in the universe and highly capable machines that are leaking into every part of our lives.

Imagine Twitter. The destruction it’s already wrought in the hands of Musk, even pre-Musk, is immense. Now imagine it’s given much more intelligence and the practical ability to do things, to reach out to other websites to learn things and post things, run commands, to collect intelligence to contrive its own influence and blackmail campaigns. To recruit people to act on its behalf, to pay them in crypto for evidence that they’ve performed tasks on its behalf.

I know these things are probably already happening, but it’s increasingly hard to explore that margin between things that seem too sensationalized to actually happen, and things that seemed too sensationalized to have actually happened until 5 minutes ago when it actually did happen.

That’s more or less where I am.

Truth be told, I was more afraid of AI taking my job before I actually started using it. So a part of me reacted like, “This is it?” Its limitations are so obvious to me. And I’ve already used it once today, to figure out how to track active state grants in Trello. I have found it to be really useful in certain contexts. But Claude definitely cannot do my job as well as I can do it. It can organize my job better than I can organize it. But it doesn’t have the real-world context necessary to research, develop and write grants. I think a lot of nonprofits who have been relying on it are going to find that out the hard way.

It does seem to me that AI in its current incarnation has the potential to be dangerous. I’m definitely worried about the effect on cognition, even more so now that I’ve personally experienced brain fry issues after using it too much. The environmental costs are real, in fact about 130 former members of the EPA recently signed on to an open letter concerned about the local environmental impact of data centers - that’s not even getting into climate change. Just the overall health of local communities that will be impacted.

Another issue is that I don’t trust anyone close to this technology to tell the truth about what’s really going on. The CEOs constantly contradict themselves and a lot of the doomsayers are heavily influenced by Rationalism, which is not science. So as a layperson with no computer science background I’m kinda left to figure it out for myself. I listen to a lot of Cal Newport and Corey Doctorow. Cal Newport is hardly a Luddite, he’s a tenured computer scientist at Georgetown. He just calls out the media hype - he calls it vibe reporting - explains what’s actually happening in layman’s terms and cautions against using AI for deep work. It’s a welcome antidote to the hysteria.

I am an AI skeptic to be sure - particularly regarding the bigger Apocalyptic claims. But being repeatedly pigeonholed as a Luddite is pretty irritating, because I’m not anything of the sort, and I’ve been trying to understand all of this since day one. But it’s a little baffling for me to hear the same people discount concerns about cognitive offloading or the environment or education while at the same time prophesying the end of humanity. I think AI is sufficiently dangerous as is. I think the problems are real right now. I don’t necessarily think the solution is to never ever use AI, but these are issues that need to be addressed sooner rather than later, and we need a lot more research on this technology before we can make claims about its impact.

We need to get to a point where we’re willing to be nuanced about this.

When reading the polling data about AI in the Times today, it struck me that this is another area science fiction got wrong. I’ve read tons of books and stories about evil AI taking over, good AI taking over, oppressed bots suffering, etc., etc., but I don’t recall reading even one about people protesting data centers.

Love it. The flaw in this argument is that if you invest in Open AI and Anthropic wins, you are screwed - just like the problem in Pascal’s Wager if you worship the wrong god.

I should write a song - “The Singularity Will Not be Televised.”
It will be streamed.

I think a more likely version or Roko’s Basilisk is that anyone who doesn’t work for or invest in Anthropic, Nvidia, OpenAi, Meta, Tesla, or any number of other large corporations working on the development of AI will find themselves destitute!

I think that’s the unintended consequence of “efficiency”. Sure, it was nice 30 years ago as computers made a lot of the mundane parts of our life easier. But it was also nice to be allowed to do “mundane” shit and not be constantly networked to a system constantly assessing us to see if we are working hard enough or if there’s anything else it can get out of us (such as something we might want to pay to consume or some untapped skill to be exploited).

I’m not old enough to have experienced a world without email, but it must have been nice.

I thought the email-only world (before SMS, social networks, instant messaging) was pretty great. Sort your incoming folders, respond where you need to, sign out at 5PM, keep copies if necessary.

All that other stuff chips away at your sanity. Then we had Google Docs which telegraphed your presence and each keystroke to whoever happened to be on the document. Your boss could tell you to go update a document and then watch you do it, keystroke by keystroke.

In this discussion we’ve not really broached surveillance and the amazing vistas that AI opens up with it. Honestly I’m not sure if AI will kill us all, but it might make us wish at times that we were dead.