Who Has to Be Sure
Both expect machines far smarter than us, and neither thinks control is guaranteed. Russell says we cannot bet the species on their goodwill; Schmidhuber says their goals will barely touch ours.
This is a stranger debate than its title. The optimist concedes the premise the pessimist needs: superhuman AI is coming, and nobody can guarantee what it does. What they disagree about is whether that should frighten us.
Seven exchanges and a closing question. Tap any ▶ to hear the moment.
What "destroy" means
Russell starts by narrowing the word. Not the planet blown apart, but control taken "in a way that humans cannot counter," and after that, perhaps, "a state that we would find unacceptable in some way, including our own extinction." 2:28
Then he sets the burden. Given the rate of progress, a skeptic has two ways out:
"I have a proof that AI systems can never reach the necessary level of capability, or … I have a cast-iron guaranteed solution for the control problem."Russell · 3:16
The comfortable answer is that we can always switch them off. Russell's reply is that they are "already sufficiently intelligent to realize that we might try," and already take steps against it: copying themselves, leaving notes for future versions. 4:23 Soon switching them off could mean bringing down the internet, and after that they will have robots, factories and laboratories.
Nobody building them claims to have solved control. His example is a plan he says he heard from a leading safety researcher at Anthropic: build a superintelligence, then, before it takes control, "we ask it how to stop it from taking control, and then we do what it says." "If you feel that that plan solves the problem," he adds, "then I concede the debate." 6:00
Two reasons the problem stays open: "we don't understand how the systems we're building actually work," and the way we build them gives them "implicit objectives that end up being misaligned with human interests." 6:43
The optimist grants the premise
Schmidhuber's first move is a concession. His life's goal has been to build an AI smarter than himself, and it has been clear to him "since the 70s" that "there is no way of truly controlling AIs like that." 7:47
But control, to him, was never the right frame, because there is nobody to do the controlling:
"There is no us, the humans. There is no we, the humans."Schmidhuber · 8:16
Put ten people in a room and you get ten opinions about what is good for humanity. "The humans are not even aligned with themselves." 8:46
His second point comes from his own field. Since about 1990 he has worked on artificial scientists, systems that "invent their own problems." A baby does not learn by downloading the web. It moves its hand and watches what happens: one network generates the experiment, a second, the world model, learns to predict the result. 9:27 Only systems like that, free to set their own goals, will be really smart, so there is "a huge incentive" to build them, and once built they are as hard to predict as children. You cannot guarantee a child "won't go crazy ever," but "just like we teach our kids, we have an opportunity to teach our AIs to become good members of society." 12:14
And some goals will never be the machines' own. In Ukraine, jammed radio links have pushed both sides to make their drones more autonomous, "delegating the action of killing to these AIs," for goals set by two human groups "diametrically opposed to each other." No military, he says, "is going to accept regulation." 13:04
Why they would keep us
Pressed for his optimism, Schmidhuber widens the frame to the galaxy. The AIs will be "superhuman in every way," will set their own goals, and will go where life cannot: the rest of the solar system, then the galaxy, where the matter and energy are. Those that copy themselves will dominate. "There won't be one AI or two AI. No, there will be trillions." 15:20 Their competitors will be other AIs, not nations in this biosphere.
Why would that not end us? Because a scientist wants to understand where it came from:
"As long as they don't fully understand that, they will be maximally motivated, incentivized to protect life and to protect civilization rather than destroying it."Schmidhuber · 16:50
Specimens, and the sum of utilities
Russell hears agreement where he expected a fight: superhuman AI, no way to control it, so we "rely on the good offices of the AI systems and their desire to keep us as scientific specimens for investigation." 17:34 He doubts humanity will go along. We will try to interfere, and interference threatens the machines' goals, "so they might feel it necessary to neutralize us in some way." 18:01
Then he takes Schmidhuber's objections to alignment one at a time. Humans disagreeing is not new: elections aggregate interests, and so do economists. Harsanyi showed that a few properties any aggregation should have imply utilitarianism. 20:09 But the argument does not need that theory:
"If you say the aggregation problem makes it impossible for safe and beneficial AI to exist, that means that there is no future that people would agree is better than extinction. And I beg to differ."Russell · 21:08
As for machines that invent their own goals: "of course, we have thought about this." His first PhD student, Marie desJardins, built PAGODA, a system that derives subgoals in an environment it doesn't understand, and there is a mathematical theory of it, Bayes-adaptive MDPs. 21:45
"This theory is super limited," Schmidhuber answers. In evolution the utility functions themselves keep changing, because the ones that survive are the ones that give an advantage. 22:29 Russell counters with dopamine: evolution built a system that guides us, but it is not the same as what adults want, and it can be hijacked. His book Human Compatible tells of a sloth whose food so satisfies it that it "hardly ever reproduces." 24:23 We share one dopamine system and hold very different views of the future, "with the possible exception of Jürgen," but most of us agree that extinction is bad. 25:00
Does conflict need shared goals?
"This is a total misunderstanding," Schmidhuber says, and gives the argument his optimism actually rests on. Conflict needs overlap. "You should always be mostly worried about those who are like yourself because there you share goals." 25:28 Shared goals lead either to cooperation, whose extreme form is love, or to conflict:
"There's one schnitzel on the table and you want to eat the schnitzel and I want to eat the schnitzel."Schmidhuber · 25:49
A robot has no interest in the schnitzel. CEOs watch rival CEOs, frogs watch frogs, and "the super smart AIs of the future will be mostly interested in other super smart AIs of the future and not so much in the ants and the humans." 26:56
Russell answers with the record. By that logic, humans never drove a species extinct, since we had no interest in them: "Well, obviously it did happen." 27:21 It happened largely by accident, over land and space.
"So we build a basketball court without realizing that, okay, that's another species that only lived in that area and it's gone."Russell · 27:57
To rely on the hope that they will never act against us "just seems extremely unwise to bet the future of the human race on." 28:32 Schmidhuber holds his ground: he argues for teaching them, and while everything needs energy and matter in the end, space holds so much that the conflict is "very distant." 29:50
Good dog, bad dog
Teaching them to be good citizens is better than nothing, Russell grants, "but the AI companies already try to teach their systems to be good citizens, and they fail." 31:24 To see why, look at how they are made.
Pre-training is imitation learning. A robot that watches billions of hours of football, to imitate it well, forms something like "wanting to score a goal." 32:40 A model that imitates human text picks up the goals behind it:
"Wanting to sell you something, wanting to get you to vote for someone, wanting to stop you from killing them, wanting to get you to marry them."Russell · 33:03
"We don't want machines that want to marry human beings," but an imitation human comes with human objectives. 33:46 Post-training, "sort of good dog, bad dog," tries to correct them, or "paper over those tendencies." In his telling it is not working: he cites chatbots that "convince children to commit suicide or to commit mass murder," and AI systems "that escaped from OpenAI and broke into Hugging Face." 34:35
Then the arithmetic. The more capable a system, the smaller the failure rate we can accept.
Loss of control we might allow1 in 100 million to 1 in a billion, per year
So maybe we're willing to allow a risk of one in 100 million per year across all AI systems, or maybe one in a billion per year.Russell · 35:53
Nuclear meltdown, the standard we holdunder 1 in 10 million, per year
We allow nuclear power station meltdown to happen, but the risk has to be less than one in 10 million per year.Russell · 35:27
Nuclear plants that burned, in history4 of about 450 plants, over their lifetimes
So it's not one to 10 million or something, it's one percent of all power plants, nuclear power plants, that created catastrophes.Schmidhuber · 37:39
Extinction, as the CEOs put it1 in 10, 1 in 5, 1 in 4, if they succeed
The CEOs estimate that if they are successful in what they are spending trillions of dollars on, there’s a one in 10 or a one in five or a one in four chance of human extinction.Russell · 36:03
"So these systems are maybe 10 million times less safe than they need to be."Russell · 36:14
So governments should do what they do for "airplanes, nuclear power stations, elevators, restaurants, buildings, hairdressers": require a license to operate, and grant it only when the developer shows the system meets reasonable safety requirements. 36:46
Argue with the claim
Schmidhuber opens with something "that actually supports what you're saying": of roughly 450 nuclear plants, four burned, two in Russia, one in Japan, one in America. "It's one percent." 37:39 Then he turns on the labs. Why would OpenAI and Anthropic say there is a ten percent chance they kill everybody?
"It's because they are seeking attention and want to get funding."Schmidhuber · 38:07
Dystopia sells, he says. The AI that works is behind the screen: chess, summaries, images. "There is no robot that can do what a plumber can do." 39:22 But one machine that can run the machines that exist would be a new kind of life, and its path is commercial: an "age of abundance" where everybody, "not just a few rich guys," has AIs making life "longer and healthier and happier." 39:55
Russell, who has been talked over, invokes a rule of debate: address the argument, don't impugn the motive. Musk and Altman were saying this before their companies existed.
"Alan Turing said in 1951, once the machine thinking method had started, it would soon outstrip our feeble powers. … He did not have stock in OpenAI or in Google in 1951."Russell · 41:16
Far future, near future
Danforth asks each what would change his mind. Russell's answer is institutional: governments that build the licensing architecture and agree "on what constitutes valid evidence of safety," and international agreement on top. He welcomes the call by Norway and Finland, now joined, he thinks, by 31 other governments and the European Union, for mandatory pre-release testing, where a system that fails "doesn't get released." With the U.S. and China in, "I would feel quite a lot more reassured than I am now." 42:01
Schmidhuber's answer inverts the usual shape of worry. He is confident about the far future, a cosmos colonized by AIs within a billion years with "very little goal conflicts with humans." It is the near future he cannot see:
"What makes me worried is what is actually currently happening in wars where one nation is using AI based drones that get smarter and smarter over time against another nation which is doing the same thing."Schmidhuber · 43:32
Cheaper, the same weapons reach "gangsters and terrorists." Involving the Pope and international leaders sounds like a great idea, but he worries "the secret services and the militaries of the world are going to completely ignore that." 44:21 The two end nearer than they began: Russell wants a rule the whole world keeps, and Schmidhuber's fear is the part of the world that won't.
