Among the many worries about artificial intelligence, the most dire is described in a new book: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All by Eliezer Yudkowsky and Nates Soares.
I have been extremely skeptical of the doom-and-gloomers but read this book prompted by two issues. AI is tremendously important and will certainly change the world in huge ways. And some AI experts who are both smart and knowledgeable worry about the risk of AI going wrong.
The book’s first part persuaded me that the risk was greater than I had previously thought. The authors describe this section: “… we lay out the problem, answering questions such as: What is intelligence? How are modern AIs produced, and why are they so hard to understand? Can AIs have wants? Will They? If so, what will they want, and why would they want to kill us? How would they kill us? We ultimately predict AIs that will not hate us, but that will have weird, strange, alien preferences that they pursue to the point of human extinction.”
The book uses parables, very well told, to argue that evolutionary processes are not predictable, at least not easily. As AIs improve themselves, they will evolve in ways that we cannot anticipate, and perhaps even the AI itself cannot anticipate. Their goals could be different than what the programmers’ goals for the program had been. We humans would, at some point, get in the way. We might be kept as pets, but more likely not important enough to devote resources to. I came away far more concerned than I had been before opening the book.
But Part II disappoints. It tells a story of one possible path that AI might take. Although humble about the details, the authors are convinced that they know the final outcome. (That approach is not stupid. If I play chess against a grand master, I cannot predict his moves but I can certainly predict that I will be beaten.) Their conclusion comes from this approach. But this section lacks “steel man” arguments.
Would AIs compete with one another, and would that save humanity? Yudkowsky and Soares dismiss this breezily, arguing that the first superintelligent AI would see an upcoming AI as threatening and either destroy it or capture it. Sure, maybe, but that’s not what humans have always done. Humanity has prospered more by cooperation and competition than by conquest. The authors could be right, but they did not convince me that they have taken the issue seriously.
Could an AI be “lobotomized” so that the superintelligent AI could deploy offspring that do not threaten the parent? (An interesting idea; Greek mythology has many tales of sons killing their fathers.) The authors describe the concept but have little confidence that humans might be able to do that to the AIs. So they think that AIs could spawn safe AIs, but humans cannot. More broadly, they assert that a superintelligent AI will evade any guardrails that its developers install.
The authors write about human efforts to keep AI tame: “Humanity only gets one shot at the real test.” Their assumption is that a superintelligent AI will show no sign of evil until a single do-or-die conflict is instigated. Humans, though, often test defenses in detectable ways. Maybe the AI won’t do that, but maybe it will. Perhaps the battle will not be one sudden event but a long, drawn out siege. That would make a difference to human prospects.
Pessimism is justified by the authors using a long list of technologies that went wrong, such as leaded gasoline. They fail to list the many technologies that older generations worried about but that proved tremendously helpful to humans. Writing was criticized for ending memorization; transatlantic voyage attempts were expected to end in death for the sailors; cell phones were expected to cause cancer. Of course, the long list of successful innovations does not imply zero costs to these technologies, nor does a long list of success imply that a disastrous new technology could not occur. But Yudkowsky and Soares demonstrate an affection for pessimism by cherry-picking cases of tech gone wrong. They neglect the tremendous gains in human well-being delivered by past technological development.
After reading the policy implications of the book’s Part III, one has to wonder about the “safe” world the authors propose. They assert, “All over the Earth, it must become illegal for AI companies to charge ahead in developing artificial intelligence as they’ve been doing.” Although they write “companies” at first, they clarify that they also mean governments and individuals. The major countries must prohibit—and inspect for compliance—AI activity in all parts of the world. If any place is exempted, then superintelligent AI will be developed there. For example, North Korea must cede AI authority within its own territory to some international body, using whatever threat or action is necessary. That will go well?
Government must also prohibit publication of research results on AI, they say, because research keeps making AI more efficient, and letting small systems get more powerful is bad. How far will the censorship go? How intrusive will be the monitoring? Must we worry about a teenager buying old graphics cards on eBay to create an AI? Should we monitor every household’s electricity use? Perhaps require home generators to be registered and logged?
If one believes Yudkowsky and Soares’ first two sections, the third will be quite disturbing in how severe the action will have to be to save humanity. The authors tell us that we must give up some freedom, and possibly nuke innocent people, in order to save the homo sapiens species. They may be right about what it would take. The fact that a tradeoff is very frightening does not mean it isn’t real. But if such policy can ever be justified, that justification must be proved beyond a shadow of a doubt.
The risk of AI going rogue does seem real. And perhaps that path can be deterred only by severe action. But although I am more concerned than before reading this book, Yudkowsky and Soares have failed to convince me that we need to take major action today. However, If Anyone Builds It, Everyone Dies is an important book that people should be discussing.