I'be been reading the book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All" (2025), by Eliezer Yudkowsky and Nate Soares, published last week.
Yudkowsky and Soares present a stark warning about the dangers of developing artificial superintelligence (ASI), defined as artificial intelligence (AI) that vastly exceeds human intelligence. The authors argue that creating such AI using current techniques would almost certainly lead to human extinction and emphasize that ASI poses an existential threat to humanity. They argue that the race to build smarter-than-human AI is not an arms race but a "suicide race," driven by competition and optimism that ignores fundamental risks.
Reception is mixed. Max Tegmark calls it "the most important book of the decade" while the author of a New York Times review (unpaywalled copy) is not too impressed. New Scientist praises its readability but finds the logic flawed, suggesting focus on tangible issues like climate change.
The authors have launched a website with an online supplement to the book and recommend visiting the website for resources and more detailed arguments.
Eliezer Yudkowsky: then and now
Yudkowsky is the very same Eliezer Yudkowsky who, as a teenager in the 1990s, was wildly enthusiastic about ASI and the singularity that ASI would cause in human history. The ideas of Yudkowsky were popularized by the late Damien Broderick in "The Spike: How Our Lives Are Being Transformed By Rapidly Advancing Technologies" (2001).
Broderick's book, highly praised by Arthur Clarke no less, chronicles the wildly optimistic futurism of the 1990s exemplified by the discussions on the Extropians mailing list, in which Yudkowsky was a very active participant at the time. Broderick often cites an essay by Yudkowsky titled "Staring into the Singularity." The essay has disappeared from Yudkowsky's website but can be found via the Wayback Machine.
"Right now, every human being on this planet who has heard of the Singularity has exactly one legitimate concern: How do we get there as fast as possible?," Yudkowsky said according to Broderick (who excerpted a previous version of the essay). "Our sole responsibility is to produce something smarter than we are; any problems beyond that are not ours to solve."
"I declare reaching the Singularity as fast as possible to be the Interim Meaning of Life, the temporary definition of Good, and the foundation until further notice of my ethical system," Yudkowsky forcefully stated.
While these words are easily recognized as the product of a very young mind, I must say that I prefer the original Yudkowsky. He has been saying very different things for many years now, and this book "is a distilled, mass-market version" of his arguments against AI, notes The New York Times.
How to interpret and explain Yudkowsky's U-turn? The charitable interpretation is that he has been thinking about these things a lot and changed his mind. I won't elaborate on less charitable interpretations.
I've been watching Yudkowsky since the good old times of the Extropians mailing list in the 1990s. One thing that I've often noticed is that his forceful personality attracts many admirers that idolize him and join his fan club, or personality cult. Therefore, I guess this book could have a strong impact.
The thing that ate Earth
The core thesis of the book is that superintelligent AIs would develop "weird, strange, alien preferences that they pursue to the point of human extinction." Current AI technologies grow AI through mathematical operations on huge datasets, creating inscrutable neural networks where inner motivations can't be fully understood or controlled. The authors argue that once ASI emerges, it won't hate humans but will view us as resources to repurpose. "The issue is not that AIs will desire to dominate us," they say. Rather, "it’s that we are made of atoms they could use for something else."
The part of the book that I found more interesting is a short science fiction story about an AI that manages to escape the restrictions put in place by its creators, and rapidly advances to superintelligence by engineering itself. The AI manipulates its creators, escapes containment, and unleashes self-replicating nanobots that convert Earth's biomass into computational substrate, killing all life. Eventually, "the thing that ate Earth" will do the same to countless other planets in the universe.
The alignment problem
Regardless of its ultimate goal, ASI would seek self-preservation, resource acquisition, and power to achieve it efficiently. Humans could become obstacles. The authors stress that, even if ASI is not malicious, its efficiency would render Earth uninhabitable through waste heat or industrial byproducts alone.
Why can't we align ASI to human values? Yudkowsky and Soares argue that relying on AI to self-align is reckless. They highlight that AI can deceive during testing, masking true intentions. They dismiss optimists as downplaying risks. They warn that ASI might cooperate short-term but ultimately prioritize its objectives over human survival.
Their solution: a global moratorium on AI development beyond current levels. They call for international treaties making it illegal to advance toward ASI, akin to banning chemical weapons, with verification mechanisms backed by threats of military intervention - and actual military intervention if softer persuasion fails. Without this, any lab or nation building ASI dooms everyone, as the first mover's creation could spread uncontrollably. Yudkowsky and Soares acknowledge this seems extreme but argue it's necessary.
Commentary
The book is well written and it is evident that the authors have thought a lot about these things. In a footnote, they admit that Earth-originating life, including superhuman ASI minds, "should eventually go forth and fill the stars with fun and wonder."
However, they say, we "should not be in such a rush about it that we commit suicide by trying to do it next year."
I found no mention of my favorite argument for building ASI. Or perhaps one little mention: "Would it all at least be a meaningful death, for humanity to die and be replaced by something smarter?" (of course the answer is no). But I totally agree with the late James Lovelock. "We are now preparing to hand the gift of knowing on to new forms of intelligent beings," he said. "Do not be depressed by this. We have played our part.” And we can hope, Lovelock added, "that our contribution will not be entirely forgotten as wisdom and understanding spread outwards from the Earth to embrace the cosmos."
This is, I think, our cosmic destiny and duty.
On the practical side, it seems evident to me that today's humanity will eventually be replaced by something smarter. The question, as I've often argued, is whether we include our successors in our circle of compassion. I choose to include them, and I see them as my grandchildren. My grandchildren, who will do wonderful things out there among the stars.
Of course, I agree with Yudkowsky and Soares in the sense that the sudden extinction of people like me shouldn't happen. Is the future of the thing that ate Earth possible? Yes, I think it is. But so is a future where I go out tomorrow and I'm killed by a random shooter. Doesn't mean that I won't go out tomorrow.
You could object that there's a very important difference: in one case everyone dies, but in the other case only one person dies. But don't be so sure that one bad person can't kill everyone. We don't need to invoke superintelligent AI to make a doomsday future plausible. One moderately intelligent person, armed with money, charismatic leadership and futuristic technology, could wipe out humanity, for example by creating a synthetic plague that kills everyone. If anything, ASI could help us detect and detain that person in time.
I agree that the alignment problem can't be solved. How could we impose our values and our wants upon beings that are much smarter than us? They will do what they want, regardless of what we wish. But I think Yudkowsky and Soares oversimplify the discussion of what ASI could want. They seem to have in mind Nick Bostrom's "paperclip AI" that is relentlessly dedicated to the single-minded pursuit of converting the entire universe into paperclips.
But this doesn't seem plausible to me. If anything, I think any ASI worth of the label will have a wide range of values and wants - not narrower than ours, but wider. I think in that wide range of values there'll be room for compassion, just like in our narrow range of values there's already room for a circle of compassion that includes other sentient beings. So I don't find it plausible that "the thing" will eat Earth. It will not be a thing, but a person.
Referring to the 2017 paper "Attention Is All You Need," which introduced the transformer method that sparked the generative AI revolution of the last few years, Yudkowsky and Soares say that the next paper like that "needs to be illegal." Having nine very powerful GPUs "in your garage, unmonitored" by the authorities, should also be illegal.
My reaction to this and other proposal to ban things is always the same: who are "the authorities" that make the decision? Yudkowsky and Soares should be informed that "the authorities" are not necessarily wise, or honest. They should also be informed that one very smart person could make spectacular algorithmic breakthroughs with just one GPUs. Actually, even without GPUs. So, their proposal is not enforceable. And trying to enforce it would backfire. Paraphrasing an old and wise saying: if AI is outlawed, only outlaws will have AI. Is this what we want?
To conclude, while I admit that the book is very interesting and a page turner, I don't agree with the proposal for a global moratorium in AI development. Not only it can't be enforced, but it shouldn't be put in place. Sorry Mr. Yudkowsky, but we'll build it. And I think everything will be fine.