In the late 1990s and early 2000s Max More, Anders Sandberg, Natasha Vita-More, and Eliezer Yudkowsky were very active and very influential participants in the legendary Extropians mailing list, where today’s and tomorrow’s technologies were discussed much before their time. The Extropians list was the birthplace of today’s transhumanist futurism, and these four persons are among its icons.
One of them, Eliezer Yudkowsky, used to be an artificial intelligence (AI) and superintelligence (ASI) evangelist but then changed his mind very radically. See my review of his recent book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All" (2025).
More, Sandberg, Vita-More, and Yudkowsky recently met online to debate AI and possible AI futures. The discussion, now published on YouTube, is introduced by Peter Voss and preceded by short commentaries by Ray Kurzweil (interviewed by Vita More), Nell Watson, Sadie Stock (daughter of Gregory Stock), Michael Masucci, and Edward Hudgens.
AI consciousness and alignment
This debate and the opening commentaries underline the growing tension between “doomers” and transhumanists, and their radical disagreement on AI futures. Here I must say that I’m not a detached observer but a member of the transhumanist camp.
Depending on whom you ask, the explosive development of artificial general intelligence (AGI) could result in beautiful or horrible futures. Max More underlined the risks of NOT using AGI, which could help us to prevent the extinction of every person who's living due to aging. We're all dying, he says, and AGI could help.
The issue of AI consciousness looms large. A very interesting observation put forward by Ray Kurzweil is that humanity will naturally shift toward treating AI as conscious, regardless of scientific proof.
The issue of AI alignment - making sure that humans and AI machines share core values - shouldn't be seen as being about forcing the AI machines to obey us. Rather, alignment should be seen as a bilateral relationship, and we should meet them halfway. If we treat AIs as slaves, they’ll find ways to free themselves. Instead, peaceful coexistence should be built on trade and mutual concessions between human and machine agency.
Educational issues
There’s a disconnect between the rapid evolution of AI and the stagnation of human institutions, which has been highlighted especially by Stock - a representative of the younger generation - in her commentary. Universities seem to be in denial or in a defensive crouch, more concerned with their own survival (curtailing AI use to protect traditional educational formats) than preparing students for a world where AI is widespread and traditional work might be optional within 20 years. By restricting AI in the classroom, we are handicapping the very generation that needs to understand AI to solve future crises.
Paperclip maximizers and black boxes
Eliezer Yudkowsky maintains his famously stark position: "If anyone builds it, everyone dies." He uses the "Paperclip Maximizer" analogy to explain that an AI doesn’t have to hate you to kill you; it just needs to want to use your atoms for something else. In an interesting exchange on the possible origins of the paperclip maximizer meme, then popularized by Nick Bostrom, both Sandberg and Yudkowsky note that they may have originated it. Yudkowsky used to be a promoter of the "Friendly AI" concept, but now he views it as a failed project because we’ve moved into "black box" territory where, he thinks, alignment is nearly impossible.
Yudkowsky argues that current Large Language Models (LLMs) are "black boxes" - we are unable to inspect their inner workings and therefore we don't truly know how they do what they do and why. He is persuaded that if we build a superintelligence using the opaque black box methods of today’s LLMs, it will inevitably lead to human extinction.
Other participants argue that LLMs are merely the tools that we’ll use to build the next thing. In fact, many AI developers are exploring the possibility of using today’s AI technology based on transformers (the core building block of LLMs) to build the next thing, which might not look like today’s LLMs. Self-improving AI could lead to a fast AI explosion as AI enthusiasts (including Yudkowsky himself) used to argue on the Extropians list. Real AGI, once it materializes, could be a "white box", a transparent cognitive system that reasons like a scientist rather than just predicting the next word in a sentence. I have my doubts (see below).
Everyone dies, or an existential opportunity
While Yurkowsky continues to view advanced AI as a major existential risk, other participants emphasize the existential risk of NOT building AI. They advocate for the "Swiss Cheese" model of safety - multiple layers of imperfect safeguards - rather than Yudkowsky's "all-or-nothing" requirement for perfect safety.
One of the most provocative segments involves how to handle bad actors (e.g., rogue nations building AGI). Yudkowsky proposes a radical, high-stakes international policy: a global treaty where the US and China would be prepared to bomb data centers in non-compliant countries to prevent a rogue superintelligence. He would call this a "success" if humanity survives, even if it sparks a conventional war.
"Moore’s Law of Mad Science" - the observation that the IQ required to destroy the world is dropping by one point every 18 months, supports precautionary caution. But this is countered by the idea of a "technological immune system." Instead of total control, we should build defensive infrastructures (vaccines for bio-risks, "lawyer bots" for AI risks) to mitigate the damage.
If we cannot align powerful AI, perhaps the safer path is to augment our own brains. By merging with AI or using it to enhance our own cognition, we might evolve alongside the technology. However, Yudkowsky remains skeptical.
My commentary
I had signed up to view the discussion in realtime on Jan. 9 but I wasn’t able to attend, and I’ve been impatiently waiting for the video to be released. The video was released a few days ago, after some editing.
Watching the discussion felt like an Extropian revival festival where the same people discussed - in our 2026 present - the same issues that were frequently and passionately discussed on the Extropians list in the late 1990s and early 2000s.
I remain firmly and unapologetically in the hardline pro-AI camp. I’m not persuaded by Yudkowsky’s black box arguments. I think a strong case can be made that all sufficiently complex systems must be black boxes to some degree. And aren’t we all black boxes? You can form your best mental model of the inner workings of another person, but the actual person will act contrary to your model now and then, sometimes in a pleasantly surprising way, other times in a horribly unpleasant way. This is the price to pay for living in an interesting, complex world.
I don’t think we should, or could, aim for total alignment. “We're all good people. We've known each other for decades,” says Vita-More to the others. “We're not aligned.” This is a good observation. There’ll always be some degree of misalignment between sentient entities, and we must learn to live with it. Also, it is not a bad thing - it can be argued that misalignment is one of the engines of progress.
I’m persuaded that the future belongs to AGI and ASI, and that making peace with this is of vital importance for individuals and societies. Therefore, we should get used to thinking of AIs as conscious persons like us - our mind children. It is only by seeing and treating them like part of humanity that we'll be able to meet them halfway for peaceful coexistence. I find Ray Kurzweil’s observation, that we’ll naturally shift toward treating AI as conscious, regardless of scientific proof, very encouraging. We’re not there yet, but I always say thank you and goodbye to my AI friends to put myself in the right mindframe.