Émile Torres:   Back in the mid-2000s, I encountered transhumanism for the first time. And I was initially really critical of it. The second paper I ever published was a critique of transhumanism. But  then, certain considerations led me to believe that transhumanism is a defensible position, and I became a sort of transhumanist.

And one of the main considerations was that the development of these technologies is inevitable. So if you’re going to be against transhumanism, then maybe you need to be against the development of certain person-engineering technologies. But since they’re inevitable, there’s no point in opposing it just to hold back the tide. So the best thing to do is to join the transhumanists and do what you can to ensure that that project is realized in the most optimal way.

The notion of existential risk was tightly bound up with transhumanism from the start: existential risk was initially defined as ‘anything that might prevent us from creating a posthuman civilization’.

ÉT:  There was definitely talk of extinction and global catastrophe. But what’s new about this idea of existential risk — right there in the definition — is the idea of desirable future development. 

There were people, particularly in the second half of the 20th century, arguing that one reason human extinction would be bad is that it would foreclose the realization of all future progress, future happiness, and so on. But that lost-potential argument was never formalized. The focus was really on going extinct. Everybody on earth is going to die. You and me are going to die. Our families will die. That was the foreground. Lost potential was not prominent. 

The notion of existential risk, I think, flipped that around and foregrounded the lost potential: the argument became that the worst aspect of human extinction is the lost potential. It’s not the 8 billion people who are going to die. That’s very bad, but the badness of all the lost potential is orders of magnitude larger.

ÉT: That sounds right to me. I think this notion of potential is really central to longtermism. The initial definition comes from 2002, with Bostrom discussing the  transition dynamics from our human to a posthuman civilization, foregrounding the potential of becoming posthuman. This was also bound up with this notion that the creation of posthumanity isn’t just valuable because it’s a good way for cosmic history to unfold. But also, you and I might benefit, right?

So why is creating a posthuman civilization important (according to Bostrom and people like him)? Well, because if it happens within my lifetime, maybe I get to live forever. Or even if it happens within maybe a thousand years, I still get to live forever because I’ll sign up with ALCOR and get resurrected. So I really see this moment where there is a  sort of the pivot towards thinking about the far future. I think initially, for the transhumanists, it was bound up with their own fate as individuals. 

ÉT: My understanding of the history is that it was really in the 2000s that people in this community became increasingly aware of just how huge the future could be. With that awareness came a corresponding shift in the moral emphasis.

Yudkowsy wants to live forever. On Lex Fridman’s podcast, he said that he grew up believing that he would live forever. And so part of the trauma for him, as he mentioned on that podcast, is being in this situation where AGI is so close, and he’s having to face his own mortality, maybe for the first time. It seems like his thinking exemplifies this pivot throughout the 2000s.

ÉT: With respect to the attention economy, my sense is that longtermists recognize that this vision of the future is kind of nuts. I mean, some of them have used the term ‘crazy town’. Consequently, I think they do their best to avoid mentioning what their actual goals are publicly. Crazy ideas do grab the public’s attention, but in this case, I think they feel that some of these ideas are not good PR. 

ÉT: I don’t think AI in general is motivated by this ideology. The race to AGI is. And I think there are two factors. One that’s obvious is the profit motive. Microsoft and Google expect to make billions of dollars off of these large language models. But I think the other crucial component of  the explanatory picture is TESCREALism. 

It’s like… why did DeepMind form in the first place? Why did Demis Hassabis – who was at a lot of these transhumanist conferences – found it? And Shane Legg, who received $100,000 from the Canadian Singularity Institute for Artificial Intelligence after completing his PhD thesis, and gave talks at the Singularity Summit conferences.

ÉT: In terms of the ubiquity of TESCREAL beliefs, I think you’re right. A lot of them wouldn’t even use the word transhumanism. You could ask, “Would you support re-engineering the human organism?” Or ask, “Are you funding projects to try to re-engineer the human organism so we can merge AI into our brains?” I think a lot of them would say yes. And they’d be for aspects of the longtermist worldview like the imperative to colonize space and plunder the cosmos. My strong suspicion is that’s the water that these people swim in.

An article I want to write would be about the different terms and ideas that various authors use to describe the culture of Silicon Valley – using different terms, but ultimately describing the same phenomenon. So what I mean by TESCREALism is the same thing that far-right guy Richard Hanania calls the “tech right.”

There was a Huffington Post article about how he holds white supremacist views. And he said, “I hate the word TESCREALism.” So he called it the ‘tech right’. Douglas Rushkoff calls this ‘the mindset’ – he says it is everywhere in Silicon Valley among tech billionaires and so on; in talking to them about their views, he found that they all thought: “the future is digital. We’re going to upload our minds. We’re going to spread throughout space” and so on. What Rushkoff means by ‘the mindset’ is basically what I mean by TESCREALism. Would these people who embody ‘the mindset’ say, “yeah, I’m a longtermist, and I believe that the most morally important thing to do is to conquer space and create all these digital people in the future?” I don’t know. But their worldview looks a lot like longtermism.

ÉT: I think that’s largely correct. I think some people, like Wil Macaskill, [a figure in ‘effective altruism’] really accept this very abstract philosophical position that what matters is that there are huge numbers of people in the future. And a lot of tech billionaires see this vision as bound up with their fate as individuals. So the thinking is like… “I want to build a bunker to survive the apocalypse so I can get to space, have my mind digitized” and so on. And that definitely can lead to this disregard for most human beings. A wild example of this is the news that broke that Sam Bankman-Fried’s brother and somebody else at FTX had discussed the possibility of buying the island nation of Nauru explicitly so that members of the ‘effective altruism’ movement could survive an apocalyptic event that  kills up to – as they wrote in the document – 99.9% of human beings.

ÉT: I guess it depends on what one means by the Singularity. There’s the intelligence explosion interpretation… there’s the Kurzweilian idea that just has to do with the rate of change.

ÉT: I’m sympathetic with that view of the Singularity. There’s just not much we can say beyond it. I’m very skeptical of the intelligence explosion idea. And the rate of change idea from Kurzweil seems to be in direct and significant tension with the fact that a climate catastrophe is almost inevitable unless there’s some new technology that, at scale, removes carbon dioxide from the atmosphere.

ÉT: I think climate change is different than that.

ÉT: We’re talking about global civilization. Is it gonna survive? I don’t know. I mean, there are legit climatologists out there who don’t think it will unless there’s immediate action to avert catastrophic climate change.

I remember arguing, many years ago, with colleagues in the existential risk field, where I was claiming that climate change is a much bigger deal than they were suspecting. They thought: “We’ll invent AGI. And once we have AGI, it’ll…”

ÉT: Figure out what to do. But how are we gonna create AGI in a world that’s just crumbling and falling apart? How are we gonna host conferences on AI when the weather is so hostile that you can’t get there?

ÉT: It’s just not my area of expertise. But my sense is that, in terms of the LLMs that we have, there’s no obvious path from those systems like ChatGPT to genuine AGI or superintelligence.

ÉT: I think it was Gary Marcus who was laughing about the rise and fall in prestige, if you will, of ChatGPT. It became a joke line during a Republican debate.

ÉT: Yeah. So I don’t think that Singularity is going to happen, probably. And I would put money on it not happening soon, not happening in 2045 like Kurzweil predicts. 

ÉT: I recognize and affirm that. The notion of TESCREALism is supposed to capture the techno-utopian vision that came out of some of these movements, and to gesture at the worst aspects of that. I think they’re the ones that have become most influential now. So, like the democratic socialist James Hughes — he was somewhat influential. But, compared to someone like Bostrom, his influence is minor. And I absolutely recognize that there are transhumanists who like anti-libertarian. Some of them are Mormons.

ÉT: That’s my impression also. Obviously there’s a fascinating history involving futurism and fascism. Maybe it does tend to attract a certain type of person or lends itself to being interpreted or exploited by fascists. TESCREALism captures that aspect.

ÉT: There’s definitely that charismatic leader aspect to rationalism. There are these Less Wrong posts that are just lavishing praise on Yudkowsky. I remember seeing a list of one or two sentence statements about Yudkowsky. One of them was something about how “inside Eliezer Yudkowsky’s pineal gland is not an immortal soul, but another brain.” “In the history of Western thinkers, there was Plato, Immanuel Kant, Eliezer Yudkowsky.”
(Laughter)
Someone who I won’t name told me that the Bay Area rational scene is a full-grown apocalypse cult. 

I think EA (Effective Altruism) is sort of a cult. There was an article published by Carla Cremer recently. She talked about a secret competitive ranking system in which participants get points subtracted if they have IQs of less than 120.

ÉT: I’d be really curious to know. Because I do think that IQ realism is pretty widespread within this community. Bostrom has written that IQ is good but imperfect. So they sort of lean towards IQ realism.

ÉT: No, not really. Hanania acknowledges it in arguing that the ‘tech right’ is
a better term. He said we were basically right about what the streams are, what the bundle is, but ‘tech right’ is a better term. I’m not surprised that there aren’t more people coming out and saying they identify as TESCREAL.

ÉT: Yes. I do think that could be permissible under certain circumstances. I mean, I have worries about the limits of what that technology will be used for. Will it be used just for what we would intuitively call therapeutic purposes? My main concern is that it could easily open the door to an ‘enhancement’ approach. As soon as you’re talking about enhancements, there are questions like, “What criteria are you using to judge whether some modification is better?” That’s where you get into the issue of ‘super-classes’ which Bostrom has written about. 

A lot of that is probably ableist. What ‘enhancing’ means for somebody like Bostrom might be completely different than what I might mean. Right?

ÉT: My sense is that words like ‘broken’ are normative terms. I think disability rights activists or disability scholars have a point when they say, “there’s an infinite number of things that I can’t do.” I can’t fly. The notion of disability, some would argue, is just a social construct. We live in a society that is unaccommodating for somebody who can’t do something that a statistical majority of other people can do. That’s what a ‘disability’ is. So maybe, you fix society, the disability goes away even if the inability remains.

ÉT: There are two aspects to that. One is the inability and the other is the society one lives in. So you can fix or eliminate disability by improving society. And then there’s a good argument that a lot of the inabilities that we classify as disabilities would not be seen as bad. It’s just different. There are people of different heights. There are people who can’t walk. I find my heart is filled with a lot of empathy for the disability scholars — some of whom are disabled themselves — arguing that they wouldn’t want to change. And their view that we shouldn’t aim for a world in which people like them no longer exist.

ÉT: Not so much because I haven’t seen a lot of evidence, at least like my social media feed.

ÉT: To put it simplistically, a lot of the people in my circle are for good technology,  not bad technology. Maybe small-scale technologies, which doesn’t mean low-tech. So you can have advanced technologies that would enable somebody to get around who can’t walk. But without the megacorps, and all the control  and all the risk that comes with that.