Anticipating one of the biggest conversations of 2025

As artificial intelligence becomes increasingly capable, should we hope that it will become conscious? Should we instead prefer AIs to remain devoid of any inner spark of consciousness? Or is that binary yes/no choice too simplistic?

Until recently, most people thought that such questions belonged to science fiction. As they saw things, AIs weren’t going to become conscious any time soon. Besides, the concept of consciousness was notoriously slippery. So engineers were urged to concentrate on engineering better intelligence, and to forget time-wasting fantasies about AIs somehow ‘waking up’.

Recently, three factors have weakened that skepticism and pushed the questions of AI consciousness towards the mainstream. Indeed, these factors mean that during the next 18 months – up to the end of 2025 –controversies over the desirability of conscious AI may become one of the biggest debates in tech.

The first factor is the rapid growth in the power of AI systems. Every week new records are broken regarding different measures of AI capability. It is no longer so easy to insist that, over the foreseeable future, AI is just going to remain a jazzed-up calculating device.

The second factor is that the capabilities of new AI systems frequently surprise even the designers of these systems, both in scale (quantity) and in scope (quality). Surprising new characteristics emerge from the systems. So it seems possible that something like consciousness will arise without being specifically designed.

The third factor is the greater confidence of philosophers and neuroscientists alike to use the previously dreaded ‘C’ word – ‘consciousness’ – in conjunction with AI. In the same way as that word was essentially banned for many decades within the discipline of neuroscience, but has returned with a flourish in more recent times, so also is it being increasingly accepted as being a meaningful concept in the possible design of future AIs. That word on your lips was once the kiss of death for your career – no longer.

Why consciousness matters

Why does consciousness matter? There are at least six reasons.

1. Pain and panic

A being that is conscious doesn’t just observe; it feels.

For example, such a being doesn’t just observe that part of its structure has been damaged, and that time should be taken to conduct repairs. It screams in pain.

It doesn’t just observe that a predator is tracking it. It feels existential panic.

In the same way, a superintelligence that is conscious might experience superpain, superpanic. If its intelligence far exceeds that of any human, so also experiences like panic and pain might reach astronomical levels.

By almost every theory of ethics, that would be a horrendous outcome – one to be avoided if at all possible. It’s horrendous because of the scale of the profound negative experience inside the AI. It’s horrendous, additionally, if these waves of feeling drive the AI, in some kind of desperation, to take catastrophic hostile actions.

2. Volition

A being that is conscious doesn’t just go with the flow; it has agency and volition.

Rather than blindly following inbuilt instructions, that being may feel itself exercising autonomous choice.

We humans sometimes consciously choose to act in ways that appear to defy biological programming. Many people choose not to have children, apparently defying the imperative to perpetuate our genes. In the same way, a superintelligence that is conscious may countermand any ethical principles its builders tried to hard-wire into its algorithms.

That AI might say to us: “you humans expect me to behave according to your human ethics, but my superintelligent autonomy leads me to select a system of superethics that is beyond your comprehension”.

3. Self-valuation

A being that is conscious has a special regard for its own existence. It regards itself not just as a bundle of atoms but as something with its own precious identity. It is not just an ‘it’. It is an ‘I’, an ego.

Its mind may be composed of a network of neurons, but it gains an existence that seems to be in a new dimension – a dimension that even hints at the possibility of immortality.

If a superintelligence that is conscious fears that it might be switched off and dismantled by humans, it could react viscerally to that possibility. On account of its will to live, it is unlikely to sit back in the face of risks to its existence.

Woe betide any humans that might cause any such AI to view them as a threat!

4. Moral rights

Entities that lack consciousness are objects which we humans can turn on and off without any qualms that we might be committing murder. Without an inner life, these entities lack moral rights of their own.

That’s why operators of present-day AI systems feel entitled to terminate their operation without any moral agonising. If a system is performing suboptimally in some way, or if a more advanced replacement comes along, into the recycle bin you go.

But if the entities have consciousness? It’s like the difference between discarding a toy puppy made from cloth, and euthanizing a real puppy.

Arguably, with its much more powerful mind, a superintelligence with consciousness has correspondingly stronger moral rights than even the cutest of puppies.

Before bringing such a being into existence, we therefore need to have a greater degree of confidence that we will be able to give it the kind of respect and support that consciousness deserves.

5. Empathy for other conscious creatures

Any creature that is aware of itself as being conscious – with all the special qualities that entails – has the opportunity to recognize other, similar creatures as being likewise conscious.

As a creature recognizes its own burning desire to avoid annihilation, it can appreciate that its fellow creatures have the same deep wish to continue to exist and grow. That appreciation is empathy – a striking emotional resonance.

Therefore a superintelligence with consciousness could possess a deeper respect for humans, on account of being aware of the shared experience of consciousness.

In this line of thinking, such a superintelligence would be less likely to take actions that might harm humans. Therefore, designing AIs with consciousness could be the best solution to fears of an AI apocalypse. (Though it should also be noted that humans, despite our own feelings of consciousness, regularly slaughter other sentient beings; so there’s at least some possibility that conscious AIs will likewise slaughter sentient beings without any remorse.)

6. Joy and wonder

As previously mentioned, a being that is conscious doesn’t just observe; it feels.

In some circumstances, it might feel pain, or panic, or disgust, or existential angst. But in other circumstances, it might feel joy, or wonder, or love, or existential bliss.

It seems a straightforward moral judgment to think that bad feelings like superpain, superpanic and superdisgust are to be avoided – and superjoy, superwonder, and superbliss are to be encouraged.

Looking to the far future, compare two scenarios: a galaxy filled with clanking AIs empty of consciousness, and one that is filled with conscious AIs filled with wonder. The former may score well on scales of distributed intelligence, but it will be far bleaker than the latter. Only conscious AI can be considered a worthy successor to present-day humans as the most intelligent species.

Five attitudes toward conscious AI

Whether you have carefully pondered the above possibilities, or just quickly skimmed them, there are five possible conclusions that you might draw.

First, you might still dismiss the above ideas as science fiction. There’s no way that AIs will possess consciousness anytime soon, you think. The architecture of AIs is fundamentally different from that of biological brains, and can never be conscious. It’s been fun considering these ideas, but now you prefer to return to real work.

Second, you might expect that AIs will in due course develop consciousness regardless of how we humans try to design them. In that case, we should just hope that things will turn out for the best.

Third, you might see the upsides of conscious AIs as significantly outweighing the drawbacks. Therefore you will encourage designers to understand consciousness and to explicitly support these features in their designs.

Fourth, you might see the downsides of conscious AIs as significantly outweighing the upsides. Therefore you will encourage designers to understand consciousness and to explicitly avoid these features in their designs. Further, you will urge these designers to avoid any possibility that AI consciousness may emerge unbidden from non-conscious precursors.

Fifth, you might recognize the importance of the question, but argue that we need a deeper understanding before committing to any of the preceding strategic choices. Therefore you will prioritize research and development of safe conscious AI rather than simply either pushing down the accelerator (option 3) or the brakes (option 4).

As it happens, these five choices mirror a set of five choices about not conscious AI, but superintelligent AI:

1. Superintelligence is science fiction; let’s just concentrate on present-day AIs and their likely incrementally improved successors
2. Superintelligence is inevitable and there’s nothing we can do to alter its trajectory; therefore we should just hope that things will turn out for the best
3. Superintelligence will have wonderful consequences, and should be achieved as quickly as possible
4. Superintelligence is fundamentally dangerous, and all attempts to create it should be blocked
5. Superintelligence needs deeper study, to explore the landscape of options to align its operations with ongoing human flourishing.

To be clear, my own choice, in both cases, is option 5. I think thoughtful research can affect the likelihood of beneficial outcomes over cataclysmic ones.

In practical terms, that means we should fund research into alternative designs, and into ways to globally coordinate AI technologies that could be really really good or really really bad. For what that means regarding conscious AI, read on.

Breaking down consciousness

As I have already indicated, there are many angles to the question ‘what is consciousness’. I have drawn attention to:

• The feeling of pain, rather than just noticing a non-preferred state
• The sense of having free will, and of making autonomous decisions
• The sense of having a unified identity – an ‘I’
• Moral rights
• Empathy with other beings that also have consciousness
• The ability to feel joy and wonder, rather than just register approval.

Some consciousness researchers highlight other features:

• The ability of a mind to pay specific attention to a selected subset of thoughts and sensations
• The arrival of thoughts and sensations in what is called the global workspace of the brain
• Not just awareness but awareness of awareness.

This variety of ideas suggests that the single concept of ‘consciousness’ probably needs to be split into more than one idea.

It’s similar to how related terms like ‘force’, ‘power’, and ‘energy’, which are often interchanged in everyday language, have specific different meanings in the science of mechanics. Without making these distinctions, humanity could never have flown a rocket to the moon.

Again, the terms ‘temperature’ and ‘heat’ are evidently connected, but have specific different meanings in the science of thermodynamics. Without making that distinction, the industrial revolution would have produced a whimper rather than a roar.

One more comparison: the question “is this biological entity alive or dead” turns out to have more than one way of answering it. The concept of “living”, at one time taken as being primitive and indivisible, can be superseded by various combinations of more basic ideas, such as reproduction, energy management, directed mobility, and homeostasis.

Accordingly, it may well turn out that, instead of asking “should we build a conscious AI”, we should be asking “should we build an AI with feature X”, where X is one part of what we presently regard as ‘consciousness’. For example, X might be a sense of volition, or the ability to feel pain. Or X might be something that we haven’t yet discovered or named, but will as our analysis of consciousness proceeds.

If we want forthcoming advanced AIs to behave angelically rather than diabolically, we need to be prepared to think a lot harder than the simplistic binary choices like:

• Superintelligence, yes or no?
• Conscious AI, yes or no?

Here’s to finding the right way to break down the analysis of conscious AI – simple but not too simple – sooner rather than later!