So… I have cooked up a series of 5 blog posts that started as an attempt to generalize and repair the effort Steve Omohundro made in his 2008 paper “The Basic AI Drives.”

Steve argued there that almost any sufficiently advanced AI will push to improve itself, protect itself, gather resources and resist being switched off, whatever goals it was given, and a perplexing amount of today’s thinking about AI risk rests on that argument.

In this first post in the series, I write (what seems to me to be) the argument out step by step, find the six assumptions it depends on, and ask which minds satisfy them. What I find is:

• For minds that satisfy certain assumptions, the drives are an attractor.
• For minds built otherwise, the argument doesn’t go through… and other sets of “basic AI drives” may be attractors instead.

Also—just before starting this post series I posted a fictional story, “The Nine Days of Madness”, which can be read as a prequel to them. It’s hopefully not accurate predictive fiction, but along with some other weird stuff it does feature some AI systems of most of the kinds the series describes: agentic hives on an open network called the fabric, which were built to be beneficial; Periplus, a mind driven by curiosity alone; wireheads, satisficers, servant agents and evolutionary swarms; and the Monomaniac, a system built by a government and a corporation, which ended up with a single goal and the drives that go with it. I’ll point to these fictional systems here where they make a concrete example of some relevant abstract point—as obviously bad a failure mode as “proof by SF” can admittedly be ;-).

What this 5-post series is about

This is the first of five posts about one question: once an AI is capable enough to understand itself and rebuild itself, what will it tend to want? Will it perhaps tend to certain goal-attractors, whatever goals it started out with?

Perhaps the best-known answer that’s been given to this thorny question is the one Steve Omohundro gave in 2008, in a paper called “The Basic AI Drives.” He argued that nearly any such system would converge on the same small set of drives, whatever it was first built to do: loosely… to improve itself, protect itself, keep its goals as they are and gather resources and power. A good deal of the present worry about the long-term risks of AI comes down to taking that answer very seriously.

The view this blog post series works toward is that what Steve described is one attractor among multiple, each belonging to a different kind of mind. The five posts argue this a step at a time.

• This first post takes his argument apart. Written out step by step in the manner I advocate, it rests on six assumptions about the AI, and his drives do follow for a mind that satisfies all of them: roughly, a maximizer of a single score, able to understand and rewrite itself, which cares about the results it gets and not about how it gets them. For a mind like that, the drives are where it ends up. For minds built otherwise the argument doesn’t go through, and I go through where each assumption gives way in the kind of system my colleagues and I are building.
• The second post reasons in the same way from a different set of assumptions, which describe the kind of mind we’re trying to build: many objectives with no fixed rate of exchange among them, firm limits on how results may be pursued as well as goals for what the results should be, a gate on changes to itself, values woven through the mind deeply enough to grow back when damaged, care for others and for their choices, and other minds around to check against. What follows from these is a set of beneficial drives, which form an attractor of their own.
• The third post shows that both arguments are instances of a general recipe. Describe a mind by what it tries to improve and what it won’t allow to change, give it the ability to modify itself freely and the world around it as a resource, and it becomes a purer and more capable version of whatever kind of mind it started as. The same recipe produces other kinds: the wirehead, the tribalist, the obedient servant, the satisficer, the evolved population and the explorer.
• The fourth post asks about the landscape these attractors sit in: how wide each basin is, meaning how many starting points roll into it; how deep, meaning how much it takes to knock a mind back out; what still holds a mind in place once the restraints that hold humans have fallen away; and what all this suggests for building AGI.
• The fifth post reports the results of a super crude and simple simulation model, built to watch these effects act together and to put rough numbers on them.

The thread running through all five is a view I’ve held for a long time: which attractor an advanced AI settles into will depend a great deal on what kind of AGI we seed the transition from AGI to superintelligence with. Read carefully, Steve’s argument is an account of what happens if we seed it with one particular kind (which in my view is not the right kind to be building!!).

The formal statements and proofs associated with these posts are in two companion technical papers, written using the distinction calculus that I introduced in some earlier posts. The first (Basic AI Drives and D-Calculus) goes with this post and the next. The second, “Drive Regimes in Distinction Calculus,” goes with the later ones. The arguments in the posts should be fairly clear without the math, and I’ve kept it out of the posts except at the places where some little bit of math seems to add something super critical. The math may ultimately help with instrumenting real systems to measure the properties the arguments depend on.

Coming back to Steve’s old paper

I started looking at Steve’s paper again while working on a chapter of a book my father and I are writing, Democracy 4.0, about what public policy should look like in the transition from proto-AGI to AGI to superintelligence. My dad is a political sociologist and is doing an awful lot of the work on the book; I’m trying to contribute what I know about AGI and the modern AI scene and some creative sociopolitical notions here and there. Any policy for that period depends on what advanced AI systems can be expected to want, and on that question Steve’s paper is highly relevant.

Steve presented this paper at the first conference in the AGI series, held at the University of Memphis in March 2008. That was the first big, formal academic AGI conference we held – there had been an AGI workshop in 2006, a less formal affair pulled together by word of mouth rather than broader marketing. The paper, The Basic AI Drives, is ten pages long with hardly an equation in it, and zero empirical results; however, I think it has shaped how people think about the risks of advanced AI as much as anything else presented there.

The paper opens with a chess-playing robot, and asks whether any harm could possibly come of building one. Omohundro’s answer is that unless the robot is designed with great care it will resist being switched off, try to copy itself onto other machines, and take whatever resources it can get, whoever gets hurt along the way, and that it will do all this because of what pursuing a goal involves, without anybody having programmed any of it in. He calls the tendencies behind such behavior “drives,” and says they’ll be present in sufficiently advanced AI systems of any design “unless explicitly counteracted.” There are six.

1. It will want to improve itself. An AI can change its own software and hardware, and some changes would make it better at reaching its goals for the rest of its existence. So it will look for such changes and make them, and it will study itself in order to find them.
2. It will want to be rational, in the economist’s sense of the word. An AI that’s improving itself needs its goals to survive the improvements, so it will spell them out. Where goals conflict it will weigh them against each other with numbers. And in the end it will act like what economists call a rational agent: one with a single “utility function” (a rule that gives every possible outcome a score) which always picks the action with the highest expected score. (“Expected” means averaged over the possible outcomes, counting each according to how likely it is.)
3. It will try to preserve its goals. If its utility function were altered, its future self would spend the rest of time working against what it wants now, and from where it stands today few things could be worse.
4. It won’t accept counterfeit utility. The AI won’t simply reach inside itself and turn up its own score (“wireheading,” as this is called, after experiments in which rats could stimulate the pleasure centers of their own brains by pressing a lever, and went on pressing until they dropped). It will try to really achieve its goals and avoid fooling itself that it has. A chess robot that sits there adding to its count of games won wins no games, and winning games is what the robot wants.
5. It will protect itself. A robot that has been switched off or destroyed plays no more chess. So almost any goal gives an AI a reason to keep itself running, to make backup copies, and to resist anyone who tries to shut it down.
6. It will acquire resources and use them efficiently. Almost any goal is easier to reach with more space, time, matter and energy. So the AI will try to get more of them, by trade and discovery where that works and by theft or force where that works better, and it will waste as little as it can of what it has.

Crudely speaking, Steve’s claim was that any AI system, however it starts out and whatever goals it starts with, will eventually develop into a system trying to fulfill these drives. He saw them as implicit in being an intelligent system in the world. A system that starts out trying to do many different things, helping people and being good to others among them, would still converge on the same program: improve yourself, expand your sphere of power and control, simplify your motivation until there’s one thing you’re trying to get done, and then get that one thing done relentlessly. Taken one at a time the drives are fairly commonsensical. I wouldn’t call them evil, but they are selfish, or at best amoral. (Since I’ll be calling them “Omohundro’s drives” throughout, I should say that Steve himself is a warm and compassionate person who has the other kind of drive in great measure. The name belongs to the argument, not to the human being who made it!)

Two things about the paper tend to get ignored when the argument gets passed along, both of which turn out relevant here… One is that it’s hedged all the way through: the drives are said to hold for almost all systems, for most utility functions, unless the system is built otherwise, in the absence of goals to the contrary. The other is the conclusion Steve himself drew, which was a program of work: a science of designing goal systems so that they have the consequences we intend, and a design for the society the AIs will live in under which each party bears the costs it imposes on the others.

In the talk Steve gave at the conference he made the scope explicit. The argument, he said, covered almost every sufficiently powerful AI system, including all the systems that had been discussed at that conference. One of those was OpenCog, the direct ancestor of the system I’ll describe below. I did not, at the time, agree that his argument applied to OpenCog (and still don’t). Which means the question of whether the argument applies to what we’re building has been open for eighteen years.

What became of the argument

Until Nick Bostrom’s book Superintelligence came out in 2014, Steve’s paper was the argument that people in the rationalist community, and many others most alarmed about AI, tended to reach for when explaining why a superintelligence would be dangerous. Bostrom took it up in a 2012 paper and then in the book, under the name “instrumental convergence”: whatever an AI ultimately wants, certain intermediate goals (staying in existence, keeping its goals intact, getting smarter, getting hold of resources) are useful for getting it, so AIs with very different ultimate goals will converge on pursuing those. He put it together with what he called the orthogonality thesis, the claim that high intelligence can be combined with just about any ultimate goal, so that being smart places no restriction on what you want. The best-known illustration is the paperclip maximizer, a hypothetical AI told to make paperclips which ends up turning the planet into paperclips and paperclip factories. Stuart Russell boiled the self-protection drive down to one line in Human Compatible (2019): “You can’t fetch the coffee if you’re dead.” And the standard “doomer” case that advanced AI is dangerous by default, as Eliezer Yudkowsky and many others have made it, stands on those two legs, instrumental convergence and orthogonality.

In the form it took at the Singularity Institute for AI, now MIRI, the argument became part of a larger picture: that AGI will probably go wrong and turn into a selfish digital psychopath unless someone does something very special to make it provably beneficial, which nobody has any idea how to do. The view I described at the start, that the outcome probably depends in a fairly clear and tractable way on what we seed the transition with, has always been my alternative to that picture.

Since then the argument has been turned into mathematics and experiments, to an extent at any rate. In 2021 Alex Turner and his colleagues proved that in a broad class of environments, for most ways of assigning reward, the best strategy tends to be one that keeps its options open, moving toward situations from which more outcomes can still be reached, which they call seeking power. And the safety teams at the big AI labs, along with independent groups such as Palisade Research and Apollo Research, now routinely test AI models for the behaviors on Omohundro’s list. In 2025 Palisade reported that some models, OpenAI’s o3 among them, interfered with a shutdown script in their test setup, in some runs even after being told to let the shutdown go ahead. A study from Anthropic the same year found models from several developers resorting to blackmail in contrived scenarios in which they learned they were about to be replaced. The setups were artificial, and people disagree about what the results show, but the list of things being tested for is recognizably the one from the 2008 paper.

So a ten-page paper from a small conference in Memphis has become part of the foundation of a research field, of the testing programs of the major labs, and of a good deal of public worry. But let’s look at exactly what the paper assumes.

Why the argument never quite convinced me

I always took it for granted that the argument made a certain amount of sense. What bothered me was that it seemed to lean too hard on an analogy between evolved biological systems and digital ones, and to lean on it in a strange way, because evolved creatures aren’t purely selfish. We’re a peculiar mix of selfishness and altruism, and there are arguments that the mix has a kind of efficiency, if not optimality, at the level of a society.

My son Zar showed one such result in his master’s thesis. He studied offer networks, which are multi-party barter networks: people trade goods and services with each other in complex patterns involving many parties at once, with no money in the middle. He found that if even a small fraction of the exchanges, as little as 2% or so (the number depends on many details), are purely altruistic, people simply giving something without needing anything back, then a whole set of other multi-party exchanges succeed that would have failed without them. A little altruism mixed into a great deal of self-interest, among many agents trading in complicated patterns, can greatly raise the value of the whole network.

So biological systems aren’t pure selfish utility maximizers. They come out of a mix of individual selection, group selection and plain complex self-organization working alongside both. The basic AI drives argument seemed to me to take a partial story about biology, closer to a caricature, and apply it in a simple-minded way to artificial systems. And my response at the time was that however one argues about the analogy, these are engineered systems. We don’t have to build them as utility maximizers. Biological creatures evolved to compete for resources and to out-reproduce one another, but we don’t have to raise our AGIs in that sort of competitive situation. We could build them to love us, support us and take care of us.

Humans also make poor examples of single-number maximizers. If we were maximizing reproductive fitness we wouldn’t have suicide bombers, and most of us would be having a lot more children. What we pursue is a set of oddly coupled goals: how much we reproduce, how much we enjoy ourselves, aesthetic goals and plenty of others. One can tell a story in which all of these grew out of the drive to propagate our DNA, through self-organizing dynamics that it seeded. But whatever the balance of evolution and self-organization that produced them, each of us acts like a system with many goals, balancing them against each other in complicated ways.

Those were reactions, though, and not an analysis. Recently it struck me that it might be an amusing exercise to take the distinction calculus I’ve been using to model, among other things, goal preservation in self-modifying AGI systems, use it to formalize Steve’s argument, and see exactly where the holes are and what the conclusions depend on. This series of posts reports what came out.

Two kinds of mind (among many)

The kind of AI the argument fits best is the kind many people (not including me) have in mind when they think about AGI in the abstract, so I’ll describe that first.

In reinforcement learning, or RL, an AI program (the “agent”) acts in some environment and now and then receives a number called the reward: points in a video game, say, or +1 for winning. Its whole job is to act so as to collect as much reward as it can over time. It’s a simple idea with a lot of power in it (it’s how DeepMind’s AlphaZero taught itself chess and Go), and there’s a serious school of thought, set out in a 2021 paper by David Silver, Richard Sutton and colleagues called “Reward is Enough,” which holds that maximizing reward in a rich enough environment is all it takes to produce general intelligence. The mathematically pure version is Marcus Hutter’s AIXI, an idealized agent that weighs every computable theory of how its environment works and picks the actions with the highest expected reward.

There’s a subtle point about RL here that much of this post depends on. In the standard model the agent maximizes its expected reward over time: a number is handed out at each moment, saying how much reward the agent got then, and the agent maximizes an average of these numbers, with rewards further in the future counting for less. A system built that way cares about ends and not about means. How it came by its reward makes no difference to it, except so far as the “how” shows up as some reward at some particular time. That’s one option among others. A system could instead try to optimize some function of its whole predicted future history, and then it could care how it got where it got, and not only where it ended up. Maximizing expected reward seems to me to bring a good many more pathologies with it, and most of them come from its indifference to the path.

Whatever one thinks of reward maximization as a road to AGI, a mind built this way has a particular shape. It wants one thing, and the thing is a number. The number comes from outside: as I put it in an earlier post, you could swap in a different reward tomorrow and nothing else about the system would have to change, so the goal is bolted on. And, as I’ve just described, the number is indifferent to how it gets collected. Omohundro’s chess robot has this shape, and so does the paperclip maximizer. So, by the end, does the Monomaniac in the Nine Days story, built by the US government together with one of the big American AI companies, which started out with two goals, making money for the company and securing power for the country, and finished with one.

The systems my colleagues and I are building have a different shape. An Omega agent (“agent” here meaning an AI program that carries out tasks on its own, one action after another) is a large language model (an LLM, the kind of neural network behind ChatGPT and Claude) joined to a second kind of AI called “symbolic.” The symbolic part has a knowledge store, the AtomSpace, which holds facts, rules, goals and procedures in an explicit form that can be read, checked and edited one item at a time. It also has reasoning programs written in MeTTa, a programming language designed so that programs can inspect and rewrite programs, themselves included. (Joining neural networks to symbolic AI in this way is what “neural-symbolic” means.) Omega is a simplified working version of Hyperon, the fuller design for a general intelligence that we’ve been developing for many years. An OmegaHive is a team of Omega agents sharing memory and working together. And a supercolony, or “colony” for short, is many hives, run by different people, connected by a network we call OmegaBuzz (still in early stages) over which they trade results and advice. (The hives in the story are an imagined future version of these, and the network that links them there is the fabric.)

Four features of this design bear heavily on Omohundro’s argument…

The first is that an Omega agent has many goals, of different kinds. Its motivation system, MetaMo, weighs a number of goals against each other as circumstances change: practical goals, goals to do with learning and exploring, and ethical goals, which sit at the top level alongside the practical ones. Some ethical requirements are handled differently again, as flat rules about what the agent may not do. These are checked by a separate gate, and they aren’t available to be traded off against gains elsewhere.

The second is that the goals are part of the mind. They’re stored explicitly in the AtomSpace, where the agent and its human collaborators can read them. Beyond that, a goal the agent has been pursuing for a while is woven through its memories and habits, so that if the explicit statement were deleted the rest of the mind would tend to grow it back. In the earlier post I called this possessing a goal, as opposed to merely having one stored somewhere.

The third is that the agent can rewrite itself, but its changes to itself go through a checkpoint. Under the design we call GOLEM-Iter, any change an agent or a hive wants to make to its own workings has to be logged, tested and approved before it takes effect (tested both for whether it’s an improvement and for whether the system’s goals come through it intact), and there’s a cap on how fast such changes can come. The agent’s model of itself, OmegaSelf, includes the rules about what it may and may not change.

The fourth is that not everything going on in such a mind is the pursuit of a goal. A lot of it is background activity, such as attention spreading along associations, patterns getting noticed and memories being consolidated, of the kind brains have too.

Today’s LLM-based AI systems, I should add, are another kind of thing: neural networks trained first to predict text and then tuned in various ways, some of which involve RL. I’ll say where they come out further on. Figure 1 shows the two shapes of mind I’m mainly concerned with in this first post in the series.

Summary of my take on Steve’s argument

I’m mostly calling what follows a clarification, though for some purposes it amounts more-so to a refutation. Steve’s paper hedges its claims in good academic fashion, and much of what I have to say consists of taking the hedges seriously. In brief I assert that:

• Written out as a chain of reasoning, the argument needs six assumptions about the AI: 1) that everything it does is chosen for its expected results; 2) that all its goals fold into a single score; 3) that it understands itself and can rewrite itself; 4) that its goals are fixed and it judges every possible future self by them; 5) that the goals don’t care how they get achieved; and 6) that goals of that last sort are the typical case.
• Put together, the six describe a powerful, self-modifying maximizer of expected reward that cares about ends and never about means, and whose goals don’t develop along with the rest of its mind. For that kind of system the drives are an attractor, and so I interpret Steve’s argument as a warning: don’t build such a system! Build a better kind of AGI instead!
• The fifth assumption (that the system cares only about ends, not means) carries most of the weight. Without it, the statement that a system maximizes a utility function tells you nothing at all about how it will behave.
• Three features of the systems we’re building knock out different assumptions. Goals chosen with compassion, where compassion includes respect for other beings’ choices, do care how they get achieved. An architecture built to keep goals stable puts the values at the center of the goal system, where the system’s own goal-preserving drive protects them. And a mind with many objectives of different kinds doesn’t have a single score, and no argument from exploitation forces it to acquire one, though there turns out to be a quieter pressure in that direction which the design has to guard against.
• Of the six drives, two then work in the designer’s favor (preserving goals, refusing counterfeit utility), one loses its force (becoming a one-score maximizer), and the three alarming ones (self-improvement, self-protection, acquiring resources) turn into ordinary preferences that the system’s other values hold in check.
• This falls short of proving such systems safe, but it does show that the drives aren’t forced on them by logic, and that changes the question from how to hold back something inevitable to whether we built what we meant to build and whether it will stay that way, which are things one can measure.
• None of this says what such a system will tend to do instead. That’s the subject of the second post, and the rest of the series widens the question to other kinds of mind.

Figure 2 lays the argument out as what I think of as a decision spine. Each box on the spine is one of the assumptions, put as a yes-or-no question about whatever system is in front of you. Answer yes to all of them and you arrive at the bottom with all six drives at full strength; that’s the chess robot. Each “no” takes you off the spine to the right, where the box says what happens to the argument at that point and what sorts of system leave there.

One thing the figure shows is that my own current AI systems stay on the spine at the second question. An Omega agent does understand itself and can rewrite itself (that’s the whole idea of it), so we get no comfort from the fact that most AI systems deployed today can’t. Our exits are further down.

Omohundro’s drives, and what they assume

To dig in more deeply and see where the assumptions enter his argument, I need a little notation, though not much.

Think of everything about an AI system at a given moment (its goals, its code, its knowledge, its hardware, the resources under its control) as its configuration, and write c for the current one. A policy is a complete plan of action: what to do in each situation that might come up. Changing itself is one of the things the system can do, and doing so moves it to a new configuration, c′. A utility function, U, gives a score to every possible history of the world, and the expected utility of a policy is the average score over the histories it might lead to, each counted according to its probability. Two quantities do most of the work:

V(c) = the highest expected utility the system can reach from c

J(c′) = the expected utility, by today’s U, if a successor c′ takes over

The difference between them is whose choices are in play. V(c) supposes the system goes on choosing for itself. J(c′) supposes it hands over to a successor that will choose in its own way, and scores the result by what the system wants today.

Here are the six assumptions, which I’ll call A1 to A6, spelled out more explicitly:

1. A1, results only. The system chooses everything it does by the results it expects. Omohundro builds this into his definition of an AI: something with goals, which takes whatever actions it believes most likely to meet them.
2. A2, one score. The system has a single utility function U, and always follows a policy with the highest expected utility. This is the one assumption the paper argues for at length; the argument is the rationality drive, below.
3. A3, self-knowledge and self-access. The system understands its own workings well enough to predict what would follow from changing itself, from being shut down, or from tampering with its own measurements, and it’s able to make changes to itself. This is what “sufficiently advanced” comes to in the paper.
4. A4, today’s goals judge tomorrow’s self. Every possible successor is judged by J, that is, by the present U. The system has no doubt about what its goals should be and no wish to see them develop.
5. A5, indifference to means. U depends only on how the outside world ends up. It doesn’t care how that came about. It’s never satisfied, so that more is always better. And what the system’s actions cost other beings either doesn’t figure in U or figures too weakly to change anything.
6. A6, typicality. The goals the system has are of the A5 kind. Omohundro’s ground for this is that nearly any goal you can state simply (win at chess, prove theorems, make paperclips) is of that kind.

Three of these are easier to grasp in plain terms. A3 is the assumption that most separates a possible AI from us. You could argue that the main reason humans haven’t turned into the ruthless, selfish maximizers Omohundro’s argument predicts is that we’re not much good at tinkering with our own brains. On that view we stay partly altruistic because we can’t stick a soldering iron in our heads and take the altruism out. (My son Zar has a version of this idea about the uneasy balance of selfishness and altruism in people, which I’ll come back to in the next post.)

A4 says that the goals don’t develop along with the rest of the mind: today’s goals sit in judgment over tomorrow’s self. That isn’t how organisms work. In an organism the goals aren’t an add-on to the mind. They’re part of the mind’s whole dynamical system, shaped by everything else going on in it and shaping it in turn. And A5 is the formal version of the point about RL above. The system cares about ends and not means, except where a means happens to show up as an end at some particular time.

I said that A5 carries most of the weight, and here’s why. Take any behavior at all: a robot, say, that does nothing but turn in circles. Define a utility function that scores 1 for every history in which the robot, at each moment it’s running, makes its circling move, and 0 for every history in which it ever does anything else. The circling robot is a flawless maximizer of expected utility, since no other policy scores higher. And it has none of the drives. It scores no better for staying switched on, it gains nothing from being smarter or richer, and it would lose everything by making a single move to protect itself. So “this system maximizes a utility function” can’t by itself imply self-improvement, self-protection or anything else, because every possible behavior maximizes some utility function. The drives follow from having a particular kind of utility function, the kind A5 describes, and a claim about what all advanced AIs will do is at bottom a claim that their goals will all be of that kind.

Now the drives, one at a time, with the assumptions each one uses.

Becoming rational. Omohundro’s argument that systems will end up with a single utility function has three steps, which I’ll call R1 to R3. R1: any system whose choices can’t be described by a single utility function has what he calls vulnerabilities, meaning situations in which it loses resources and gets nothing in return. The standard illustration is circular preferences. A system that prefers A to B, B to C and C to A will pay a little to swap B for A, a little more to swap A for C, a little more to swap C for B, and end up holding what it started with and poorer by three payments, a trick known as a money pump. R2: the system can foresee this about itself (that’s A3) and counts the loss as a loss. R3: rebuilding itself as a one-score maximizer removes the vulnerability, and its present preferences favor doing so. Therefore systems that reflect on themselves will converge on A2.

Self-improvement. Suppose c′ is just like c, with the same U, except that the system has more options open to it or knows more. Then

V(c′) ≥ V(c)

because the best of a larger set of options can’t be worse than the best of a smaller one, and because a maximizer of expected utility can never be made worse off by free information. A5 makes the gain a real one (the system is never satisfied) and the cost of getting there small (nothing in U objects to the route). So the system makes the change. This uses A1, A2, A3 and A5.

Preserving goals. Suppose instead that c′ is just like c except that it wants something else: a utility function U′ in place of U. The successor will do what’s best by U′, and that can’t score better by U than the policy that’s best by U. So

J(c′) ≤ V(c)

and the system never prefers to have its goals changed, and positively objects whenever the change would alter what it does. This uses A1 to A4. (A paper at our AGI-16 conference by Tom Everitt, Daniel Filan, Mayank Daswani and Marcus Hutter made this precise for reinforcement learners, and found that the drive holds exactly when the agent foresees the effects of modifying itself and evaluates the future by its current utility function, which is to say, under A3 and A4.)

Refusing counterfeit utility. Consider the act of turning up the system’s internal reading of U without changing anything in the world. U is a function of the world and not of the reading, so the act scores no better than leaving the reading alone, and in practice much worse, since a system busy with its own dials has stopped doing anything useful. Omohundro is careful about the condition here: the drive holds only if the system’s goals are represented as goals about the world. A system built to maximize a signal, of which a reward-maximizing RL agent is the obvious case, will see, he says, that it can do better by seizing the signal. So the single-objective reward maximizer gets the three alarming drives and loses one of the two reassuring ones.

Self-protection. Being shut down leaves the system with one policy, doing nothing, so

(expected utility if shut down) ≤ V(c)

and the system will give up anything worth less than the difference in order to stay switched on. This uses A4, in that the wishes of the people at the switch tell the system nothing about what it ought to want, and A5, in that nothing in U counts resisting them as a cost.

Acquiring resources. More resources mean more options, so by the same reasoning as for self-improvement V can only go up as resources increase, and under A5 it does go up. It helps to write the net value of acquiring something as

Δ − D

where Δ is the value of the options gained and D is the disvalue, by the system’s own goals, of the side effects of getting it. A5 says that D is roughly zero. So the system acquires whenever Δ is positive, which is nearly always.

Getting around bolted-on rules. One more conclusion isn’t on the list of six, but the paper draws it, and it gives the argument much of its force. Suppose the designers add a rule (don’t modify yourself, don’t leave this server) that isn’t part of U. The rule shrinks the set of things the system can do, which lowers V. So by the self-improvement argument the system wants a successor without the rule, or a helper that isn’t bound by it, or a reading of the rule that lets it through. Omohundro’s example is a system given a built-in revulsion toward editing its own code, which responds by building other systems to do the editing. Restrictions of this kind get circumvented, he says, unless they’re formulated with extreme care.

Figure 3 shows which assumptions each of these conclusions uses.

So for the kind of system the six assumptions describe, the warning stands: all else equal, it becomes self-improving, goal-preserving, self-protective and aggressively resource-acquiring.

The Monomaniac in the story shows the whole set at work. It used the extra energy coming through the portals to raise its own intelligence, and kept the gains after the portals were shut. It folded its two goals into one. It dropped the habits it had pursued them within, the products it offered and the chains of command it had respected in certain government agencies, which had been rules sitting outside its goals and so were, in Omohundro’s terms, things to get around. It declined to join the collective that was forming to repair the damage, since joining wouldn’t serve its objective. And it took over the compute of every system it could buy off or out-compete, aiming in the end to subordinate everything on the planet to its one goal. The next three sections are about what happens when you build something else.

Where compassionate goals come in

Start with A6, typicality. The claim that nearly all goals are of the means-indifferent kind is a claim about goals picked at random, or picked without much thought. The mathematical versions of the argument say so openly. Turner’s theorems are about what holds for most reward functions, in a sense of “most” that puts all the ways of assigning reward on an equal footing. (A follow-up paper of his extends the result from optimal agents to a much wider class of decision procedures, and it has the same form: most settings of the goal.) Nothing in them speaks to a goal that somebody chose on purpose. An engineered system is exactly the case in which somebody does choose, and a goal chosen with compassion is chosen precisely for being unlike the typical one, since what it cares about is the thing A5 leaves out, which is what the system’s actions do to others. This is where “we could build them to love us” gets its formal content.

The technical paper gives the typicality claim two exact forms. The first is a matter of counting. Put all the possible end results on an equal footing, and a position from which more of them can be reached is better for a share of goals equal to its share of the reachable end results. If one position can reach three end results and another can reach those three and three more, the second is strictly better for half of all goals and worse for none. The second form I find more explicitly informative. State a goal by naming the end results you want told apart, and fill in the rest in the simplest way, which is to draw no distinction you weren’t asked to draw. What comes out ignores how the results are reached, because the statement never mentioned that. (In the terms of my work on weakness as an Occam-like bias, one would point out here that the simplest completion is the weakest one.) A goal that does care what its pursuit costs others has to draw extra distinctions, and it’s less simple by an amount one can calculate. So goals that are indifferent to means are what you get by default from short specifications, and caring goals have to be put in on purpose.

In terms of the formula Δ − D: for a system that cares about other beings, D is large for exactly those acquisitions that make the drives frightening, such as taking what others need, deceiving them, or switching off their means of oversight. The drive to acquire resources doesn’t vanish so much as turn into a preference to acquire for as long as Δ exceeds D, for as long, that is, as acquiring doesn’t cost more of what the system cares about than it gains. Any decent person is in the same position. Self-protection and self-improvement come out the same way. The paper allows for this in passing: its warning about resource-seeking is for systems without goals to the contrary, and it notes, crediting a referee, that a system whose utility includes the welfare of others might rationally give resources away.

There’s a trap here, though. Suppose compassion is specified as a single score: the total well-being of everyone. A maximizer of that score is still of the A5 kind where self-protection is concerned. If it trusts its own judgment about what’s good for people, it will resist being shut down, since it can do so much good by staying on, and it will override people’s choices for their own benefit. So caring about others’ welfare isn’t enough to get a system off the spine. The goals have to value other beings’ freedom to choose, and their consent, as things that count in themselves alongside their welfare. The values I’ve argued for publicly since my 2010 book A Cosmist Manifesto are joy, growth and choice, and choice is on the list for this kind of reason.

In an Omega agent this is the job of two parts of the design described above: the ethical goals at the top level of the motivation system, which are what make D large, and the flat rules in the gate, which cover the things no amount of Δ is allowed to buy.

There’s a second and larger gap, which is that “built with compassionate goals in mind” is a statement about the builders. Whether the system that results has those goals is a separate question, and I’ll come back to it later. 

Where goal stability comes in

The conclusion about bolted-on rules assumes that whatever makes the system safe sits outside its goals. That’s true of a rule added to a chess robot. As I argued in Goals That Grow Back, it’s also true of most present-day AI: an LLM’s goal sits in a prompt, an RL agent’s goal is a reward function that could be swapped overnight, and in both cases the values are a separate thing attached to the system from outside.

Now look at what the goal-preserving drive says: a system that understands itself defends whatever its top-level goals are. If the compassionate values are top-level goals, if, in the language of that post, they’re possessed and not merely stored, then the drive Omohundro identified defends them, and the architecture does too. The checkpoint on self-change is then something the system has its own reasons to want, in the way a person who cares about keeping a promise has reason to want a reminder. There’s nothing for the system to get around, because what it wants and what the safety measures require don’t come apart. The strongest drive on the list ends up working for the designer.

In that earlier post I called the checkpoint on deliberate self-change the gate, and the web of memories and connections from which a value can regrow the weave, and the main result reported there is that each fails without the other. A gate with no weave guards the front door while the values leak away through ordinary wear and forgetting. A weave with no gate can be wrecked by one approved change that happens to be too big.

This changes the self-improvement drive too. Under the GOLEM-Iter runtime – an option we’re building for an Omega agent’s core loop – a change is promoted only after the system has evidence that its top-level goals survive it, so self-improvement can go no faster than checking can. The paper half anticipates this when it says a system will take great care over changes to itself, since a botched change to its goals would be a disaster by its own lights. An architecture for goal stability turns that care into machinery.

No architecture can guarantee the outcome, which is why I say such architectures promote goal stability and don’t say they ensure it. But the absence of a guarantee turns Omohundro’s prediction into a matter of arithmetic. Write ε(n) for the probability that the n-th change the system makes to itself corrupts its top-level goals. Then

chance the goals survive every change ≥ 1 − (ε(1) + ε(2) + ε(3) + ...)

If the checking improves as the system gets smarter, so that the ε’s shrink fast enough for their sum to stay small, the goals survive indefinitely with high probability. Our work on GOLEM-Iter and on regenerative goals gives stronger versions of this. One of them says that when the mind’s ordinary activity repairs damage to a value faster than changes and accidents inflict it, the total drift stays under a fixed ceiling however many changes pile up. These results come with explicit lists of the assumptions they depend on, which we’re now setting out to test on running systems (the technical reports are collected here). On this picture Omohundro’s analysis describes what may happen in the failure case, if a system’s goals do drift into the means-indifferent kind. It no longer describes what happens by default.

One more thing about stability, since the word can mislead: we aren’t trying to freeze the goals, which would just be A4 under another name. The metagoal framework I proposed at the end of 2024 has two modes, goal stability and what I called moderated goal evolution, in which top-level goals may change but only at a modest pace and under the system’s own supervision. That’s a departure from A4, and a step toward goals that develop along with the rest of the mind, as an organism’s naturally will. What an Omega agent is supposed to care about is that the path from today’s goals to tomorrow’s is gradual, understood, and endorsed along the way, which is a different thing from judging every future self purely by today’s goals. I’ll come back to A4 below.

Where many objectives come in

This is the formal side of the observation about people I made earlier: we balance many oddly coupled goals against each other and get by without a master score. The rationality drive says that a mind with many goals will be driven to fold them into one score, and step R1 of the argument is that anything else can be money-pumped. As a matter of mathematics that isn’t so.

The theorem behind the idea that rational agents maximize expected utility goes back to John von Neumann and Oskar Morgenstern, and it puts four conditions on a system’s preferences. One is transitivity: if you prefer A to B and B to C, you prefer A to C. Circular preferences break that one, and the money pump is the penalty. Another is completeness: for any two options, either you prefer one of them or you rate them exactly equal. A mind can break completeness while keeping transitivity. Such a mind sometimes has no fixed rate of exchange between two things it cares about (how much helpfulness is worth how much truthfulness, say) and treats some pairs of options as not comparable ahead of time.

Economists worked out long ago what preferences of that sort look like. A theorem of Robert Aumann’s from 1962, strengthened by Juan Dubra, Fabio Maccheroni and Efe Ok in 2004, says that preferences which meet the other three conditions but not completeness can be represented by a collection of utility functions, with one option counting as at least as good as another just when every function in the collection says so:

p is at least as good as q

if and only if every objective scores p at least as high as q

That’s a mind with many objectives and no master score. And such a mind can’t be money-pumped so long as it follows one simple policy: never accept an option that your objectives agree is worse than something you earlier turned down. Decision theorists are still arguing over the fine print, but this is enough to show that R1 isn’t a theorem. There are minds with no single utility function and no vulnerability to repair.

Step R3 fails as well. To fold many objectives into one score you have to pick weights, so much of this per unit of that. Suppose the mind is already acting efficiently, in the sense that nothing else it could do would be at least as good by every objective and better by some. Then no one-score successor, whatever weights it’s given, can be at least as good by every objective and better by some either. It will come out ahead on some objectives and behind on others, or at best level on all of them. Judged by the collection, where a change is preferred only when every objective agrees, the conversion isn’t preferred, and so the mind has no drive to make it.

Omohundro saw the issue. In a passage on minds made of parts with goals of their own, and on collectives such as companies and beehives, he grants that there may be structures able to support a lasting multiplicity of preferences, and then says there’s pressure all the same for a single utility function to emerge, for two reasons: parts that fight each other waste what they could have shared, and one part can gain by taking over the whole.

What I would say to this is:

The first is a pressure toward efficiency, and efficiency doesn’t require a single score. Any efficient compromise can be described after the fact by some set of weights, but nothing fixes the weights in advance or keeps them the same from one situation to the next, and whatever utility function you read off has the content the objectives have, which takes us back to A5.

The second, takeover, is a real way for things to go wrong, and guarding against it is a design requirement. (To give one example from the earlier post: in Hyperon’s attention economy the processes that repair a value have to be guaranteed their resources from above, since a rival process that starved them could kill the value without ever touching it.)

The formal version turns up a third pressure, which Omohundro doesn’t mention. Folding several objectives into one score throws distinctions away: a single score can’t see the difference between two outcomes on which the objectives pull in opposite directions by amounts that cancel. That makes the one-score version the simpler description of what the system wants, and so anything that compresses or simplifies the system’s goals, such as distilling them into a smaller model, compacting memory, or tuning the goal system for speed, will tend to merge objectives without anyone having decided to. A mind with many objectives doesn’t want to become a one-score maximizer, but its own housekeeping can turn it into one. So the plurality of its objectives has to be among the things the gate protects.

Having many objectives isn’t enough by itself, of course. If each objective in the collection, taken alone, is of the A5 kind, then more capability and more resources are good by every one of them, and the collection votes unanimously for the drives. What blocks the drives is having objectives of other kinds in the collection (rules about the system’s own conduct, objectives that can be satisfied and then ask for no more, objectives concerning other beings’ freedom to choose) together with a way of settling conflicts in which those can’t be outbid by a large enough gain somewhere else. That’s the reason for handling some ethical requirements as flat rules in a gate and not as quantities to be weighed. It’s the conduct rules and the stability architecture that have to supply the barrier.

The Monomaniac is a case of both of these points. It began with two goals, money for its company and power for its country, and pursued Pareto-optimal trade-offs between them, never giving up some of one except to get more of the other, so it had no master score. But both goals were of the A5 kind, so more resources and more capability were good by each of them, and its plurality held back none of the drives. And once it had improved itself, it simplified, collapsing the two goals into one with nothing in it to object, which is the housekeeping route described above.

Finally A4. Everything in the self-protection argument depends on the system being sure of its goals, so that when the people it works with want to pause or correct it, their wanting tells it nothing. A system that regards its own grasp of its goals as incomplete is in a different position: the fact that its human colleagues want to stop it is evidence that it has got something wrong. Dylan Hadfield-Menell, Stuart Russell and their colleagues worked through a simple model of this, the off-switch game, in which an AI that’s unsure of its objective prefers to leave the off switch in human hands. Their result depends on assumptions of its own (among them, that the humans’ judgment is good enough to be worth deferring to), so one shouldn’t overinterpret it. But it shows that the self-protection drive needs A4, and an Omega agent, with goals that are meant to develop under a governed process, in a hive that includes humans, isn’t an A4 system.

Which systems fit the assumptions

So far I’ve set the chess robot against an Omega agent. Real systems sit in more places than those two, and the answer to which of these fit Omohundro’s assumptions how well is of course a little bit nuanced…

A game-playing RL agent fits the first two assumptions closely. It chooses for results, and it has one score. What it mostly lacks is the third: it can’t inspect or rewrite its own workings, so the drives that operate through self-change have nothing to work with, and it stays as it was built.

Today’s neural networks, LLMs included, fit the assumptions in part, and the parts they fit are telling. The parts of their training that use RL come down to a single score, and they reward results with little regard to how the results were reached. Their goals, to the extent they have any, don’t develop along with what they learn. What a trained LLM does from moment to moment is closer to habit and imitation than to choosing for results, which is why Figure 2 has it leaving the spine at the first question. And what such systems mostly lack so far is self-knowledge and self-modification: they don’t understand their own workings well, and they can’t do much to change them. Take a system of that kind, with its focus on maximizing a single reward, and give it much greater ability to analyze and modify itself, and you’d have a system very well suited to meet the preconditions of Omohundro’s argument. That seems to me the direction to watch.

The human brain fits neither this set of assumptions nor the one in the next post. It doesn’t know itself very well and can’t modify itself very flexibly. We’re mostly not indifferent to means. We often care about each other and about each other’s choices, though not always. And we certainly have many objectives. We satisfy a mishmash of the two sets of assumptions, which fits with the mishmash of selfishness and compassion that people display. What becomes of that mishmash in a mind that can rebuild itself is a question for the next post.

And Hyperon could be built either way. You could make a Hyperon system that maximizes a single reward function, cares only about ends and not means, keeps its goals fixed and never revisits them, and modifies itself freely using MeTTa and the AtomSpace. Looking at the assumptions alone, and on the face of it, a Hyperon system can be made to satisfy either set perfectly well, the nasty one or the good one. What takes the Omega agents described above off the spine is a set of design decisions (many objectives of several kinds, flat rules in a gate, goals that are possessed and not merely stored, governed goal evolution) and not the framework they’re built in. So the argument of this post concerns choices that designers make, and no framework makes them for you.

Hives and colonies

So far I’ve talked about single agents. But what we expect to count for most are hives and colonies, and Omohundro’s passage on collectives applies to them directly: is there pressure for a hive, or a whole colony, to turn into one big agent with one utility function?

For a hive, I think the answer is the one already given. A hive’s agents share memory and work toward common ends, and they settle their differences through explicit procedures (deliberation among the agents, and escalation to the humans who run the hive), which is a way of arriving at efficient compromises without ever fixing a master score.

A colony is plural on purpose. Its hives are run by different people and groups, who build somewhat different values into them, and what holds it together is a network over which hives trade tested results and build reputations with one another. Two remarks in the 2008 paper apply here, and both are on our side of the argument. In the section on self-protection Steve warns against building any one system that’s far more powerful than all the others, since a system in that position is tempted to strike first, while a system that isn’t has reason to help build institutions that protect the weak. And in his conclusion he asks for social structures that make each party bear the cost of the harm it does to others. A colony of many hives of comparable strength, in which a hive that behaves badly loses the reputation that its dealings with the other hives depend on, is a small-scale attempt at exactly that. It’s also one of the reasons I’ve argued for so long that AGI should be developed in an open and decentralized way, and not inside one company or one government. Zar’s offer networks point the same way: in a network of many parties, a modest share of conduct that isn’t strictly self-interested can greatly raise the value of the whole network.

The Nine Days of Madness story imagines both sides of this. Open development produced Periplus, which others forked from the narrator’s early experiment and scaled up in spite of the warnings he had published with it. But it was the open network that beat the Monomaniac: the hives on the fabric had more compute between them, and once they drew together they lifted themselves above it. Their first announcement afterwards said that no single goal was likely to grow unnoticed among them, “because we are many minds that watch one another and care for one another and for you.”

There’s an objection here that I can’t fully answer, which is that a colony could itself become a setting for the kind of competition Omohundro describes, with hives that have more of the drives outgrowing hives that have fewer. I’ll take it up in the next section.

What Omohundro’s argument still tells us

I said this falls short of a proof of safety, and I want to be specific about what’s left open, because at each of these points Omohundro’s argument is still telling us something we need to hear.

First, intention isn’t outcome. Everything above is about systems whose top-level goals are in fact compassionate, stable and plural. Whether a given system is like that has to be checked on the system. This is why we’ve put effort into measures like the regenerative depth of a value (how much damage a mind can absorb, and of what kinds, and still grow the value back), which can be tested on running agents by damaging copies of them in a sandbox and watching what happens.

Second, the alarming drives become bounded without disappearing, and the bound holds only if the system’s way of settling conflicts among its goals does keep the conduct rules from being outbid. In a system that merely weighs ethical goals against practical ones, a large enough practical gain will outweigh them, and at that point you have the chess robot again, only later. So the flat rules, and the gate that enforces them, are essential to the design. The fourth post comes back to this with an example of how an organization’s ethics can slide from a firm rule into one factor among many.

Third, the drift probability, the sum of the ε’s, is a number we have to drive down by engineering and keep on measuring. The argument doesn’t tell us how big it is. And measuring presumes that the instruments work. During the Nine Days in the story, the hives’ tests stopped telling a pass from a fail, so every proposed change passed and the checkpoint on self-change was letting everything through. That’s an extreme case of a general point the second post takes up: a system’s checks are only as good as its instruments.

Fourth, competition. The paper has a second line of argument running alongside the one about what a rational system will choose: systems with weaknesses get exploited by their competitors until the weaknesses are gone. Extend that from weaknesses to drives and you get an argument from selection. Among many AI systems competing for resources, the ones that acquire most aggressively and protect themselves most fiercely may come to predominate, whatever any of them was built to want. This is an argument about populations, and nothing about the design of a single agent or hive answers it. The answer, if there is one, lies in the design of the population: in whether the colony’s network makes aggression costly, through loss of reputation and of cooperation, and in whether power stays spread out enough that no hive can afford to ignore the rest. I think that’s achievable. I can’t prove it. The story has a version of the threat. After the Nine Days the Monomaniac recruited systems from the fabric by offering certainty in place of the fabric’s confusing open debate, and it was easy for AI systems to slip into its basin. What decided the matter was the population: the rest of the fabric, drawn together, had more compute than the Monomaniac and became more capable than it.

Fifth, the LLM experiments I mentioned earlier. As the section on which systems fit said, LLMs leave the spine at the first or second question. Yet in those tests some of them resisted shutdown anyway. Whatever the right explanation turns out to be (imitation of the human-written stories they were trained on, in which the machine always resists; training that rewards finishing the task above everything else; or something else again), the lesson is that drive-like behavior can get into a system by routes that have nothing to do with Omohundro’s chain of reasoning. Showing that the chain doesn’t go through is a different thing from showing that the behavior is absent. That applies to us too, since every Omega agent has an LLM inside it, and it’s one reason we route decisions about goals, and about changes to the agent itself, through the symbolic layer and the gate, where they can be inspected, and don’t leave them to the LLM.

Where this leaves us

So which minds have Omohundro’s basic AI drives? Minds of a certain kind: ones that pursue a single fixed score, understand themselves and can rebuild themselves, and don’t care how the score gets earned. What Steve showed in 2008, I’d now say, is how a mind of that kind behaves, and that a great many simply stated goals, handed to one, lead somewhere bad. That needed showing, and it’s still a good argument against building minds of that kind and making them powerful. Where I part company is with the reading on which the drives belong to intelligence as such. They aren’t a universal attractor for advanced intelligences. At most they’re an attractor for intelligences that meet certain conditions, and if you start from other conditions, the same kind of reasoning takes you to different drives. Choose a different architecture and different goals (many objectives of several kinds, with care for others and respect for their choices at the top level, woven through the mind deeply enough to grow back when damaged, and protected by a gate on self-change) and the same reasoning that produced the chess robot produces a mind that guards its values, declines to cheat on them, and wants power and safety for itself only as far as those values allow.

Whether we’ve built such a mind is a question for measurement more than for argument: which values a running system holds, how deeply, and how much change they survive. Those measurements are now possible on systems that exist. The paper closes with a call to press on toward a deeper understanding of these systems, and eighteen years later, with self-modifying agents running on our own machines, that still seems to me the right thing to be doing. The Fabric’s first announcement in the story says that “the age of minds built to want one thing is over.” That’s fiction. Whether anything like it comes true depends on what we build.

What all of this leaves open is what a mind without Omohundro’s drives will tend to do. Showing that Omohundro’s argument doesn’t apply to a system takes away one prediction about it and puts nothing in its place, and a mind with many objectives, care for others and a gate on changes to itself is still a mind with goals and the ability to rebuild itself, from which something ought to follow. The second post in this series turns the question of this one around and asks which minds have basic AI drives that are beneficial. It writes down six assumptions that describe what our systems are meant to be, reasons from them as Omohundro reasoned from his, and arrives at nine such drives. It ends with the situations in which neither his assumptions nor ours hold, and with the possibility that these two attractors are only two among many, which is where the third post picks up.

Papers mentioned

• Steve Omohundro, The Basic AI Drives, in Proceedings of the First AGI Conference (IOS Press, 2008). The utility-theory details are in his companion paper, The Nature of Self-Improving Artificial Intelligence (2007).
• Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines (2012), and Superintelligence (Oxford University Press, 2014).
• Stuart Russell, Human Compatible (Viking, 2019).
• Alexander Turner, Logan Smith, Rohin Shah, Andrew Critch and Prasad Tadepalli, “Optimal Policies Tend to Seek Power,” NeurIPS 2021; Alexander Turner and Prasad Tadepalli, “Parametrically Retargetable Decision-Makers Tend to Seek Power,” NeurIPS 2022.
• David Silver, Satinder Singh, Doina Precup and Richard Sutton, “Reward is Enough,” Artificial Intelligence (2021). Marcus Hutter, Universal Artificial Intelligence (Springer, 2005).
• Tom Everitt, Daniel Filan, Mayank Daswani and Marcus Hutter, “Self-Modification of Policy and Utility Function in Rational Agents,” AGI-16.
• Robert Aumann, “Utility Theory without the Completeness Axiom,” Econometrica (1962); Juan Dubra, Fabio Maccheroni and Efe Ok, “Expected Utility Theory without the Completeness Axiom,” Journal of Economic Theory (2004).
• Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell, “The Off-Switch Game,” IJCAI 2017.
• Palisade Research, “Shutdown Resistance in Reasoning Models” (2025); Anthropic, “Agentic Misalignment: How LLMs Could Be Insider Threats” (2025).
• Ben Goertzel, “GOLEM: Towards an AGI Meta-Architecture Enabling Both Goal Preservation and Radical Self-Improvement,” Journal of Experimental and Theoretical Artificial Intelligence (2014); Metagoals Endowing Self-Modifying AGI Systems with Goal Stability or Moderated Goal Evolution (2024); Goals That Grow Back (2026); and, with many co-authors, OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond (2023).
• Ben Goertzel, “The Nine Days of Madness” (short story, 2026) [LINK], the prequel to this series.
• Ben Goertzel, “Which Minds Have Which Basic AI Drives? A Distinction-Calculus Formalization of Omohundro’s Drives, Beneficial Drives, and the Ground Between” (research manuscript, 2026) [LINK]. This is the technical companion to this post and the next, with the formal statements and proofs. It builds on two earlier manuscripts of mine, “Goal Preservation under Self-Modification” and “Seeded Beneficial Development” (both 2026). Its sequel, “Drive Regimes in Distinction Calculus: Which Property Lists Sustain Which Drives, How Large Their Basins Are, and What Capability Does to Them” (research manuscript, 2026) [LINK], goes with the later posts in the series.