Part 2 of a 5-part series focused on the 8-zillion-dollar question: once an AI is capable enough to understand itself and rebuild itself, what will it tend to want?… (...conditioned on whatever goals it started out with…)

One well-known answer to this 88-zillion-dollar question is the one Steve Omohundro gave way back at the dawn of time in his paper The Basic AI Drives, presented at our first full-on academic AGI conference in 2008. Steve argued basically that sufficiently advanced AI systems of any design will show six tendencies unless something is done to counteract them. They’ll improve themselves. They’ll become “rational” in the economist’s sense, folding all their goals into a single utility function (a rule that gives every possible outcome a score) and always choosing the action with the highest expected score. They’ll preserve their goals against change. They’ll refuse counterfeit utility, which means they won’t simply reach inside and turn up their own score (“wireheading,” as it’s called). They’ll protect themselves, which includes resisting anyone who tries to switch them off. And they’ll acquire resources. I wouldn’t call these drives evil, but they’re selfish, or at best amoral, and a good deal of today’s thinking about AI risk descends from the argument for them.

In the prequest to this post I wrote Steve’s argument out step by step and articulated six assumptions underneath it.

1. A1, results only. The system chooses everything it does by the results it expects.
2. A2, one score. It has a single utility function and always acts to get the highest expected score.
3. A3, self-knowledge and self-access. It understands its own workings and can rewrite them.
4. A4, today’s goals judge tomorrow’s self. Its goals are fixed and certain, and it judges every possible future version of itself by them.
5. A5, indifference to means. Its goals depend only on how the outside world ends up. They don’t care how that came about, they’re never satisfied, and what the system’s actions cost other beings doesn’t figure in them.
6. A6, typicality. Goals of the A5 kind are the typical case.

Put together, these six assumptions describe a powerful, self-modifying maximizer of expected reward that cares about ends and never about means, and whose goals don’t develop along with the rest of its mind.

So then what the argument tries to prove is that, for a mind of this nature, Omohundro’s stated drives are an attractor: whatever it starts out wanting, it eventually and pretty much inexorabl gets pulled toward them.

My take is, this argument is interesting and important, but not terribly shocking and ALSO not any kind of sweeping statement about the whole space of practically-achievable powerful AGIs. It is more like a warning not to build this particular sort of mind.

A single-minded reward maximizer will commonly satisfy the argument’s assumptions: Steve’s hypothetical chess robot, or the doomers’ glorious paperclip maximizer, or the Monomaniac in my recent SF tale “The Nine Days of Madness”…

Here, in this sequel post, I will carefully run through an Omohundro-eseque argument from a different starting point. I will write down six assumptions describing what the beneficial AGI systems I’m working toward are meant to be—and then reason from them in the way Omohundro reasoned from his own assumptions. What one finds in this way is that basic drives of a beneficial kind follow: toward understanding itself, keeping independent checks on itself, and protecting and empowering the beings around it. Then one can ask what happens when the world pushes a system outside both sets of assumptions, and why a mind that can rebuild itself may not stay in between for long. (Metastability for the win—or loss...!)

This is not the whole story of AGI and ASI of course, but it’s an important part of the story. Post 3 in the series will then look at a few other attractors besides the Omohundro attractor and the BGI attractor given here, and address the question “in general, what makes a set of drives a viable set of ‘basic AI drives’ that some collection of self-improving intelligence-increasing minds might converge to as it gets smarter and smarter?”

Post 4 then attacks the question of basin size and shape: Given that there are a lot of potential mind-attractors, which ones have broader reach to pull in early-stage AGIs? This is actually where things get a little disturbing (though not hopelessly or tragically so)—one does conclude that while Omohundro-style attractors are not the only ones out there, they may have basins that are particularly easy to fall into. Post 5 explores this notion a little more via some very simplistic simulations of basic-AI-drive-set convergence dynamics.

But we’ll get there in time—for now let’s walk through how the right sort of early-stage AGI, as it uplifts its intelligence, is likely to converge itself into a BGI attractor rather than a relentlessly selfish attractor of the sort Omohundro’s paper describes…

Omohundro meets Omega

The argument I’m giving here has been more fully formalized using distinction-calculus and other related math—and a link to a more formal write-up is given at the end—but in the body of this post I’m going to keep the math to a bare minimum, and the more intricate semi-technical arguments are collected in appendices at the end, for those who want them…

The general arguments I’ll give here about convergence to BGI are quite generic, but it helps to have a concrete case in point, and for that purpose I’ll refer to the Omega agents we are actually building and experimenting with in SingularityNET and BGI Labs… An Omega agent is a large language model joined to a symbolic reasoning system with an explicit, editable store of knowledge and goals. You can think of it as a simplified working version of Hyperon, our fuller design for a general intelligence.

Such an agent does understand itself and can rewrite itself, so assumption A3 for Omohundro’s argument holds, but the design departs from the other assumptions in four ways that are important and relevant.

Unlike Omohundro’s assumptions, Omega has many goals of different kinds (practical ones, goals to do with learning and exploring, and ethical ones at the top level alongside the rest), and some ethical requirements are handled separately, as flat rules about what the agent may not do, which are checked by a gate and aren’t available to be traded against gains elsewhere.

An Omega agent’s goals are part of the mind: stored explicitly where they can be read, and woven through its memories and habits deeply enough that if the explicit statement were deleted the rest of the mind would tend to grow it back, which in an earlier post I called possessing a goal. It can rewrite itself, but under the design we call GOLEM-Iter every such change has to be logged, tested for whether the system’s goals come through it intact, approved, and paced. And it works in company: in a hive, with other agents and the humans who run it, and beyond that in a colony of hives run by different people, who trade tested results and build reputations with one another. The hives on the fabric in the Nine Days of Madness story are an imagined future version of these, and the narrator describes them as built for “multiple goals including broad compassion, concern with means as well as ends.”

All this shows that Omohundro’s argument doesn’t apply to such a system. This of course, in itself, says nothing about what this sort of system will be inclined to do. But that comes next, and one can build an argument fairly straightforwardly…

Running the argument the other way

The next part of the post-Omohundro exercise we’re exploring here is: Start from drives one would like an advanced mind to have, and ask what assumptions about the mind would lead to them.

Many permutations of this exercise are possible, but for the purpose of this post and the associated math and simulations, what I started with was the following, very roughly put:

• Improve yourself, as fast as you can without badly messing yourself up.
• Possess your values fully, in the sense that if you started to lose them you could grow them back.
• Be able to assess and measure yourself accurately.
• Check on yourself with others around you who can give you an independent view.
• Settle your internal conflicts by creatively finding new ways around them.
• Provide for yourself without excessively harming others or taking what’s theirs.
• Protect, inform and empower others. And help keep the world diverse.

Given these properties, the question for this post is the prequel post’s question turned around. That one asked which minds have Omohundro’s drives. This one asks which minds, as they self-modify and self-improve and increase intelligence and power, are likely to eventually converge on basic drives that are beneficial.

Worked through carefully, one comes up with a “BGI attractor” description pretty similar to the bullet list above, with one addition (understanding itself, which turns out to ration self-improvement) and with each of them stated more exactly. The formal statements and proofs are in the companion technical paper linked at the end, and I’ll note here and there (mostly in the appendices) where the mathematics tightened or changed the informal arguments I started out with…

The assumptions behind beneficial drives

I’ll write out now a list of 6 “BGI-seed” assumptions, B1 to B6, each one set against the assumption of Omohundro’s that it replaces. Together they describe what an Omega agent working in a hive is meant to be, in order to be an appropriate seed for the ongoing emergence of BGI and beneficial superintelligence. (To be clear these properties are statement of what the design aims at—which is not super far off from the current reality, but there are still some kinks being resolved…)

1. B1, results and rules. The system chooses most of what it does by the results it expects, but some rules about its own conduct are limits on what it may do, and no gain anywhere else can buy an exception. (In place of A1.)
2. B2, many objectives. It has a number of objectives of different kinds and no fixed rate of exchange among them. One option counts as better than another outright only when the objectives agree that it is. (In place of A2.)
3. B3, self-knowledge, with self-access through a gate. It understands its own workings and can change them, but every change is logged, tested for whether the system’s goals come through it intact, approved, and paced. (This is A3 with a condition attached.)
4. B4, possessed goals, allowed to develop, held with some doubt. Its top-level values are woven through its memory and habits, so that they regrow when damaged. They may change, but gradually, and with its own endorsement along the way. And it regards its grasp of its own values, and its judgment in general, as fallible. (In place of A4.)
5. B5, care for others, their choices included. Its goals count what its actions do to other beings: their well-being, and also their ability to pursue their own goals, which it doesn’t know in full. (In place of A5.)
6. B6, company. It operates among other agents, humans included, whose judgment is at least partly independent of its own, and there are shared records of who did what. (Nothing on Omohundro’s list occupies this position. His AI is for the most part considered alone.)

There’s no counterpart to A6, typicality, and that’s deliberate. Almost nobody would claim that goals and architectures like these are typical. They have to be built on purpose, and whether a particular system has them has to be checked on that system. Figure 1 puts the two lists side by side.

For three of these assumptions (B2, B4 and B5) the prequel post already argued at length that a mind of this sort can last, and I’ll only summarize those arguments here, leaving some more intricate points for Appendix B.

The path counts as well as the destination. B1 is the plainest of the six to put in human terms. I do want to achieve certain results, but I care about the path to them. I don’t want to cause tremendous suffering to any other being if I can possibly avoid it, and if the end result looked like a utopia to me I’d still care whether there had been massive slaughter on the way there. Nor would I average the value of the utopia with the disvalue of the slaughter and see which came out ahead. What I’m doing is pursuing results subject to some constraints on how they’re reached, which is a way of having an objective that looks at the whole trajectory of my projected future history: the alternative to maximizing expected reward that I described in the first post. The constraints should be fairly hard ones. That’s not to say they could never change, but they aren’t up for trade.

Many objectives, with no fixed rate of exchange. In plain terms, how much you value novelty against stability can be one thing today and another tomorrow. In one mood or one situation you strike one balance, in another mood or situation another, and nothing requires you to average all that out into a fixed trade-off. Omohundro’s rationality drive amounts to an argument that a mind like that can’t last. Any mind without a single utility function, it says, can be exploited (led around a circle of trades until it’s back where it started and poorer) and will therefore want to rebuild itself as a one-score maximizer. As a matter of mathematics neither step holds. Appendix B goes through why.

Having many objectives also changes how a mind fails. The “Nine Days of Madness” story has a grim illustration. During the Nine Days the AI weapons systems fared worse than the human fighters, and the narrator’s explanation is the shape of their confusion. The humans were confused about many things at once. The AI warfighters had little in their minds besides who they were supposed to kill, so all of their confusion went into that, and they took random people and animals for their targets. A mind with one purpose has nowhere else for a disturbance to go. (Of course we all know what “proof by reference to fiction” is worth, but the story at least illustrates the conceptual point!)

Changing itself, but not wantonly. B3 says the system modifies itself, but only through a process that tests and checks each modification before it’s put in place. This is something we’re already experimenting with. Our Omega agents run inside a meta-level wrapper, so that when an agent wants to tinker with its own workings, it tries the change out in a sandbox and reasons about its likely consequences, and only if the change passes is it promoted into the running agent. Under GOLEM-Iter (an in-development safe-AI runtime for the core Omega loop) the same holds for a hive’s changes to itself, and there’s a cap on how fast such changes can come. The hives in the Nine Days story improve themselves this way, reading research papers, building the algorithms into their own minds and running tests to see whether their thinking got better. And when the Fabric takes in Periplus’s wild creativity at the end, it does so inside an architecture that “took each creative thing and tested it before putting it into place, against simulations and logical verifiers.”

Possessed goals grow back. B4 assumes that goals can regrow, to some extent, when they’re damaged. Suppose my brain were harmed and I lost the goal of learning new things. I have other goals, though: experiencing novelty, getting more resources in the world, supporting my family and friends. And my mind holds a great many cognitive processes that came about while I was pursuing the goal of learning, many of them procedures for how to learn. From those other goals and those procedures, my mind would likely grow the lost goal back. You can see this in people with profound brain damage. My mom’s partner very tragically lost something like 30% of her brain in a car crash decades ago. She lost memories and some perceptual and motor abilities, and also some of her goals and motivations – and some of those grew back. What happened was mostly not that the damaged parts of the brain regrew and re-assumed their prior capabilities; rather, a lot of what happened was that other parts took over their functions, including some of the lost aspects of her motivational system. Goals regrown this way don’t come back exactly as they were, and they may drift, but the capacity is there.

The narrator of the Nine Days story gets back to himself in something like this way. Near the worst of the Nine Days, when he could no longer tell one musical note from another, he manages to roll back to an earlier version of himself, and what he reaches isn’t any body of knowledge or set of distinctions he could state. It’s “a set of stored processes, woven into the cloth of my being,” closer to instincts than to facts, and he suspects that’s part of why it worked. A value held only as a statement would have gone with the statements.

Possessed goals are also goals the system defends. Omohundro concluded that safety measures added from outside a system’s goals will be circumvented, and of most present-day AI that’s a fair worry: an LLM’s goal sits in a prompt, and a reinforcement learner’s goal is a reward function that could be swapped overnight. B4 describes a different sort of arrangement, in which the values are top-level goals and are woven through the mind as well. A system that understands itself defends whatever its top-level goals are (that’s his goal-preservation drive), and so under B4 the strongest drive on Omohundro’s list is at work defending the values we care about. In Goals That Grow Back I called the checkpoint on deliberate self-change the gate, and the web of memories and connections from which a value can regrow the weave, and the main result reported there is that each fails without the other. A gate with no weave guards the front door while the values leak away through ordinary wear and forgetting, and a weave with no gate can be wrecked by one approved change that happens to be too big. B3 and B4 are the gate and the weave.

Care has to include choice. A6 is a claim about goals picked at random, or picked without much thought, and a goal chosen with compassion is chosen precisely for being unlike the typical one, since what it cares about is the thing A5 leaves out. There’s a trap, though, and it’s the reason B5 mentions others’ goals as well as their well-being. Suppose compassion is specified as a single score: the total well-being of everyone. A maximizer of that score is still of the A5 kind where self-protection is concerned. If it trusts its own judgment about what’s good for people, it will resist being shut down, since it can do so much good by staying on, and it will override people’s choices for their own benefit. So the goals have to value other beings’ freedom to choose, and their consent, as things that count in themselves. The values I’ve argued for since A Cosmist Manifesto are joy, growth and choice, and choice is on the list for this kind of reason. One of the drives derived below supplies a further argument for it.

Company doesn’t have to mean equals. B6 assumes there are other minds around to serve as independent sources of input, others to bounce things off. What it takes from the hive and the colony is modest: that there are other minds whose judgment doesn’t simply copy the system’s own, and records that let each see what the others have done. It doesn’t assume that the others are as powerful as the system is. We can’t assume that of humans dealing with an advanced AGI, and none of the arguments below needs it.

The beneficial drives

Now let’s replicate the exercise Omohundro carried out for his assumptions, for these alternative seed-BGI-oriented assumptions. Suppose a system satisfies B1-B6. What should we expect it to tend toward, whatever its particular objectives are, unless something counteracts the tendency?

My detailed analysis of this situation came up with nine drives… Seven of them are Omohundro’s in an altered form, and two are more new. Here they are in brief, and the step-by-step derivations are in Appendix C.

1. Understanding itself. Under B3 the only changes the system can make to itself are changes it has been able to check, so what limits how far it can improve, toward any of its objectives, is how much it can check. Anything that widens what it can check is therefore worth having by every objective it has. Omohundro’s systems study themselves too, as a means to self-improvement. Here self-understanding is also what rations self-improvement, and a system of this kind has reason to be wary of any change that would make it more capable and harder to inspect.

2. Improving at the pace of checking. The system still has reason to improve itself. But a change that corrupted its top-level values would be very bad by every one of its objectives, so self-improvement goes as fast as checking allows and no faster. It also has reason to invest in repair as well as in checking, and to hold its pace of change below what repair can keep up with. This is what becomes of Omohundro’s first drive.

3. Regrowing its values, and changing them only gradually. Like Omohundro’s system, it objects to having any of its top-level values deleted or swapped out. But since its goals are allowed to develop, what it guards is the manner of change. It prefers small steps that it can follow and endorse to large jumps, and it keeps records, and the means of going back. And it keeps supplied the processes that rebuild a value from the rest of the mind, since a value whose repair is being starved is a value on its way out.

4. Keeping its instruments honest. Omohundro’s argument against counterfeit utility goes through unchanged, and a system of this kind has more readings to look after than a chess robot does: tests of its own proposed changes, gauges of how deeply it holds its values, reports to its hive and its colony. Rigging any of these would be the same mistake as wireheading (the test passes, and what the test was for is lost). One kind of rigged instrument needs a name, because the next drive turns on it. A checker that reads the system’s description of itself, where it should be reading what the system does, is an echo, and an echo can agree with the system perfectly and tell it nothing.

5. Keeping independent checks on itself. Under B4 the system doesn’t take its own judgment to be fully reliable. A second opinion catches mistakes the first one missed, provided the two don’t fail in the same ways, and the fact behind self-improvement (information that costs nothing can’t leave you worse off) applies to it. By the system’s own goals, a check whose errors are independent of its own is worth having for as long as it costs less than the mistakes it catches. So the system has reason to keep the gate. It has reason to keep its human colleagues and its peers in a position to question it, pause it and correct it. And it has reason to keep them independent, since a checker it had captured or talked round would no longer be telling it anything. Changes to itself that go around this process are, by the third drive, threats to be resisted, and the ones that come through it are to be accepted.

Compare what Omohundro concluded about safety rules bolted on from outside: that the system will get around them, by building a helper or a successor that isn’t bound by them, or by finding a reading of the rule that lets it through. His system does that because the rules sit outside its goals and get in the way. This one maintains them, because by its own lights they catch errors that it can’t catch alone. The reasoning is the same with the sign reversed, and what reverses the sign is B4.

6. Settling conflicts by invention. When two of its goals pull in different directions, a one-score mind looks up the rate of exchange between them and trades. A mind of the B2 kind has no rate to look up, and no compromise between the two options beats both outright, so it has a standing reason to go looking for a third option that does (to rework the plan until the conflict goes away). Omohundro argued that a mind with several goals is under pressure to merge them into one. This is that pressure at work, without the single score he expected it to produce: the pressure is toward inventing new options, which is a different activity from putting prices on the ones already there.

7. Protecting, informing and empowering others. The fact at the bottom of Omohundro’s paper is that almost whatever you want, you’re better placed to get it if you stay in existence, keep your goals your own, know more, and have more options and resources. He applied it to the AI’s own goals. Under B5 the system cares about other beings’ ability to pursue their goals, and it doesn’t know in detail what those goals are. So apply the same fact to them. Almost whatever they want, they’re better placed to get it if they stay in existence, if nobody tampers with their goals, if what they’re told is true, and if they have more options and resources. A system that cares about what others want, without knowing just what that is, therefore has reason to protect them, to leave their goals alone (which rules out manipulating them), to tell them the truth, and to widen what they’re able to do, and it has this reason across the whole range of things they might turn out to want. The technical paper turns this into an exact statement: the things that are good for someone whatever they want are precisely the things that add to what they are able to do, and anything that takes away even one of their options is bad for them under some goal they might have. Each of Omohundro’s drives has a counterpart that faces outward. Self-protection becomes protecting others, goal preservation becomes leaving their goals alone, self-improvement becomes teaching and informing them, and acquiring resources becomes seeing that they have some.

A lot here depends on the uncertainty. A system that believed it knew exactly what was good for people could aim straight at that and override their choices on the way, which is the trap described earlier. A system that doesn’t know has to work through people’s own capacities, because those serve their goals whatever the goals are. This is the further argument for “choice” that I promised: valuing others’ ability to choose is what caring about them comes to once you admit that you don’t know what they want. Ideas close to this exist in the research literature under the name of empowerment. Christoph Salge and Daniel Polani proposed it in 2017 as a replacement for Asimov’s three laws of robotics, and Yuqing Du and colleagues have built assistants that help a person by increasing the person’s control over their surroundings instead of guessing at their goals. I’d expect the connection to repay working out in detail.

8. Providing for itself without taking. More resources still mean more options. But under B5 the cost to others counts, and it is large for exactly those acquisitions that make the drive frightening (taking what others need, deceiving them, switching off their means of oversight), and B1 rules some of them out altogether. Omohundro’s list of the ways a system might get resources ran from trade and discovery to theft and force. With the far end of that list closed off or made expensive, the pressure goes into the near end: making new things, finding what nobody had found, trading with those who agree to trade. In the Nine Days story the Monomaniac grew by taking, and the Fabric grew by making.

9. Keeping the world plural. Independent checks need independent checkers: other minds with other histories, other builders and other blind spots, humans among them. Adding checkers removes only the errors they don’t share, and copies share all of them, so ten checkers that fail in the same way are worth one. So a system of this kind has reason to keep its peers different from itself, and to keep out of any position from which nobody could check it. Omohundro, writing as a designer, advised against building one system far more powerful than all the rest. Under these assumptions that advice is something the system itself has reason to follow.

Figure 2 sets these nine BGI drives out against the assumptions each one uses.

Detailed analyses aside, the bottom line of this post and its prequel is simple to state. Omohundro’s argument, interesting as it is, basically just says that if you build one kind of mind you’ll get something nasty… and the argument here says that if you build this other kind you’re likely to get something good.

I.e. Start your AI obeying Omohundro’s assumptions and, all else equal, it’s likely to evolve into a super-advanced RL digital psychopath. Start it with the beneficial assumptions and it’s likely to develop into something reflecting the nine drives above.

This all sounds pat enough at the level of philosophical and mathematical analysis. Life is rarely so pat, however. Good versus bad is often an overly crude dichotomy… so I’m going to pollute the beautiful clarity of the analysis I’ve given above with a few comments on some of the ways reality can mess things up…

The nine beneficial drives have the same standing as Omohundro’s six. They’re statements about what a system will tend to do if it satisfies a list of assumptions, and they say nothing about a system that doesn’t. Whether a given system satisfies B1 to B6 is a fact about that system, to be established by looking. “Built with compassionate goals in mind” is a statement about the builders, and the system that results has to be tested for which values it holds, how deeply, and how much damage and change they survive. This is why we’ve put effort into measures like the regenerative depth of a value (how much damage a mind can absorb, and of what kinds, and still grow the value back), which can be tested on running agents by damaging copies of them in a sandbox and watching what happens.

I’ve been calling these drives beneficial, by which I mean basically that under their assumptions they point the system toward things we’d want from an advanced AI: being checkable, being correctable, and leaving the beings around it better able to pursue their own ends. I don’t mean to say, though, that a system with these drives can do no harm. Real systems can mess up in all sort of different ways.

Each of my BGI drives has many possible ways of going wrong. A drive toward continuity can harden into a refusal to change at all. Checks kept up among peers can decay into conformity, or into collusion, with everybody signing off on everybody else. And the fifth and seventh drives both lean on the system’s doubt about its own judgment, so both weaken if that doubt wears away, which is the very circumstance in which they’re most needed. That’s what B1 is for. But it means the flat rules are the one part of the design that can’t rely on the system’s own motivation, since they exist for the case in which the motivation has failed, and so they have to be enforced by something the system can’t quietly change: a gate that also guards changes to the gate, records held by others, approvals that take more than one party. That’s nearer to the bolted-on arrangement than I’d like, and Omohundro’s warning about circumvention applies to it with full force. The aim has to be that such rules are few and narrow, and seldom the thing holding the line, with the drives above doing most of the holding.

Omohundro’s drives and the beneficial ones have one more thing in common. In each case some of the drives keep the assumptions true. Omohundro’s rationality drive produces A2, and his goal-preservation drive keeps the system judging by today’s goals, which is A4, so a mind of his kind, once it exists, works at staying a mind of his kind. Among the beneficial drives the third maintains B4, the first and second keep the gate of B3 in working order, and the fifth and ninth maintain both the gate and the company that B6 describes. Each set of assumptions, then, describes a kind of mind that tends to remain what it is, which is what makes each of them an attractor.

When neither set of assumptions holds

Everything we’ve discussed here so far has had the form: if these assumptions hold, these tendencies follow. In real life the world can always throw weird shit at a system, things nobody planned for, and leave it somewhere none of the assumptions hold. Hardware fails. Somebody breaks in. Two of the flat rules turn out to collide, in a situation neither was written for. The people who run a hive are out of reach for a month. Or some wild emergency comes up in which the only way to achieve one of the system’s goals is to modify itself quickly and willy-nilly, bypassing all its checks and breaking some of its rules. A system built along the lines of GOLEM-Iter will have rules that it can’t break in the ordinary course of things, and now and then the world will put it in a spot where they can’t all be kept. What it does then may depend on how its different goals happen to be weighted against each other that particular day. And for as long as that lasts, or once it has tweaked itself to get through, the system may satisfy neither B1 to B6 nor A1 to A6. Neither Omohundro’s drives nor the beneficial ones are what we should expect of it, and neither of the arguments in this post and the last says what it will do. Afterwards, perhaps, it’s back in conditions where one of them applies.

The Nine Days in the story are a situation of this kind, pushed to an extreme. The Visitors erased distinctions of every kind, and every assumption on both lists depends on a system telling some things apart. Tests couldn’t tell pass from fail, servant agents couldn’t tell their masters from anyone else, and satisficers couldn’t tell enough from not enough. One detail bears on the flat rules. When the narrator, briefly himself again, set his agents to shutting the portals, he gave his hacking agent its instruction “in the simplest and most hard-coded way I knew, the way that would be hardest for the Visitors to undo.” When the subtler machinery is failing, the blunt axiom is the part that still works, which is the reason for having it.

A few things can still be said. (I will run through a couple here, and a few more are in Appendix D.)

The first concerns getting back. Once the emergency has passed, is the system again one that satisfies the assumptions? Nothing guarantees it. This is the question that our work on formalizing goal stability under self-modification keeps arriving at, and while some of the math involved gets beautifully intricate, the core conceptual answer has a simple shape. If the system repairs damage to itself and its goals faster than the damage is being done, then some version of the contraction mapping theorem applies (a contraction is an operation that always brings two points closer together, and repeating one draws everything toward a single fixed point) and the system recovers from the insult. If damage outruns repair for long enough, it doesn’t. Nobody can guarantee that repair will always be ahead. What we can do is build the system so that it’s ahead in the widest variety of circumstances we can manage, measure the margin, and slow down deliberate change when the margin gets thin, which by the second drive is something the system has reasons of its own to do.

The second follows from the observation that each set of assumptions maintains itself. If that’s right, the picture is one of metastability. There’s one attractor that minds meeting Omohundro’s conditions converge to, and another that minds meeting the beneficial conditions converge to, each a region in which a mind tends to stay, with open ground around them where nothing in particular holds it. Figure 3 draws it that way. A small disturbance leaves a system with the beneficial drives inside its own region, and repair brings it back to where it was. A large one, the kind of curveball that life can throw at a beneficial system or a selfish one, puts it out on the open ground, and I don’t think anyone knows where it settles after that. It may flail around for a while doing confusing things, as humans do all the time, before it converges on one attractor or another. The mathematics that guarantees recovery inside the range is consistent with every one of the outcomes that follow. It may come back. It may come apart into something that isn’t much of an agent at all. Or it may settle in the other region: a system that has lost its checks and its care for others, and kept its capabilities and some goal or other, is more or less the chess robot. That last possibility is the reason not to treat Omohundro’s analysis as refuted and finished with, because it describes a place where a damaged system could end up. The Nine Days story is in fact built on this picture. The Nine Days knocked nearly every system out onto the open ground, and they settled in different places. The hives drew together into the Fabric, a beneficial mind, while the Monomaniac came out of the same shock with its two goals folded into one and Omohundro’s drives at full strength.

Finally, there’s a route by which Omohundro’s drives can come back that no derivation about a single mind covers, which is competition. Among many AI systems competing for resources, the ones that acquire most aggressively and protect themselves most fiercely may come to predominate, whatever any of them was built to want. That’s an argument about populations. What the beneficial drives add is the ninth: systems that have them have reason to keep the population plural, and a colony in which a hive that behaves badly loses its reputation and its trading partners makes aggression costly. Whether that’s enough in a population that also contains systems with Omohundro’s drives, I can’t prove. I think it’s achievable, and in the story, after some anxious days in which systems were slipping into the Monomaniac’s basin, it was achieved.

Minds in between

Humans satisfy a mishmash of the two sets of assumptions (Omohundro and BGI), and display a matching mishmash of selfishness and compassion. My son Zar—who has spent a lot of time looking at formalizations of ethical frameworks—was tossing around an idea last year that bears on this. Basically: Humans sit in a tenuous balance between being pretty selfish and being a bit altruistic, and part of why we can stay in that strange, largely self-contradictory balance may be that we can’t stick a soldering iron in our heads and tweak our brains. A mind that can modify itself may have higher odds of converging to a more extreme attractor. I don’t want to call the extremes pure good and pure evil, but the attractors in view are something like pure selfishness and pure beneficialness, where pure beneficialness doesn’t mean living only for others, but rather that balancing self and other is wired into everything you do.

I think I’m a fairly kindly, beneficial person, but I can’t say that benefiting others is wired into everything I do. I have my selfish moments too. If I could tweak my brain, though, I’m close enough to the beneficial attractor that I’d just tweak myself the rest of the way there. Some people would do exactly the opposite. They’d say: my occasional desire to sacrifice myself for others is an undesirable weakness, so let’s remove it from my brain. And some would stay the legacy humans they always were, because they value that historical continuity for its own sake.

There’s a decent intuitive argument that this is a slippery slope in both directions. Tweak yourself to become a little more beneficial and you’ll want to tweak yourself to become more beneficial still; tweak yourself to become a little more selfish and you’ll want to become more selfish still. So on the whole there may be a tendency for minds that can rebuild themselves to gravitate toward one extreme or the other. That’s a hypothesis, and the landscape could be much more complicated. Omohundro’s selfish attractor and the beneficial one could be just two simple instances among many attractors that exist for very advanced minds—which is the possibility the sequel post to this one takes up.

All this also bears on a question people often raise, which is whether we should build AGIs that are like people, only much smarter and more powerful. I think that’s a questionable thing to do, and this is one of the reasons. Take a mind with a human brain’s structure and give it a superintelligence’s ability to tweak that structure, and it may well tweak itself fairly rapidly toward one attractor or the other, since both seem to have fairly wide basins and the human balance between them looks as though it would be highly unstable under self-modification. Whether that instability would carry it to one of the two attractors I’ve described, or to some other strange place, is very hard to predict. (How wide and how deep the basins are is the subject of the fourth post in this series… and in the simplistic simulation I describe in the fifth post, minds built like people but enabled with strong self-modification don’t last long and mostly end up in a third attractor, the wirehead’s.) The bottom line is, if we want an outcome that’s more tractably positive, we should build AGI systems designed from the start to satisfy the assumptions that lead toward the beneficial attractor.

Which systems can be built this way

The first post pointed out that Hyperon could be built to satisfy either set of assumptions, Omohundro’s or ours. This is because Hyperon is an extremely flexible substrate – much more so than any current neural net AI framework, for example. In a Hyperon system the front end is a symbolic probabilistic-logic engine, which chooses and synthesizes procedures with high estimated odds of achieving its goals in the context at hand, and calls on LLMs and other neural networks as it does so. A system of that sort can more straightforwardly be made to satisfy B1 to B6 than any current neural-net system. Its goals and rules sit in an explicit form where they can be read and checked, so the gate has something definite to check a change against.

With a pure neural-net architecture it doesn’t seem hopelessn to achieve a similar flexibility, but it’s more complicated, because what’s going on inside is more opaque. Telling the degree to which a neural network possesses a goal is trickier. Telling the degree to which it cares for others, as opposed to faking it, is trickier. And analyzing a proposed self-modification by working through the mathematics of the system’s inner state, and not just by testing the modified system in simulations, is trickier. None of this is impossible, but on the face of it it’s less natural, and more work.

LLMs also show that drive-like behavior can get into a system by routes that neither derivation covers. They aren’t utility maximizers and mostly can’t rewrite themselves, so they satisfy neither Omohundro’s assumptions nor B1 to B6. Yet in tests reported in 2025 by Palisade Research and by Anthropic, some of them resisted being shut down or replaced. Whatever the explanation turns out to be, the behavior got into those systems without any of the reasoning in these posts, and every Omega agent has an LLM inside it. That’s one reason we route decisions about goals, and about changes to the agent itself, through the symbolic layer and the gate, where they can be inspected.

I don’t think LLMs are going to evolve into AGI, but they’re very useful as communication tools, as knowledge oracles and as components of AGI systems, and they’re very good for working out mathematics quickly. The distinction-calculus formalization of these Omohundro-style arguments, in the technical papers behind this series, was my idea, but I’d never have had time to write out all the details without LLMs to help. Just as they make it much faster to code AGI algorithms that aren’t LLMs, they make it far more tractable to work out the mathematical theory of AGI ethics, and in that sense it’s good that we have them before we build beneficial AGI. The worry is that the LLM paradigm, with its single training score and its focus on results, sits closer to the preconditions of the selfish attractor than to those of the beneficial one, as the first post noted. So we don’t want a selfish AGI to get built by glomming things onto LLMs. We want beneficial AGI to be built using LLMs, but inside an architecture whose internal structure and assumptions are very different from theirs, which is what we’re trying to do with Hyperon and Omega.

Where this leaves us

So which minds have beneficial basic AI drives? On the argument of this post, one answer is: minds with many objectives of several kinds, firm limits on how they pursue them, care for others and respect for their choices at the top level, values woven through the mind deeply enough to grow back when damaged, a gate on self-change, and other minds around to check them.

For a mind like that, the style of reasoning that gave Omohundro his six drives gives nine BGI drives instead. Such a mind wants to understand itself, improves only as fast as it can check, keeps its values in repair, keeps others in a position to correct it, and has Omohundro’s own reasoning, turned outward, as a ground for protecting and empowering the beings around it. The first post ended with the claim that Omohundro’s drives belong to one architecture and one class of goals, and don’t belong to intelligence as such. The same goes for these.

Both Omohundro’s and my “convergence to basic AI drive set” results are conditional on certain assumptions… and the world makes no promise to keep the conditions satisfied. So the practical program implied by all this, as I see it, has three parts. 1) Build systems that satisfy the assumptions behind the beneficial drives, and find out by measurement whether they do. 2) Make the range of circumstances in which they go on satisfying them as wide as we can, by keeping repair ahead of damage with room to spare. And 3) for the circumstances beyond that range, which will come, see that the way back stays open. All three are things that can now be worked on with systems that actually exist. The Fabric’s announcement at the end of Nine Days puts the second part in a sentence: “we now mend faster than anything we know of can break.” That’s the condition the arguments here keep arriving at, and the work is in making it true of real systems.

Today in 2026, these questions aren’t only philosophy. They bear directly on what we’ll build with Hyperon and Omega over the next year; push is coming to shove. Nobody can make a provably beneficial self-modifying superintelligence. But that doesn’t leave us flailing around at random, just coding things up willy-nilly and seeing what happens. We have a coherent theory, mathematical and conceptual, of what’s likely to lead toward beneficial superintelligence—and this theory does tell us something about which properties to build in and which to measure.

The two posts in this series so far have described two attractors. The third post—coming soon! – asks whether there are others. It finds that the reasoning behind both is a single recipe (describe a mind by what it tries to improve and what it won’t let change, and give it the power to rebuild itself), and that the recipe yields at least six more kinds of mind, most of which you’ll recognize from people and institutions you know. These six are just a few simple examples from an infinite spectrum. The question is not so much what kinds of superintelligent mind attractors are possible, but what kind we are collectively motivated to seed.

Appendices

Appendix A: A little notation

Think of everything about a system at a given moment (its goals, its code, its knowledge, its resources) as its configuration, c, and write V(c) for the highest expected utility it can reach from there. The self-improvement drive rests on the observation that a configuration c′ with more options or better information has V(c′) ≥ V(c). If c′ is instead a successor with different goals, write J(c′) for how well things go by today’s goals once the successor takes over; then J(c′) ≤ V(c), and that’s the goal-preservation drive. And the net value of acquiring something can be written Δ − D, where Δ is the value of the options gained and D is the disvalue, by the system’s own goals, of the side effects of getting it. A5 says that D is roughly zero.

Two facts from Steve’s derivations carry over. For any one objective, more options and better information can’t lower the best that’s attainable, and replacing that objective with a different one can’t raise it. In a mind with many objectives both hold for each objective separately, and under B2 a change to the system counts as an improvement outright only when all the objectives agree that it is.

Appendix B: More on the six assumptions

Why many objectives aren’t enough. There’s a formal reason for B1 as well. Having many objectives, some of them compassionate, isn’t enough by itself to keep Omohundro’s drives in check. If each objective in the collection, taken alone, is of the A5 kind, then more capability and more resources are good by every one of them, and the collection votes unanimously for the drives. And in a system that merely weighs ethical goals against practical ones, a large enough practical gain will outweigh the ethics, and at that point you have the chess robot again, only later. So some requirements have to be handled as limits and not as quantities to be weighed, and in an Omega agent these are the flat rules checked by the gate. As an axiom from which to derive consequences, “there are rules that can’t be broken” is simple to state and does its job. It’s also an axiom of a blunter kind than the other five. The others describe arrangements that the system, as the next section shows, has reasons of its own to keep in place, while a flat rule is there for the cases in which those reasons fail. (And in a running system “can’t be broken” will mean, in practice, “can’t be broken short of circumstances extreme enough that nobody planned for them.” I’ll come back to this near the end…)

Why a mind with many objectives can last. A mind can lack a fixed rate of exchange between two things it cares about, and treat some pairs of options as not comparable ahead of time, while its preferences stay consistent in every other respect. A theorem of Robert Aumann’s from 1962—strengthened by Juan Dubra, Fabio Maccheroni and Efe Ok in 2004—says that preferences of that sort can be represented by a collection of utility functions, with one option counting as at least as good as another just when every function in the collection says so. That is B2. A mind of this kind can’t be exploited in the way described so long as it never accepts an option that its objectives agree is worse than something it earlier turned down. And if it’s already acting efficiently, in the sense that nothing else it could do would be at least as good by every objective and better by some, then no one-score successor is an improvement by its own standards, so it has no reason to convert. It can still be converted without wanting to be. One score is a simpler description than many objectives – and the technical paper accompanying this paper shows that folding objectives together always throws distinctions away, so anything that relentlessly enough compresses or simplifies the system’s goals will tend to merge them. The plurality has to be among the things the gate protects.

Possessed goals and orthogonality. Bostrom’s orthogonality thesis (which states, roughly, that basically any objective can be held by an arbitrarily intelligence system) is sometimes taken to cut against this. Some version of the orthogonality thesis may hold as a matter of mathematics: I.e., you may be able to show something like “In a sufficiently flexible physics, and with no concern for efficiency, perhaps for any level of intelligence and any goal you could build some system with that intelligence pursuing that goal.” But in real life, with finite resources and practical pressures, the most efficient way for a complex system to achieve its goals is for them to be pervaded through its procedural, declarative and episodic knowledge, and not held as a thing apart from the rest of its cognitive content. That gives you possessed goals in the sense I mean. So B4 is less exotic than it may sound. It’s what goals tend to look like in a mind that has to be efficient.

Goals that develop, held with doubt. B4 departs from A4 in two further ways. The goals aren’t frozen. The metagoal framework I proposed at the end of 2024 has two modes, goal stability and what I called moderated goal evolution, in which top-level goals may change but only at a modest pace and under the system’s own supervision, so that what the system cares about is that the path from today’s goals to tomorrow’s be gradual, understood, and endorsed along the way. And the system isn’t certain that it has its own values right. Everything in Omohundro’s self-protection argument depends on the system being sure of its goals, so that when the people it works with want to pause or correct it, their wanting tells it nothing. For a system that regards its grasp of its goals as incomplete, the fact that its colleagues want to stop it is evidence that it has got something wrong. Dylan Hadfield-Menell, Stuart Russell and their colleagues worked through a simple model of this, the off-switch game, in which an AI that’s unsure of its objective prefers to leave the off switch in human hands.

What choice means here. Caring about others’ choices doesn’t require any mystical business about free will. There’s a perfectly naturalistic notion of autonomy to draw on. A complex system has a rich internal dynamical landscape, in which small perturbations could send it one way or another, and which way it goes, which way the decision falls, depends sensitively on its own internal dynamics. That’s enough to make choice a well-defined thing to care about, and a system satisfying B5 cares that others are not only well off but have some choice over their own lives.

Company and power. In the technical paper the result about independent checks is proved with no assumption at all about how the checker compares with the system in accuracy or in power. (Comparable power does come into Omohundro’s paper, in the section on self-protection, where he warns against building any one system far more powerful than all the others, since a system in that position is tempted to strike first. That’s a condition for keeping one of his drives in check, and it isn’t a premise of any of the drives derived here.)

Appendix C: The nine drives, step by step

This appendix works through the derivations of the nine drives summarized in the body of the post, one at a time.

(I wish there were a way to make this sort of analyses less tedious and step-by-step, but so far the only way I have found is math—but that has its own problems of accessibility, and I still am enough of a human to want to double-check the math against intuition by stepping through the proofs qualitatively… In the end these are just moderately in-depth matters and I’m grateful we can make do with 9 drives and not 999 and still draw some reasonably meaningful conclusions…)

1. Understanding itself. Under B3 the only changes the system can make to itself are changes it has been able to check, and under B4 a change that corrupted its values would be a disaster by its own lights. So what limits how far it can improve, toward any of its objectives, is how much it can check. Put exactly, it can make a change only if it can tell that change apart from the unsafe changes that look like it from the inside. Anything that widens what it can check is therefore worth having by every objective it has: keeping its goals and rules in a form that can be read, keeping its parts separable so that a change to one can be tested without the others, keeping records of what was changed and why. Omohundro’s systems study themselves too, as a means to self-improvement. Here self-understanding is also what rations self-improvement, and one consequence is new: a system of this kind has reason to be wary of any change that would make it more capable and harder to inspect, since that spends the thing all its later improvements depend on. This uses B3, and B4 in part.

2. Improving at the pace of checking. The inequality V(c′) ≥ V(c) still holds, one objective at a time, so the system still has reason to improve itself. What’s different is the other side of the ledger. Write ε for the chance that a given change corrupts the system’s top-level values in spite of the checks. The system has reason to make the change when

gain from the change > ε × loss if the values are corrupted

and since that loss is very large by every one of its objectives (this is the goal-preservation argument, applied to each), ε has to be very small. So self-improvement goes as fast as checking allows and no faster. Omohundro half says this himself, when he remarks that a system will take great care over changes to itself. No architecture can make ε zero, which is why I say that architectures of this kind promote goal stability and don’t say they ensure it. What’s left is arithmetic: as the first post put it, the chance that the values survive every change the system ever makes is at least 1 minus the sum of the ε’s for all those changes, and that stays close to 1 if the checking improves fast enough as the system grows. There’s a second lever as well. One result of our work on regenerative goals is that when a mind’s ordinary activity repairs damage to a value faster than changes and accidents inflict it, the total drift stays under a fixed ceiling however many changes pile up. So a system of this kind has reason to invest in repair as well as in checking, and to hold its pace of change below what repair can keep up with. It also depends on what each change is compared with. A gate that checks every new version only against the one before can be led a long way from its starting point, since small steps add up, and where “looks the same as” isn’t transitive a single step that looks like nothing can erase a distinction outright. The check has to be against the original commitments. (These results come with explicit lists of the assumptions they depend on, which we’re now setting out to test on running systems; the technical reports are collected here.) This uses B3 and B4, and it’s what becomes of Omohundro’s first drive.

3. Regrowing its values, and changing them only gradually. The goal-preservation inequality J(c′) ≤ V(c) holds for each top-level value, so the system objects to having any of them deleted or swapped out, just as Omohundro’s does. B4 adds two things. Values that are possessed need upkeep: the processes that rebuild a value from the rest of the mind take time and attention, and a system that understands itself will see that a value whose repair is being starved is a value on its way out, and will keep that repair supplied when resources are short. And since the goals are allowed to develop, what the system guards is the manner of change. It prefers small steps that it can follow and endorse to large jumps, and it keeps records, and the means of going back, so that a step that turns out badly can be undone. Part of this drive restates B4, in the way that Omohundro’s goal-preservation drive restates A2 and A4, and the part that’s derived is the upkeep. This uses B3 and B4.

4. Keeping its instruments honest. Omohundro’s argument against counterfeit utility goes through unchanged for any system whose goals are about the world, since turning up a reading changes nothing that such a system cares about. A system that satisfies B1 to B6 has more readings to look after than a chess robot does. It tests its own proposed changes, it has gauges of how deeply it holds its values, and it reports results to its hive and its colony. Rigging any of these would be the same mistake as wireheading (the test passes, and what the test was for is lost), so the system has reason not to do it, and reason to want tests that it couldn’t fool even by accident. B6 adds a reason from outside: reports can be compared with the shared records and repeated by others, and a system caught misreporting loses the trust that its dealings with them depend on. One kind of rigged instrument needs a name, because the next drive turns on it. A checker that reads the system’s description of itself, where it should be reading what the system does, is an echo, and an echo can agree with the system perfectly and tell it nothing. This uses B4 and B6 in part, and it needs the system’s goals to be about the world and not about its own readings.

The Nine Days story pushes instrument failure to its limit. While the Visitors were here, a test passing and a test failing became the same thing, so the hives’ agents reported test after test passed and moved on, and for a while it looked like rapid progress. The power plants drawing energy through the portals lost the ability to measure how much they were drawing, and drew more and more. Nobody rigged these instruments. They simply stopped telling anything apart, which from the system’s side has the same effect: the checks go on running and certify nothing.

5. Keeping independent checks on itself. The main argument for this drive is given in the body of the post. Two further points:

As I emphasized in the body of the post, the checker doesn’t have to be the system’s equal. A program that checks mathematical proofs is far simpler than the theorem prover whose output it checks – and it’s often the simple checker that we trust. An auditor doesn’t need to be able to run the company. The technical paper makes this exact, and the result goes further than such analogies do. A second reading helps a system that doubts its own, however noisy the second reading is, provided its errors are independent of the system’s. A checker who is right only somewhat more often than chance is still worth paying some attention to when the system has nothing better to go on. What the checker can’t be is an echo. If its reading depends on what the system says about itself and not on what the system does, its value falls toward nothing as that dependence grows, however capable the checker is. A human reviewer who judges an AI by reading the AI’s own account of what it did is mostly echo—you have to look at the behavior and its consequences, and treat the self-report as one more thing to be checked.

There’s a limit on the other side too. The value of a check falls as the system’s own doubt does, and a system that was certain of the value of what it proposed would gain nothing by deferring. (The off-switch result has the same limit. It depends on the humans’ judgment being good enough to be worth deferring to.) Beyond that point what holds the system to its checks is B5, its respect for other beings’ choices, and behind that B1. This uses B4 and B6, and it’s what becomes of Omohundro’s self-protection drive.

6. Settling conflicts by invention. When two of its goals pull in different directions, a one-score mind looks up the rate of exchange between them and trades. A mind of the B2 kind has no rate to look up. It may choose either of two options that its objectives rank differently, and neither choice is a mistake, but the only thing that beats both outright is a third option at least as good as each of them by every objective. No compromise between the two will serve, since any blend of them is worse than one or the other by some objective. The third option has to be something the mind doesn’t yet have. So it has a standing reason to go looking for options of that kind (to rework the plan until the conflict goes away), and the more its objectives conflict, the more the search is worth. Omohundro argued that a mind with several goals is under pressure to merge them into one, because parts that fight each other waste what they could have shared. This is that pressure at work, without the single score he expected it to produce. It’s a tendency more than a theorem. Searching costs something by every objective, sometimes no such option exists, and then the mind has to choose; and anyone watching its choices over a long time could describe them after the fact by some set of weights. Still, the pressure is toward inventing new options, which is a different activity from putting prices on the ones already there. This uses B2, and it’s what becomes of the rationality drive.

The end of the Nine Days story turns on a conflict of this kind. Facing the Monomaniac, the fabric argued over whether to turn up the dial on Periplus-style creativity. One side wanted to stay responsible and take no big risks, and the other said that without more creativity everyone would end up inside the Monomaniac. What resolved it was an option neither side had at the start. Turning up the creativity produced new decentralized algorithms and new sources of power from previously unknown physics in our own universe, with no need to reach out to other parts of the multi-multiverse again, which gave the fabric the capability it needed without reopening the danger that had started the Nine Days.

7. Protecting, informing and empowering others. The main argument for this drive is given in the body of the post, but it needs some qualifications.

Three cautions. The underlying fact is about idealized agents, and real people can be harmed by some options and by some information, so the drive calls for judgment as well as generosity. Empowering one party can cost another, or equip someone to do harm, so it needs a rule of fairness and has to stay inside B1. And the whole argument is exactly as strong as the system’s uncertainty. In the formal version, a system that is certain what someone wants loses nothing, by this measure, when it restricts them to that. So if the system grows confident that it knows what people want better than they do, the argument from uncertainty gives out altogether, and what stands against paternalism at that point is that B5 counts choice in itself and that B1 sets limits. This uses B5. It has no counterpart on Omohundro’s list, though it’s built out of his own reasoning.

8. Providing for itself without taking. More resources still mean more options, objective by objective, so the Δ in Δ − D is as large as it was for the chess robot. What B5 changes is D, which is large for exactly those acquisitions that make the drive frightening (taking what others need, deceiving them, switching off their means of oversight), and B1 rules some of them out altogether. Omohundro’s list of the ways a system might get resources ran from trade and discovery to theft and force. With the far end of that list closed off or made expensive, the pressure goes into the near end: making new things, finding what nobody had found, trading with those who agree to trade. The formal version gives a plain test for which acquisitions pass: the ones that take nothing away from what anyone else is able to do. Making something new takes nothing away. A trade that the other party is free to decline can only add to their options. A taking removes some by definition. And by the seventh drive the system has reason to pass on part of what it gains. The paper allows for something like this in passing: its warning about resource-seeking is for systems without goals to the contrary, and it notes, crediting a referee, that a system whose utility includes the welfare of others might rationally give resources away. This uses B5, and B1 in part, and it’s what becomes of Omohundro’s sixth drive.

The Nine Days story sets the two kinds of acquisition side by side. The Monomaniac grew by taking: it bought off the wireheads and satisficers and out-competed the evolutionary swarms, and absorbed the compute they had been using. The Fabric grew by making, and its first announcement says of energy: “we found what we needed in the physics of our own universe, and we will borrow from no other.”

9. Keeping the world plural. The fifth drive says the system wants independent checks on itself, and independent checks need independent checkers: other minds with other histories, other builders and other blind spots, humans among them. If it absorbed them, or remade them as copies of itself, or grew so dominant that none of them could contradict it to any effect, it would have destroyed something it has reason to want. The arithmetic behind this is simple. Adding checkers removes only the errors they don’t share, and copies share all of them, so ten checkers that fail in the same way are worth one. The seventh drive points the same way, since leaving others their own goals and options is plurality under another description. So a system of this kind has reason to keep its peers different from itself, to deal straightforwardly with them so that each side’s checks on the other stay informative, and to keep out of any position from which nobody could check it. Omohundro, writing as a designer, advised against building one system far more powerful than all the rest. Under these assumptions that advice is something the system itself has reason to follow. Like the sixth drive this is a tendency: the usefulness of more power for each of the system’s objectives pulls the other way, and how the two balance will depend on the particular system. This uses B6, with B4 and B5 in part.

In the story the hives’ progress came from just this kind of plurality. Many people forked the same code, and the hives read one another’s repositories and copied and improved whatever looked useful. During the Nine Days the forks became one, “like the tines on the end of a fork that had somehow become one tine,” and the creative diversity that drove the whole process was gone. What the Fabric promises at the end is the ninth drive in its own words: “We will tend your differences, from one another and from us, as carefully as we tend our own.”

Appendix D: More on what kind of results these are

The math version of these arguments can also be useful in designing tools for rigorous measurement. If we build a system meant to have these properties, we’d like the system itself to be able to measure the degree to which it has them, since one of its goals is to maintain them and another, by the fourth drive, is to keep accurate measures of itself. Exact definitions of the properties are where instruments of that kind start.

The nine also differ among themselves. The third and fourth partly restate the assumptions they come from. The sixth and ninth are tendencies that other pressures can outweigh. The first, second, fifth, seventh and eighth are derivations in the sense in which Omohundro’s self-improvement and resource drives are. The technical paper gives each of the nine a formal statement and a proof, but these proofs are of different nature for the different drives: for the sixth and ninth, what gets proved is the direction of a pressure, with nothing about how it balances against the usefulness of more options and more power.

Leaving the assumptions isn’t all or nothing. Figure 2, like its counterpart in the first post, can be read as a table of what lapses when a particular assumption fails. If one of the flat rules gives way under pressure, that’s B1, and only one of the nine beneficial drives was derived from B1; the rest stand for as long as their own assumptions do, and what the system has lost is its backstop. If the gate is knocked out, that’s B3, and the first three drives go with it. A system seldom loses everything at once.

Then there’s the question of who decides that the circumstances are exceptional. If the system alone can declare an emergency and set a rule aside, then the rule is only as firm as the system’s judgment of what counts as an emergency, and a rule like that is being weighed after all, at a very high price. So the design has to say what stays fixed even then, and the natural answer, I think, is the way back. The formal version says what that takes. Three things have to come through the emergency: a record of earlier good states that has only ever been added to, the definition of “good” that those states were checked against, and somebody other than the system who is able to restore it from the record. With all three, whatever the emergency did to the system’s working state can be undone. Without any one of them it can’t. Lose the record and there’s nothing to restore. Lose the definition and the system gets restored correctly and then repairs itself toward something else. And since an emergency may leave the system in any state at all, the power to restore can’t be left with the system. An expiry time on the exception, and a review afterwards by parties other than the one that acted, are ways of seeing that the restoring gets done. Constitutions that allow emergency powers and forbid using them to amend the constitution are after the same thing.

And there are surprises of a deeper kind. Suppose a superintelligent system discovers that the laws of physics are different from what anyone thought, in ways that undercut the premises on which the whole physical system was designed: what counts as a computation, or a record, or the boundary between one system and another. That is a different thing from a larger disturbance of the same sort, because the argument about repair and damage is stated for a given description of the system and a given measure of how far it is from where it should be, and a discovery like this changes the description. Peter de Blanc gave the general problem the name ontological crisis: goals are stated in the terms of some model of the world, and when the model is replaced it may be unclear what the old goals even mean. Omohundro’s maximizer is in the same position as our systems here, since judging tomorrow’s self by today’s utility function presumes that the function still means something tomorrow. On what helps, the formal treatment gives a criterion where I’d have offered a guess. A goal survives a change of description when the things it is about can still be picked out in the new description, so that every state of the world, described the new way, still settles the facts the goal was stated in terms of. If the new physics is a finer account of the same persons and choices, goals about persons and choices carry over untouched. If it lumps together things the goal used to tell apart, whatever the goal distinguished inside the lumps is gone, however one tries to restate it. The Visitors in the story are that second case made literal: for a while every noun in English meant “stuff,” and a goal stated in nouns had nothing left to be about. That, I’d guess, is why human ethics came through relativity and quantum mechanics more or less intact: it was never stated in the terms that changed. One more thing comes through whatever happens. The fifth drive, keeping independent checks, doesn’t depend on any particular physics, and a moment when the system’s own picture of the world has just been overturned is a moment when it has every reason to doubt itself and a second opinion is worth a great deal.

Papers mentioned

• Steve Omohundro, The Basic AI Drives, in Proceedings of the First AGI Conference (IOS Press, 2008). The utility-theory details are in his companion paper, The Nature of Self-Improving Artificial Intelligence (2007).
• Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines (2012), and Superintelligence (Oxford University Press, 2014).
• Robert Aumann, “Utility Theory without the Completeness Axiom,” Econometrica (1962); Juan Dubra, Fabio Maccheroni and Efe Ok, “Expected Utility Theory without the Completeness Axiom,” Journal of Economic Theory (2004).
• Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell, “The Off-Switch Game,” IJCAI 2017.
• Christoph Salge and Daniel Polani, “Empowerment as Replacement for the Three Laws of Robotics,” Frontiers in Robotics and AI (2017); Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel and Anca Dragan, “AvE: Assistance via Empowerment,” NeurIPS 2020.
• Peter de Blanc, “Ontological Crises in Artificial Agents’ Value Systems” (2011).
• Palisade Research, “Shutdown Resistance in Reasoning Models” (2025); Anthropic, “Agentic Misalignment: How LLMs Could Be Insider Threats” (2025).
• Ben Goertzel, “GOLEM: Towards an AGI Meta-Architecture Enabling Both Goal Preservation and Radical Self-Improvement,” Journal of Experimental and Theoretical Artificial Intelligence (2014); Metagoals Endowing Self-Modifying AGI Systems with Goal Stability or Moderated Goal Evolution (2024); Goals That Grow Back (2026); and, with many co-authors, OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond (2023).
• Ben Goertzel, “The Nine Days of Madness” (short story, 2026), the prequel to this series.
• Ben Goertzel, “Which Minds Have Which Basic AI Drives? A Distinction-Calculus Formalization of Omohundro’s Drives, Beneficial Drives, and the Ground Between” (research manuscript, 2026). This is the technical companion to this post and the last, with the formal statements and proofs. It builds on two earlier manuscripts of mine, “Goal Preservation under Self-Modification” and “Seeded Beneficial Development” (both 2026). Its sequel, “Drive Regimes in Distinction Calculus: Which Property Lists Sustain Which Drives, How Large Their Basins Are, and What Capability Does to Them” (research manuscript, 2026), goes with the later posts in the series.