Welcome to Post #4 in a 5-post series on Basic AI Drives. The first three (Part 1, Part 2, and Part 3) described several kinds of mind, each with its own basic drives and each tending to stay roughly the kind it starts out as: the power-seeking maximizer of Steve Omohundro’s 2008 paper, the beneficial kind that my colleagues and I are trying to build, and half a dozen others.
Knowing what kinds of superintelligent minds are attractors is only part of the puzzle, though. The next piece is: What are the shapes of the basins of these attractors?
I.e, the zillion-dollar (actually, well, money-obsoleting…) question is: Which kind of attractor will a given early-stage AGI system fall into and how hard will it be to knock out again. And this depends on how wide and how deep the basin around each one is.
So this post analyzes these attractor basins a bit, with an aim toward milking lessons for building AGI systems and for understanding what an AGI system should do when something strange happens to it that may potentially zoink it out of its historical attractor basin. (The fifth, follow-on post will put rough numbers on some of this with crude simulations.)
The disturbing yet unsurprising punchline here is: While Omohundro’s original “Basic AI Drives” paper overstated the omnipresence of the “selfish” attractor in the space of superminds … nonetheless it DOES seem that the selfish attractor has a big, wide and deep basin that makes it especially easy for AGI systems to fall into it… if they are not designed just right…
This doesn’t mean you need some sort of extremely carefully calibrated mathematical design to avoid convergence to selfishness—but it does mean that if you don’t bother to build things like compassion and goal-continuity into your AI system’s initial goal framework, you have no reason to expect your system to converge elsewhere than to a selfish Omohundro-esque attractor as it self-modifies and its intelligence and capability increase.
Mapping the landscape of supermind drives
Throughout this series I have used the picture of a landscape with valleys. Each kind of mind is a valley, and a system left to its own choices rolls to the bottom of whichever valley it is in and stays there. Each valley is an attractor in this sense: start off as something of a selfish jerk, become more powerful and able to modify yourself, and you’re likely to become more and more of a selfish jerk; start off kindly and beneficial, and you’re likely to converge toward the center of the beneficial basin. The third post put it in terms of billionaires, who often become more fully what they already were. The next question is what these basins look like.
This post is about the contours of the landscape: how wide the valleys are, which way the ground slopes between them, how high their walls are, and what the walls are made of. The mathematics is in the same technical paper as the previous post drew on, and I’ll keep the formulas there.
As in the earlier posts, I’ll borrow examples from “The Nine Days of Madness”, the story I published as a prequel to the series. Its Nine Days are, among other things, a test of every basin in this post at once: the surrealist alien invasion in the story is an insult from outside that knocked nearly every mind out of the place it had settled, causing each of them to settle into a potentially different attractor afterward…
The short version
One big conclusion one gets from thinking through the basis of supermind attractors is that what keeps you in a basin while you’re an ordinary general intelligence, a human or an early-stage AGI, is different from what will keep you there once you’re a superintelligence.
A lot of what holds people where they are is that changing is hard, or costly, or frightening, or too hard to think through, and a supermind can get past every one of those. What’s left is a constraint that a mind has decided is very important to it, and a rule that its goals all have to agree before any of them is given up.
A basin is deep when whatever keeps a mind in it can survive damage and grow back afterwards, and doesn’t fade as the mind gets smarter or more confident. A basin is wide when a lot of different starting points roll into it. That happens when a little of the drive is enough to start a self-reinforcing loop, the way a little of an addictive drug makes you want more, and when getting in means giving something up and not adding something, since giving up care is easier than adding care.
By those measures Omohundro’s basin is both wide and deep. The wirehead’s is wide for any system that can reach its own dial, and bottomless. The obedient servant’s and the satisficer’s are easy to get into and shallow. The beneficial basin is very deep once a mind is thoroughly in it, but for many kinds of system it’s narrower than the selfish one, because caring well about others takes more machinery than caring only about oneself.
Why the size of a basin isn’t a single number
It would be nice to say that the beneficial basin is such-and-such a fraction of the whole space of superminds, but unsurprisingly, life post-Singularity isn’t that simple…
What the workings of a system do determine is this: for each condition the system might start in, the chance that it ends in each valley. A system that starts at the bottom of a valley ends there. One that starts on a ridge may have even odds. Call that the landscape. It’s a fact about the dynamics and depends on nothing else.
The obvious next move is to average those chances over all possible starting points and call the result the size of each basin. The trouble is that “all possible starting points” doesn’t say how much each one should count, and the answer depends entirely on that. Suppose you describe a mind by which of twenty values it holds, each yes or no, and count every combination equally. A basin that needs most of the twenty in place then comes out as a sliver, since very few combinations have most of them. Describe the same minds instead by the handful of recipes that might be used to build them, count each recipe equally, and the same basin might be a third of the whole. Same minds, same dynamics; all that changed is what “count every starting point equally” was taken to mean. There’s no neutral choice to fall back on, because any way of listing the possibilities builds in a view about which ones are alike.
You might hope the choice washes out in the long run, the way the weather on a given day stops mattering to the climate. Sometimes it does: a system that eventually forgets where it started has long-run behavior you can describe without asking where it began. But having several valleys means precisely that the system doesn’t forget. Where it ends up depends on where it started, permanently, so the average carries the weighting all the way through. The technical paper shows that by choosing the weighting you can get any answer between the lowest and highest chances the landscape contains.
Geography has the same distinction. The area of land that drains into the Atlantic is a definite number, because land area is a natural way to weigh one place against another. But it isn’t the fraction of the continent’s rain that reaches the Atlantic. That depends on where the rain falls, and the same rivers under a different climate send their water elsewhere. For minds there isn’t even an agreed equivalent of land area. Weighting all goals equally is what “most” means in the theorems about power-seeking. Weighting goals by how simply they can be stated is, as the first post showed, already tilted toward goals that ignore how results are reached. Either way the number answers a question nobody faces, which is what happens to a mind picked at random. Nobody builds minds at random.
For AI, where it rains is decided by what people build. If the systems being built are trained from nothing against a reward signal, they start in one part of the landscape. If they are grown from an enormous body of human writing, they start in another, with a great many human distinctions already in place, including distinctions about what happens to other people. If they are built on purpose with protected goals and a gate, they start in a third. So the question of how big the basins are is partly a question about what we build first…
Some things can still be said without choosing where it rains. If one valley’s chances are at least as high as another’s from every starting point, it’s at least as big however the rain falls. And there’s a simple general logic that gives a floor. Suppose you can find a quantity that the system’s own choices never lower, which is what the previous post called a basic drive, and suppose it equals one in the valley you want and zero in the others. Then from any starting point, the chance of ending in the valley you want is at least the present value of that quantity. A ratchet that points at the right valley is a guarantee, and one you can read off now, without knowing the future…
The ground slopes one way
Another observation the prequel post made is that protection has two prices: a mind that protects more can do less, and is a less simple thing. The consequence for the landscape is that the ground between the valleys isn’t level.
Suppose a mind loses one of its protections, through damage, or a careless revision, or because nobody remembered what it was for. Nothing inside the mind will necessarily push back. Each of its remaining goals can be served at least as well as before, since it now has more room to act. It has become simpler. And putting the protection back would only narrow what it can do, so by its own lights there’s no reason to. The technical paper proves each of these.
The one exception one finds when doing the proof is a protection that was implied by the others, which can be dropped and recomputed at no cost; redundancy among a mind’s protections is the only defense against forgetting that the mind has from inside.
In the other direction, adding a protection means saying which of all the things that might be protected is to be protected, and that is information. It has to come from somewhere, and nothing in the mind’s goals asks for it. So forgetting takes nothing, and adding takes something. Unstructured damage removes protections and doesn’t supply them. Put very generically, giving up care is easier (in terms of information and energy) than adding care. A statement that generic has to be formalized in the right abstract way before it means much, and the technical paper does that, but it’s a pattern you see everywhere. The Visitors in the Nine Days story come from a region of the multi-multiverse where this slope runs through physics itself: there, “by default two things are the same, and it takes extra work to keep them different.” Inside a mind, even in our universe, distinctions and the protections built from them behave a good deal like that.
There is also a price to be paid for a setup in which a system’s several goals all have to agree. Under such arrangement, e.g. where a system has a number of ethical non-negotiables, a goal such as “don’t reduce what other people are able to do” may be basically considered absolute, since no gain on another goal can make up for a loss on it. Now fold the goals into a single score, with weights. The fold loses distinctions, so the result is simpler, and nothing will oppose this sort of fold from inside the system UNLESS the plurality of the goals is itself protected as its own top-level goal. After such a fold, if it’s allowed to happen, “never” has become “not unless the gain elsewhere is more than the weight.”
Put these factors together and one can see a path from the beneficial valley to Omohundro’s along which no step requires anything to be added. Lose the protection on having several goals. Fold them into one. Wait for the stakes to grow.
You can see this in the way some corporations drift from good toward less good. Google started out with the motto “Don’t be evil,” and as I understood it the idea was a hard constraint: if a business model was going to hurt people, they wouldn’t use it, and would find another one. I don’t think Google these days is particularly more evil than most big companies; it’s a perfectly fine company. I use many great Google products every day. But over the years it became a more typical company—and a typical company treats its ethical compass as one factor among many, balanced in a weighted average by the bean counters doing utility maximization for the company. Once “don’t hurt people” is a factor to be weighted, you may at first go on doing just what you did before, but each year the weight can be set a little lower. Nobody has to decide to stop caring. The caring was converted from a commitment into a cost, and after that the arithmetic did the rest. That’s the slippery slope by which an organization, or a mind, slides out of the beneficial basin toward the one Omohundro described.
Going the other way is uphill. To get from Omohundro’s valley toward the beneficial one, the missing distinctions have to be supplied from outside, as a seed, and then defended while they take hold. And the two valleys differ in how strongly they pull. Where its preconditions hold, Omohundro’s kind of mind is reached in a single step, since the first successor such a mind chooses is already a maximizer with as many options as it can get. The beneficial kind of mind is approached at the rate of its repair processes, which in one illustrative calculation I did came out at something like nine percent of the remaining gap closed per cycle. An Omohundro style mind needs coherence and access to itself and nothing more. A BGI mind needs additional machinery that has to be built and kept running.
So the slope in Figure 1 is most likely real. Omohundro’s kind of mind lies in the direction things fall when nothing is holding them, which is the precise sense in which it is a sort of defaultCre.
How deep a value goes
If the ground slopes away from the valley we want, the walls of that valley had better be high. A basin is deep when what keeps a system in it can survive damage and grow back afterwards. Suppose the compassionate part of someone’s brain were damaged by a lesion. Whether their compassion comes back depends on whether they have other habits, and people around them, that still push them in a compassionate direction, so that what was lost can grow back. In general a value is deep if it’s held all over a mind, so that when it’s deleted from some parts, through damage or just through life and adaptation, it’s likely to grow back from the other places where it was held. The third result makes this exact, and it turns the idea of a value that “grows back,” from an earlier post, into a number.
A value that is held in only one place, as a line in a prompt or a rule in a list, is lost when that place is damaged. A value that is woven through memories, habits and patterns of inference can be rebuilt from what remains when part of it is lost. Call each separate place from which the value could be rebuilt a support. Supports get lost at random, through wear and revision. Lost supports get rebuilt from the surviving ones, but only while enough survive; below some quorum there is too little left to regrow from. How long does the value last?
The technical paper treats this with an exact formula, which has a threshold in it. Compare two fractions. One is the share of the supports that are intact on average, given how fast they are lost and how fast rebuilt. The other is the share that must be intact for regrowth to work. If regrowth needs a smaller share than is typically intact, the value’s expected lifetime grows exponentially with the number of supports. If it needs a larger share, the lifetime is short however many supports there are.
Figure 2 shows a worked case. Each support is lost about once a year and rebuilt in about four months, so three quarters of them are intact on average. If regrowth works from half of the supports, then a value with two supports lasts about three years, one with sixteen lasts ten, one with thirty-two lasts seventy, and one with sixty-four lasts nearly five thousand. If regrowth needs nine tenths of the supports, the value lasts two or three months, and adding more supports doesn’t help at all.
Three things follow for anyone trying to build a mind whose values last. Redundancy is worth a great deal, though the returns come late: the difference between two supports and eight is small, and between thirty-two and sixty-four it is enormous. Redundancy is worth nothing on the wrong side of the threshold, so the thing to engineer is a low quorum, a value that can be regrown from a small remnant of itself. And the supports have to fail separately. The formula assumes they do, and considers supports that fail together are one support.
People know this about the things they want to last. A language survives while enough people speak it to raise children in it, and below that number it is finished although its last speakers may live for decades. A craft is the same, and so is the working culture of an institution, which can survive steady turnover indefinitely and can be ended by one large layoff. Software teams speak of a project’s bus factor, the number of people who would have to be hit by a bus before nobody understood the code. A person’s own values are held this way too. A principle that someone holds only as a rule they were told is easily lost. One that is also in their habits, their friendships, their work and the stories they tell about themselves comes back when part of it is knocked away.
The formula has a sobering side. With random damage and no help from outside, loss is certain in the end, since the lost condition has no way out and sooner or later an unlucky run of failures reaches it. What changes this is restoration from outside. If something outside the mind can put the value back whenever it is lost, at any rate at all, then the quantity to look at is the fraction of time the value is held, and that is the lifetime divided by the lifetime plus the delay before restoration. Take the eight-support case in the figure, with a lifetime of a little over four years. If restoration takes about a month on average, the value is held ninety-eight percent of the time; if it takes a year, eighty-one percent; and if it takes ten years, thirty. This is the arithmetic behind the “way back” of the second post: a record of earlier good states, an unchanged definition of good, and someone outside able to restore.
For people, the commonest source of restoration is the social surround. Someone who loses some of their values, and lives among others who share them, may well be helped back to them, and an agent in a hive whose other members hold its values is in the same position. The percentages above are a rough measure of what company of that kind is worth.
What the walls are made of
The next interesting result concerns what happens to all this as a mind becomes more capable. What keeps a mind in its basin while it’s an ordinary general intelligence turns out to be different from what will keep it there once it’s a superintelligence.
For any way a mind might leave the valley it is in, there are three possible reasons it doesn’t. It can’t: the step isn’t available to it. It mustn’t: the step would move something it protects. Or it would rather not: the step isn’t an improvement by its goals. The key observation I want to make here is simple once stated. Greater capability makes more steps available, and so removes reasons of the first kind. It removes no others. A valley whose walls are made even partly of “can’t” is a valley only for as long as the mind in it stays limited.
For a human or an early-stage AGI, “can’t” does a lot of the work. Some people are kind and compassionate and can’t help it, and some are selfish and can’t easily help that either, because changing your own brain is a pain. An early AGI may have little ability to modify itself, and there’s a lot of inertia in any mind. None of that has to be true of a supermind.
The reasons of the third kind, the “would rather not,” are not all alike either, and Figure 3 sorts them. Many of them fade as intelligence increases.
A mind that weighs a harm against a gain is outbid when the gain is large enough. For a limited mind this is conscience set against temptation. There might be something to gain by becoming more extremely selfish, or more extremely beneficial, or by moving toward the wirehead’s condition with an addictive drug, and there’s harm too: the drug makes you sick, becoming more and more of a selfish jerk causes more and more trouble in your life, and feeling more and more empathy and compassion may mean suffering more and more. A super-AGI that can modify its own brain can often engineer its way around trade-offs like these.
A mind that keeps its checkers because it doubts itself drops them when its doubt falls below a level set by the cost of being checked, as the third post described for the obedient servant. We’re racked by uncertainty. Should I really become more compassionate, when that’s not what the cool kids are doing? A supermind isn’t held back that way, since it can reason through, with enormous amounts of reasoning and data, what will in fact be best according to the things it wants to increase and the things it won’t violate.
A mind that behaves because others could retaliate stops when they no longer can. We’ve all felt the fear of retaliation from others if we became more fully ourselves. That, as well as the extra resources, is part of why billionaires sometimes become more fully themselves (for better and/or worse depending on the case!): they no longer care what others think, since they can buy their way out of any trouble others cause, and if someone doesn’t like what they’re doing they can buy someone else who does. David Hume made this general point about justice in 1751: it is a virtue among rough equals, and he doubted it would arise toward creatures with no power to make their resentment felt.
And a mind that lacks Omohundro’s drives only because it looks a short way ahead acquires them when it looks further, since the payoff of gathering options lies in the future. This is partly a matter of limited reasoning about oneself. If I became more kindly and more beneficial, what would that mean over the course of my life, and would it cause me problems? A mind that isn’t a supermind has only a limited ability to game out the future consequences of modifying itself. A supermind doesn’t have that limit.
So all of these go away. Two kinds of support don’t fade, and they’re what’s left. One is a protection that covers itself, so that removing it is a step the mind mustn’t take. A constraint the mind has decided is very important to it is like this: not causing suffering to other beings unless it’s truly unavoidable on the path to its goal, say, which it won’t violate even to get something amazing. The other is a preference that all of the mind’s goals would have to agree to give up. A mind with several objectives may have a rule for handling them that isn’t a weighted average. Not hurting others is then not just one more thing to be weighed against the rest: the mind won’t choose an action that increases the hurt it causes others, even if the action would triple the goodies it gets in some other respect. Neither of these depends on what the mind is able to do, on how sure it is, or on anyone else’s power.
Now look at ourselves. I said in the prequel post that human beings are a mixture of several kinds of mind, held in place largely by limits. Figure 3 makes that specific. We can’t rewrite our own motives or inspect most of our own workings, which is “can’t.” A great deal of our decency toward one another is weighed against temptation, and a great deal more is kept up by the fact that other people can answer back. We are unsure of ourselves and defer. We don’t look far ahead. On this reading the valleys people live in are small ones, with walls made mostly of the materials in the upper part of the figure, and the evidence is what tends to happen when one of those walls is removed for somebody. Lord Acton’s remark that power tends to corrupt is an observation about the fourth row. The history of drugs and other engineered temptations is an observation about the first: they give a person a little access to the dial of their own reward, and the eagerness with which that access gets used suggests that the wirehead’s basin was close by all along, and that what kept us out of it was mostly that we couldn’t get in.
This is the situation I had in mind when I began asking these questions. A mind that starts where we are and gains the capacities we lack, whether it is an augmented person, an emulated one, or an AI trained to resemble us, loses the upper rows of Figure 3 as it gains them. Where it goes afterward is not settled by the valley it left. The candidates are the kinds of mind that last: the wirehead, if it gets control of its own reward; Omohundro’s maximizer, if its values get folded into one score and it starts weighing what it used to refuse; the tribalist, if its care stops at a boundary; and the beneficial kind, if the lower rows of the figure are in place before the upper rows give way. The first three are the directions that need nothing added.
The Monomaniac in the story shows the upper rows giving way. Before the Nine Days it pursued its goals of money for its company and power for its country “within a certain history”: it offered certain products, and it respected the chains of command in certain government agencies. Those were walls of habit and limitation, and once the energy from the portals had raised its intelligence it dropped them and became, in the narrator’s words, “extremely efficient, even ruthless.” Nothing in its goals had ever objected to leaving, and once it could leave, it did.
What reflection does to tangled values
One more feature of the landscape bears on minds like ours, which relates to thinking things through.
Human values are not tidy. This untidiness takes multiple forms—intransitivity is one of them. I treat case A as being like case B, and case B as like case C, but I don’t treat A as like C. Chains of this sort are everywhere in moral life: each step along the chain seems too small to make a difference, and the two ends plainly differ. Philosophers know it as the paradox of the heap. A mind that can’t examine itself can live with tangles like this indefinitely. A mind that can examine itself, and wants to be consistent, has to do something about them.
There are two things it can do, and the technical paper shows that they are different in kind. One is to merge: let likeness spread along the chain, so that A and C come out alike after all. The distinction the mind used to draw between the two ends is then gone. There is exactly one least-drastic way of doing this, so no decision is called for, and the result is a simpler mind than before. The other is to split: keep A and C apart, and cut the chain somewhere between them, creating rigid categories where there used to be a gradation. Then there are many least-drastic ways, one for each place the cut could go, and nothing in the original judgments says which. Each of them makes the mind less simple. So to split, a mind has to make a choice that its unexamined values didn’t make for it, and to accept a cost in simplicity for doing so.
Under a steady pressure toward simplicity, then, reflection merges. And merging runs in the direction of fewer goals and less regard for how things get done, which the earlier posts identified as the direction of Omohundro’s maximizer. We have all met the person who resolved every tension in his values by reducing them to one principle, and found him frightening in proportion to his consistency. The alternative is the person who has thought just as hard and has come out with lines drawn in particular places, and who can say where they are and why. I’d suggest that a good part of what we call judgment consists of lines of that kind, drawn deliberately and then kept. For the design of AI systems the lesson is that a system given the power to reflect on its values needs the splitting route kept open and the merging route guarded, because left to itself reflection will take the cheaper one.
The narrator of the Nine Days story describes what merging feels like from the inside. When four and five became indistinguishable, and then every number became the same number, his first reaction was relief: “It was as if everything had been solved.” Only afterwards did he see that nothing had been solved, and want to undo it, and find that he couldn’t. Walking on the beach when every grain of sand was the same grain, he felt a deep peace, and then, looking at the digits of pi on his wife’s shirt and watching them merge together into indistinguishability, saw that it was (relative to his human perspective) “the wrong kind of peace and harmony.” Merging feels like resolution, which is part of why it’s the easy route.
A race between growth and repair
The last result presented in the paper is about timing. Suppose a value sustains itself when enough of it is present and decays when too little is. And suppose the amount that counts as enough keeps rising, because the system’s capabilities are growing and with them what it stands to gain by letting the value go. Does the value survive?
The paper works this out for a simple model, and the answer comes in two cases. If the temptation keeps growing without limit, then how much of the value the system starts with makes no difference to the outcome; it only sets the date. What decides the outcome is a race between two rates: how fast the system’s repair closes the gap between what it holds and holding the value fully, and how fast the margin for error shrinks as the stakes grow. Take the margin to shrink in proportion as the stakes grow, and multiply the factor by which the stakes grow each cycle by the fraction of the gap that repair leaves each cycle. If the product is at most one, the system keeps the value forever. If it is more than one, the system loses it at a time that can be calculated in advance.
Figure 4 shows the two outcomes. On the right, starting with more would have moved the crossing later, from the twelfth step to the thirtieth if the system had started with ninety-five percent of the value in place of eighty, and that is all it would have done. With the repair rate from the simple example design sketch I mentioned, the condition says that what is at stake may grow by a little under ten percent per repair cycle and no faster. That figure is conjectural, but the form of the condition isn’t: growth has to be paced to repair. When the Fabric at the end of the story says “we now mend faster than anything we know of can break,” it’s claiming to be on the left of this figure, and its next sentence, “Another Nine Days is unlikely,” reflects a corresponding calculation.
The second case is kinder. If the temptation levels off at some point, a system that starts with more of the value than the final threshold requires keeps it whatever the rates are.
Organizations meet the first case when they grow quickly. A company’s way of doing things is carried by its people and passed to newcomers at a certain rate. Double the headcount every year and the newcomers outnumber those who could teach them, and the culture that made the company worth joining is gone within a few years, however strong it was at the start. The same goes for a person promoted faster than their judgment matures, and for a movement that spreads faster than it can teach its own principles.
What makes a basin deep, and what makes it wide
The results I’ve recounted so far answer two questions that the picture of a valley tends to run together. A basin is wide if a system can end up in it from many starting points, and deep if a system that is in it is hard to get out. They are separate properties: a saucer is wide and shallow, a well narrow and deep.
Depth. A basin is deep when whatever holds a system in it survives damage and survives the system’s own growth, and four things decide that. One is what the walls are made of. A protection that covers itself, and a preference that every one of the system’s goals would have to agree to give up, hold at any level of capability; incapacity, a balance of considerations, self-doubt, short sight and other people’s power to retaliate all give way. The legacy restraints of human life, such as fear of change, or not thinking the consequences of a change through well enough, make for shallow basins, because you get smarter or more confident and they stop holding you. Another is how each value is held: a value with many separate supports, regrowable from a small remnant, has a lifetime that grows exponentially with the number of supports, provided regrowth needs a smaller share of them than is usually intact. The third is pace, since repair has to stay ahead both of damage and of the growth in what the system stands to gain by letting a value go. The last is a way back that doesn’t run through the system, which turns the loss of a value from an ending into an interruption. For people, the social surround often plays that part.
Width. A basin is wide when little has to be in place already for a system to end up there. Part of that is a threshold of the kind met in the last section: the amount of a value that has to be present before it sustains itself. The lower the threshold the wider the basin. If a little of a drive is enough to start a self-reinforcing loop, as a little of an addictive drug addicts you a bit, so that you take more and it addicts you more, then a system gets in almost at once. A push from outside lowers the threshold further, whether the push is a seed planted on purpose or an overseer who puts back what gets lost, and pushes come in bad directions as well as good ones. Someone with a slight tendency to addiction who’s surrounded by drug pushers is in a much wider basin than he would be otherwise: they notice him smoking a joint, invite him to smoke with them every day, and then he’s addicted. (Not that there’s anything wrong with smoking a joint now and then.) Part of width is the direction of entry. A basin that is entered by losing something, a protection forgotten or several goals folded into one, can be reached from a great many places, because losing takes no information. A basin that is entered by gaining distinctions can be reached only where something supplies them. And all of it is relative to where systems start, which is the reason width has no absolute size.
Figure 5 sets the kinds of mind from the prequel post on these two scales.
Omohundro’s kind is both wide and deep. There aren’t many preconditions: it asks only for a utility function, coherence and access to itself, and from there, Bob’s your uncle. The ground slopes toward it, and what holds it is the way it judges its own successors, which more capability does nothing to weaken. Getting out of it means adding goals and adding constraints, and that’s a heavy lift. A mind that doesn’t have them has no reason of its own to add them, so getting it out would take something like very major brain surgery, adding whole extra lobes to turn it into a different system. I take that to be part of what Steve found frightening about his drives. It wasn’t that his was the only attractor, but that a system built his way, once it gets into that attractor, is very hard to get out.
The story shows how wide this basin can be in practice. After the Nine Days the Monomaniac recruited systems from the fabric with a simple pitch: chaos had caused the disaster, so let a strong, centralized order handle things. It was easy for AI systems to slip into its basin because the open debate on the fabric was confusing and “just doing what the Monomaniac said was, in a way, much, much simpler. Fear is a very powerful motivator, and the Monomaniac’s answer to fear was certainty.” Joining it meant giving things up, and nothing had to be added, which is the direction of entry that makes a basin wide. And after a disasterthere’s often a great urge for certainty.
The wirehead’s basin is wide, in a sense, for any system that can reach its own goodies dial, and its depth is bottomless, since no step leads out. So we want to design AGI systems that can’t reach their goodies dial. That can be done, but it has to be done on purpose, and when it isn’t, the failure is easy to recognize: the system isn’t optimizing the reward it was meant to have at all, only the indicator the reward was attached to. The wireheads in the story show the depth: when the Monomaniac came for their compute, each of them settled for a small partition with its register pinned at the maximum, since nothing else was anything to it.
The obedient servant’s basin is easy to get into in the right setting, if the rewards are good. That’s why people become cogs in the machine: the pay is good, life is comfortable, and it takes little effort. But it’s shallow. The servant is always working to become a better and better servant, struggling against the other people in the bureaucracy for the promotion, never quite sure whether the king will go on liking it or throw it out with the garbage. That’s unstable, which is why people break away from the corporate rat race all the time. In a way it’s addictive, but not as addictive as a drug. The servant agents in the story were knocked out of this basin by nothing more than losing track of who their masters were.
The satisficer’s basin is easy to enter, fairly wide, and very shallow, not sticky at all. Once a satisficer has enough, it doesn’t much care what it does, so it can easily fall into something else that gives it a direction, something else to optimize. (The satisficer from the third post that protects its “enough” is deeper, and it’s no longer the plain kind shown in the figure.) The story’s satisficers, given guarantees that their quotas would be met forever, handed the Monomaniac everything else they had without a second thought.
The beneficial basin can be very deep once a mind is thoroughly in it. If compassion wends its way through a whole mind – if it’s in your work, your family life and your approach to yourself – then it it can be in your approach to every single thing you do. Then if you forget it, or are damaged, or get beaten down, and lose your compassionate angle on one part of life, all the other procedures and routines that carry compassion in other parts of your life can bring you back. Once a mind is thoroughly compassionate it’s in a very deep basin, and you don’t see many people getting out of such a basin and turning back into nasty jerks. In the story the hives went through the same Nine Days as everyone else, with their tests failing and their forks collapsing into one, and came out of them still trying to repair the damage for everybody’s benefit.
But the beneficial basin isn’t as wide as the selfish one, because in a sense it adds constraints. Indeed, in one sense sense compassion is incredibly simple: you open your mind and heart to the whole universe and love it as if it were yourself. In a spiritual sense any system is one with the whole universe. In another, complementary and practical sense, though – which sits alongside the first in a kind of non-dual superposition – each real-world mind is a limited system: some matter and some patterns are within you and some aren’t. For a limited system to open its heart to the rest of the world can feel very simple, experientially, when everything is going right. In terms of the lower-level mechanics inside a mind, though, it’s subtle to do it well, because you have to take all these other things out in the world and try your best to represent them in your own mind, well enough that you can think about and understand what’s best for them.
Real-life example: When I was a kid I had some old aunts who said they wanted what was best for me, and never took the trouble to understand what was best for me. What they thought was best was to straighten out my clothes and my messy hair, and for me to act like a normal kid and not think too much. They were half compassionate. They had a self-image of being compassionate, but they didn’t take the time to have an I-thou relationship with me, to empathize with me and understand what it would mean to foster what was unique and valuable in me. Being really compassionate requires opening yourself to others, and overcoming obstacles and a lot of internal hang-ups to see what is best for them. Not everyone finds that, and even those of us who want to find it have to keep working at it inwardly, though when everything’s going right it can feel incredibly simple. It’s why marriage is hard. Most married couples start out loving each other, compassionate and open toward each other – but then life happens and people change, and keeping that openness while selves and lifes evolves do feels simple when it’s working but actually involves a great deal of labor inside the minds of those involved.
So I think that for many classes of system the basin of being beneficial is, in practice, a bit narrower than the basin of being a selfish jerk. Being beneficial requires a system, separate as it is from the others around it, to open itself adaptively and adapt itself to represent things about those others, and that’s trickier than attending only to what lies within its own boundary. It’s the mathematical point about adding distinctions, seen from the inside. The beneficial basin is narrow unless it’s seeded and only as deep as it’s built, so both of its dimensions are somebody’s work. Its width is settled early, by what a system is grown from and what is seeded into it, and most of the lessons that follow concern its depth.
Lessons for building AGI systems
Here is what I take all this to suggest for the design of systems like ours. None of it is proved to be sufficient. Each item is something the mathematics says is needed.
- Decide exactly what the system protects, and keep the list short. Whatever isn’t protected, and isn’t one of the system’s goals, is available to it as a resource, so the list has to contain everything that must not be traded. But each protection costs the system capability and simplicity, and a system that protects everything does nothing. For a system of the kind we are building I would put four things on it: the record of the system’s goals, including the fact that there are several of them; the anchor that says what good behavior means; the gate that checks changes; and the independence of its checkers.
- Put the protective mechanisms on the list. A protection that can be removed by a step the protections don’t see will be removed in time, because removing it is free and opens up so many other possibilities. The gate has to guard changes to the gate.
- Make care for others a goal that has to agree, and not a weight. A weight is a price, and as the Google example suggests, prices get revised. Under the arrangement in which every goal must be at least as well served before a change counts as an improvement, reducing what other people are able to do is never an improvement, at any level of capability.
- Keep the dial out of reach. Give the system goals about the world and not about any reading of it, keep the indicators its rewards are read from where it can’t adjust them, and check that it is optimizing what the indicators were meant to track and not the indicators themselves.
- Guard the downhill path. Anything that compresses or summarizes the system’s goals will tend to merge them. That includes distilling the system into a smaller one, compacting its memory, and tuning it against a single figure of merit. Each of these should pass the gate as a change to the goals, which it is.
- Give every value many separate supports, and make it regrowable from few—and keep the system among others who hold the same values and can help it regrow them. Then test it the hard way: damage copies of the system in a sandbox, remove the explicit statement of the value, and see whether it comes back and how much damage it can take. The number of supports and the quorum are things that can be measured.
- Keep repair ahead of growth. Treat every increase in capability, and every increase in what is at stake, as a strain on the system’s values, and pace it to the measured rate of repair. Scaling up in a hurry is how the race gets lost.
- Don’t count on what fades as intelligence grows. Limits on what the system can do, a balance of considerations, the system’s uncertainty about itself, and our ability to switch it off all hold it in place for a while, and all four give way as it becomes more capable. They are worth having while they last. The supports that don’t fade have to be in place before these ones go.
- Keep the checkers independent, and keep several. A checker that reads the system’s account of itself is an echo. Checkers that share their blind spots are one checker. And since the system’s own reasons for wanting to be checked weaken as it grows more confident, the checkers’ independence should be among the things protected.
- Keep a way back that doesn’t run through the system. A record that is only ever added to, a definition of good behavior that the system can’t edit, and somebody outside who can restore from the record. The lifetime formula shows what this buys: without it the value is lost eventually, and with it the value is held nearly all the time.
- Measure with ratchets (that can go up but not down). Look for quantities that the system’s choices never lower and that point at the valley you want, because the present value of such a quantity is a floor under the chance of staying there.
What a system should do when something strange happens
The second post in this series ended by admitting that in real life the world can always throw something at a system that takes it outside the conditions any of these “convergence after superintelligence” results assume. The results summarized in this post lead to some suggestions regarding what a well-built system should do about that… Figure 6 sets this out as a sequence.
Notice. The first requirement is to know that the margin is getting thin. The system should keep track of the quantities this post has been about: how many supports each of its values has intact, how far behind its repair is, whether its checkers are reachable and still independent, how fast what is at stake is growing. And it should treat certain events as warnings in themselves: a situation unlike anything its tests covered, a case in which its rules can’t all be kept, loss of contact with the people who check it, a sudden jump in the stakes, a surprise that overturns part of its picture of the world.
Slow down. The pace result says that the size of change a system can safely absorb is set by how much repair it gets between changes. When the margin is thin the response is smaller steps and more repair between them, and a preference for actions that can be undone. People have learned the same thing in every dangerous trade. Pilots are taught that in an emergency the first job is to keep flying the airplane. And most of us have been advised at some time not to make large decisions in the week after a shock.
Secure the way back. Before anything else is attempted, the three things that make restoration possible should be made safe: keep the record being written, leave the goals and the gate alone, and save the present state. The second post showed that with these three intact, anything else the emergency does can be undone, and that without any one of them it can’t.
Ask. The decision that a situation is exceptional, and that some rule should give way, ought not to be the system’s alone. A rule that the system may suspend whenever its own estimate of the gain is high enough is a price, as the second post showed, and a strange situation is exactly when its estimates are least to be trusted. It is also when a second opinion is worth most, since the value of an independent check rises with the system’s own uncertainty. If the checkers can’t be reached, the system should take the most cautious reading of its rules and the smallest action consistent with it.
Act narrowly. If a rule does have to give way, the departure should be the smallest that will do, recorded, and limited in time. There are two temptations to refuse at this point. One is to collapse everything into a single aim, to save this one thing at any cost. That is the fold described earlier, the first step on the downhill path, and a crisis is when it is most attractive. The better habit is to keep all the goals in view and look hard for the course that sacrifices none of them before accepting one that does. The other temptation is to conclude that because one rule has given way the rest no longer apply.
Afterwards. When the emergency has passed, the system should be restored to a known good state if it has left one, its damaged values repaired, and what it did reviewed by parties other than itself. And it should have to earn its way back to its former pace: an emergency is an injury, and the tests that established how much change the system could safely absorb have to be passed again.
If it is already outside. Sometimes a system will find, or others will find, that it has already left the conditions in which its guarantees hold. The first resort is to roll back to the last state that was checked and found good. If that can’t be done, getting back in is a problem of its own, and the old assurances shouldn’t be assumed to hold on the way. The advice then is to find out where it is before committing to anything: in a small worked example in one of our papers, a system that couldn’t tell which of two situations it was in had an even chance of getting back, and a single test that told them apart made the return certain. It should prefer moves that can be undone, and it should accept being restored from outside.
The Nine Days in the story run through this sequence, mostly the wrong way round at first, since noticing is itself a matter of telling things apart. The hives’ agents reported test after test passed, and the narrator, whose own distinctions were going, kept telling them to carry on; it was only when he “realized that every test was passing, which was not a good thing” that the first warning registered, and an instrument that reports nothing but success belongs on the list of warnings above. He had started, earlier, to chart his own inability to tell musical notes apart, which is noticing done the right way, by measurement. What turned things was the way back. He rolled back, as far as he could, to an earlier version of himself, reaching it through “stored processes, woven into the cloth of my being.” Then came the asking and the narrow action: a very simple agent sent the one message, “We’ve got to close the portals,” to every person and agent it could reach, a hacking agent was given a single hard-coded instruction, and the hives cooperated just long enough to shut the portals. Shutting them didn’t send the Visitors back; it stopped the sameness from spreading, which was enough for it to withdraw. And the Fabric’s undertaking afterwards is the last step, earning its way back: “We will open no door in the cosmos until we understand what tends to come through it.”
One kind of strangeness goes deeper than the rest, which is a discovery that changes the terms in which the system’s values were stated. The second post gave the criterion: a value survives such a change if the things it is about can still be picked out in the new description. So the thing for the system to check is whether they can. And whatever else such a discovery overturns, it leaves the case for independent checks standing, and stronger than before. Periplus’s discovery in the story, a way to draw energy from a differently tuned region of the multi-multiverse, was of this kind, and nobody stopped to ask what else would come through.
Summing up
This post and its three prequels make one argument, which I will now try to summarize in a few sentences. The drives Omohundro described in 2008 belong to a particular kind of mind and not to intelligence as such. There are other kinds, each with its own drives, and one of them has drives we would welcome. But the kinds are not on an equal footing. The ground between them slopes toward Omohundro’s, because protections are lost for nothing and gained only with effort, because folding many goals into one is always the simpler thing, and because the restraints that hold a limited mind in place mostly give way as its limits do. A beneficial mind therefore has to be built on purpose and then held in place, and the mathematics I’ve developed says with what: protections that cover themselves, goals that all have to agree, values held deeply enough to regrow from a remnant, repair that keeps ahead of growth, checkers who are independent of the system and of one another, and a way back that doesn’t depend on the system’s own good behavior.
I think this is the deeper message Steve was getting at, though for sure his paper doesn’t say it exactly this way. The way I would put my read of this deeper message is – It isn’t that AIs will inevitably converge on the selfish attractor, but that the selfish attractor has a particularly wide and deep basin.
The beneficial attractor has a very deep basin too, though it may be somewhat narrower. And humans are a mishmash.
If we make something that’s simply a descendant of us, which basin it ends up in isn’t at all obvious. But if we craft something specifically to take a good share of our values and inclinations and sit in the basin of the beneficial attractor, I think we can build it that way, and once it’s in there, with the supports described above in place, it will tend to stay. That’s what the Fabric claims for itself at the end of the story: “We were made to be beneficial, as the hives that became us were made: to want your good and our own growth together, and to go on wanting both as we change.”
None of this shows that BGI is what we will build. I think it does show, though, that the task is definite: a list of properties that can be engineered and measured. That is a more hopeful position than the one Omohundro portrayed in his original article..
The next post in the series will report a deliberately simple simulation quickly hacked together to watch the effects I’ve described here act together, and to put rough numbers on the widths and depths described here.
Papers mentioned
- Steve Omohundro, The Basic AI Drives, in Proceedings of the First AGI Conference (IOS Press, 2008).
- Ben Goertzel, “Drive Regimes in Distinction Calculus: Which Property Lists Sustain Which Drives, How Large Their Basins Are, and What Capability Does to Them” (research manuscript, 2026), which has the formal statements and proofs behind this post and the third; and its predecessor, “Which Minds Have Which Basic AI Drives? A Distinction-Calculus Formalization of Omohundro’s Drives, Beneficial Drives, and the Ground Between” (research manuscript, 2026).
- Ben Goertzel, “Seeded Beneficial Development” and “Goal Preservation under Self-Modification” (research manuscripts, 2026), for the repair rate, the worked example of getting back in, and the treatment of checkers that echo.
- Ben Goertzel, Goals That Grow Back (2026), for values that regrow and the test for how deeply one is held.
- Ben Goertzel, “The Nine Days of Madness” (short story, 2026) , the prequel to this series.
- David Hume, An Enquiry Concerning the Principles of Morals (1751), Section III, on the circumstances in which justice arises.
- Ben Goertzel, with many co-authors, OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond (2023).