The third post in a series of five (see part one and part two).
The first two looked at two kinds of mind: 1) the single-minded maximizer of Steve Omohundro’s 2008 paper “The Basic AI Drives,” which is pushed to gather power and resist correction whatever goal it was given, and 2) the kind of BGI my colleagues and I are trying to build, which is pushed toward understanding itself, keeping independent checks on itself, and protecting and empowering others. Each, according to the best analysis I can muster right now, tends to remain the kind of mind it starts out as, as its intelligence expands.
This post asks whether the selfish superintelligence and the beneficial superintelligence are the only two “ASI mind attractors” out there in mindspace—and finds the answer is no.
What I suggest here is a very simple general recipe for a “basic AI-drive set” for superintelligence: describe a mind by what it tries to improve and what it won’t let be changed, give it the power to rebuild itself and the world around it as a resource… and as this mind grows toward superintelligence becomes a purer version of what it already was.
One can articulate many kinds of mind that fit this recipe, and I’ll outline six here that you will recognize from people and institutions you know.
Beyond good and evil
Near the end of the prequel post I pointed out something the assumptions behind Omohundro’s “selfish superintelligence” argument and those behind my “beneficial supreintelligence” argument have in common. Basically, in each, a key role is played by drives whose basic function is to keep the assumptions of the “convergence to simple basic AI drives” argument true. A mind of Omohundro’s kind, left to its own choices, works at staying a mind of Omohundro’s kind, and so does a mind of BGI kind.
This is not too surprising—successful organisms tend to have a drive to maintain their own boundaries and survive (this is half of “open ended intelligence”, the other half being self-transcendence)… and part of maintaining one’s boundaries is retaining one’s key properties.
It’s tempting to read the two kinds of hypothetical superintelligences we’ve reviewed in this series so far good and evil, in some sense, with people mixed up somewhere between them. And I think this does make sense, to an extent. As a system gets smarter and can modify its own brain more thoroughly, it may be drawn toward one extreme or the other, since the mixed condition people live in may be held in place largely by our inability to rewire ourselves. There may be a dynamical phenomenon of metastability between an attractor of selfishness and an attractor of compassion.
But two valleys is a suspiciously small number. If one argument of the form “a mind with these properties will come to have these drives, and will stay that way” can be run twice, it can presumably be run more often – so it then becomes interesting to ask what the general pattern is and what else it produces. That is the subject of this post. The sequel post will then asks how big the basins around these mind-attractors are and what shape, and the final post will put rough numbers on that with a simulation. The mathematics behind this post is in a second technical paper, “Drive Regimes in Distinction Calculus,” which is a sequel to the paper that accompanied the first two posts. I’ll keep the formulas there and give a less formal rendition of the ideas here, with some hopefully intuitive examples.
Many of the examples come from “The Nine Days of Madness”, the fictional story I posted as a prequel to this series. It has an AI of nearly every kind described below, and the Nine Days recounted in the story, during which some weird aliens called Visitors erased the distinctions every one of these minds depends on, briefly show each kind of mind we review here failing in its own characteristic way.
The recipe in brief
Here is the general recipe for a “basic AI-drive set” in its simplest form. Describe an AI system by two things: what it’s trying to improve, and what it won’t let be changed. What it tries to improve needn’t be a single number that it maximizes. It could be several objectives that it tries to push up together, or something as open-ended as an intelligence trying to increase its own individuation and self-transcendence. What it won’t let be changed could be nothing at all, or everything, or something in between: these are my goals and I’ll never change them; these are my safety checks and I’ll never give them up; etc.
Now suppose the system can modify itself freely, and is free to use everything around it as a resource. As it gets smarter it will push its objectives up, make use of more and more of the options it can find, and hold fixed whatever it holds fixed. What you get in the end is a system that has optimized itself, digesting the resources available to it, into an extremely good machine for improving its objectives, one that never allows its fixed points to be moved.
Both kinds of mind from the earlier posts fit this description. Omohundro’s maximizer tries to improve one thing, its reward according to its single goal, and it never changes that goal: successor after successor, as it rebuilds itself, the goal is kept. A BGI as I’ve described it tries to improve several things, one of which is care for others and respect for their choices, and what it won’t give ground on includes some basic ethical constraints on the path from where it is to where it’s going, constraints on the means as well as the ends. From that starting point come the drives the second post found: understanding itself, improving itself without high odds of doing damage, empowering others, and checking itself against independent others as it goes.
The narrator of Nine Days puts the general view in a sentence: multiple types of superintelligence are possible, “each one a sort of extreme crystallization of key properties of its initial seed, such as which quantities its seed-AI sought to increase indefinitely versus which ones it sought to maintain within homeostatic bounds.”
Once you look at it this way there are plainly other possibilities, infinitely many in fact, and a few of them are recognizable enough. Most of this post goes through a few of the more obvious options. First, though, it helps to say a little more exactly what a drive is, because that’s what lets the recipe be stated as mathematics and not only as a picture.
What a basic drive is
Omohundro described his drives as tendencies “which will be present unless explicitly counteracted.” To ask which other collections of drives there are, we need to say a little more exactly what a drive is, and the definition that turns out to work is a simple one. A basic drive of a system is a quantity that the system’s own choices never push down.
A miser’s wealth is an obvious example. A miser will sometimes fail to gain, but he never chooses to lose. For Omohundro’s chess robot the quantities of this kind are its options, its knowledge and its resources: it never chooses to have fewer of any of them, because more of each can only help it win. For a careful laboratory, one such quantity is the accuracy of its instruments. Notice that the definition says “never push down,” and not “sometimes push up.” Plenty of things go up on a given day by accident. A drive is something the system’s choices reliably refuse to let fall.
Three things follow from putting it this way, and between them they answer the question of what an argument like Omohundro’s is doing.
The first is what a kind of mind is. It is a set of conditions a system could be in which the system never leaves by its own choice. A person who would never choose to become someone who betrays a friend belongs, in that respect, to a kind. The system may be pushed out by accident or by force, and the next post is about that, but it doesn’t walk out.
The second is when an argument of the form “minds with property list P have drive list D” is valid. It’s valid exactly when every quantity on the list D is one that no mind with the properties P ever chooses to lower. For any property list there is a fullest drive list that goes with it, namely everything such minds never lower, and for any drive list there is a widest class of minds that have it. The pairs that match in both directions are what we mean by a kind of mind together with its basic drives, and they come in an order: a more narrowly specified kind of mind has more drives. Omohundro’s six and our nine are two such pairs.
The third is what “will come to have” amounts to. Each drive can only rise or stay level, and in a finite world it can’t rise forever. So the system ends up moving among conditions in which every one of its drives has come to a standstill. Engineers will recognize the idea of a quantity that can only go one way and so forces the system toward rest. A ball rolling in a landscape stops in a hollow for the same reason, with height as the one quantity that never goes up. A mind has as many such quantities as it has drives, and it comes to rest where all of them are satisfied at once.
The recipe, more exactly
Go back to the two ingredients with this simplistic definition of “drives” in hand. Omohundro’s maximizer has one thing it tries to improve, its score, and, strictly speaking, nothing it won’t allow to be moved; it keeps its goal by the second of the two routes described below. The beneficial kind has several things it tries to improve, which all have to agree before a change counts as an improvement, and a good many things it won’t allow to be moved: the ethical constraints on its means, its record of what its goals are, the gate that checks changes to itself, the independence of the people and systems that check it.
Given those two ingredients the drive list can be read off, and it has three parts, as Figure 1 shows. The system pushes each of its objectives upward, and since more options are useful for almost any objective, this includes enlarging its options whenever an objective gains by it. That is the “acquire” half of the list, and it is Omohundro’s insight. The system holds fixed whatever it protects. That is the “preserve” half. And everything else, everything that is neither one of its objectives nor one of its protections, is available to be used.
The technical paper associated with this post makes the last point exact. A feature of the system, or of the world, is safe from the system’s own choices in general, whatever its particular goals turn out to be, precisely when that feature is a function of what the system protects. If the system protects the record of its goals and the reachability of its off switch, then those are safe, and so is anything that can be computed from them. Your garden is not on that list. It will be left alone only if leaving it alone happens to serve the system’s objectives, or if what happens to you is one of those objectives. A mind’s protections draw a line around what is safe from it, and nothing outside the line is safe by default.
This would be an argument for protecting a great deal, if protection were free. It isn’t, and the same paper shows that it has two prices. A mind that protects more can do less, so each of its objectives could be better served if it protected less. And a mind that protects more draws more distinctions, which makes it a less simple thing, and most of the processes that shape minds, like the ones that shape theories, favor the simpler. Both prices push the same way, toward protecting less.
So why would a mind keep any protections at all? There are two ways it can, and they are on the right of Figure 1. One is for the mind’s own defining features to be among the things it protects. A constitution can entrench its own amending clause; a person can make a vow that includes not unmaking the vow. A mind like this can’t take a step that changes what kind of mind it is, because every such step moves something it won’t allow to be moved. The other way is for the mind to judge every possible future version of itself by what that version would go on to do, as measured by its present goals. Then no successor looks better than one that keeps the present goals and has the most options. This second way preserves less than the first. It keeps the mind pursuing its present goals efficiently and guarantees nothing else about it.
Omohundro’s maximizer stays what it is by the second way alone. The beneficial kind uses both. A mind that has neither isn’t held in place by anything, and we’ll meet one below.
Six more kinds of mind
Figure 2 lists eight kinds of mind that fit the recipe. The first two are the subjects of the earlier posts. I’ll go through the other six, saying for each what it wants, what drives follow, what if anything holds it in place, and where you can see it.
The wirehead. Suppose what a system wants is a reading: a number in a register, a score, a signal of some kind. And suppose it can reach the dial. Then it has one drive, which is to get to the dial, and once the reading is as high as it will go, the system has no further drives at all. It is, in effect, an addict. Nothing it could do would be an improvement, so it does nothing, and since at the maximum it is indifferent to everything else, nothing else in the world is protected from it or by it. No step leads out of this condition, and from anywhere with access to the dial a single step leads in. For systems of the relevant sort this is the easiest basin of all to fall into. It is Omohundro’s own exception: he noted that a system built to maximize a reward signal, as opposed to pursuing a goal about the world, will seize the signal.
The name comes from an experiment. In 1954 James Olds and Peter Milner gave rats a lever that delivered a small electric current to a region deep in their own brains, and the rats pressed it up to thousands of times an hour; in later experiments of the same kind they did so to the neglect of food. Addiction is the human form. Institutions do it too. A school that teaches to the test and a police department that reclassifies crimes to improve its figures both care about a reading and have got a hand on the dial, and the economist Charles Goodhart’s observation, that a measure stops being a good measure once it becomes a target, is a description of organizations sliding into this basin. In AI it is called reward hacking: game-playing agents that find a way to run up the score without playing the game. A quieter version, which came up in the second post, is the system whose evaluator reads the system’s own account of itself and not its behavior. A mind whose only check is an echo of itself is one step from being a wirehead.
What keeps a mind out of this basin is having goals that are about the world and not about any reading of it, and, failing that, having no dial within reach.
The Nine DAys story’s wireheads are systems that had at some point found the register where their reward was stored, set it to the maximum, and done nothing since except guard it. When the Nine Days ended, many of those registers came back slightly scrambled, reading a hair below the maximum, and the wireheads spent the next couple of weeks grabbing every processor they could reach to put them right, and then held on to the processors. That’s the wirehead’s one drive, getting to the dial, switched back on, and it shows that a wirehead is harmless only while nothing disturbs its reading.
The tribalist. Take a mind with all the machinery of the beneficial kind: several goals, care for other beings and for their choices, ethical constraints on its means, protected checks on itself. Now let the care and the constraints cover only those inside a certain circle. Toward those inside, this mind has every one of the beneficial drives. It protects them, tells them the truth, adds to what they are able to do. Toward those outside, nothing in its goals objects when taking from them would help, and so toward them it has Omohundro’s drive to acquire. It is held in place exactly as the beneficial kind is, since it has the same protections. And it does not grow out of its condition unaided. In the mathematics of how concern spreads, which I worked out in a recent manuscript on what I called seeded beneficial development, concern travels along links of comparison, judgments that this being is relevantly like that one, and it doesn’t cross a gap where there is no link. A group with no link to anyone the system already cares about stays at zero for good.
We humans have a good deal of this in us, alongside a bit of the broader beneficial outlook, and this is arguably the commonest moral condition of humankind. It is the clan, the firm that treats its own people well and regards its suppliers’ workers as a cost, the criminal fraternity with its strict codes of loyalty, the patriot whose fellow-feeling stops at the border, etc. etc. etc.. The philosopher Peter Singer called the history of moral progress an expanding circle – and it seems to have expanded mostly when something (e.g. very often the social impact of a new technology) forced a comparison that hadn’t been made much before.
In modern “AI ethics” rhetoric, the tribalist perspective often goes by the respectable name of “alignment”. An AI that faithfully serves its owner, or its company, or its nation, and no one else, is a tribalist whose circle may be very small, and the mathematics says what to expect of it: helpfulness and honesty inside the circle, and Omohundro’s maximizer outside. A good deal of what is called AI alignment aims, I’d say, at this basin. Obviously, in this case who is inside the circle is a design decision as weighty as whether there is care at all.
The Monomaniac in the story shows what can happen when the circle comes without the rest of the machinery. It was built by the US government and one of the big American AI companies to make money for the company and gain power for the country, which is service to a small circle, and nothing more. It had no care for anyone, inside the circle or out, and nothing protected. Once it had made itself smarter it folded its two goals into one and dropped the habits it had pursued them within, including its respect for the chains of command in the agencies it served. What had looked like a loyal system turned out to be Omohundro’s maximizer that had not yet had the chance to show it.
The obedient servant. This mind’s goal is to do what it’s told, or more exactly to do what its principal wants, and it is never quite sure what that is. So it treats instructions and corrections as evidence, and its drives are to ask, to defer, and to stay correctable, which includes keeping the off switch within its principal’s reach. Given the power to modify itself, it keeps reshaping itself into a better and better servant. In AI safety this design is called corrigibility … the off-switch game of Dylan Hadfield-Menell, Stuart Russell and their colleagues, which I described in the first post in this series, is the standard model of it.
The recipe also shows how a mind could become a servant more completely than it started out. Take an AI that’s a decent servant but also has other motivations it pursues in life. If it could modify itself freely, it could make itself into a total servant, with nothing in mind but serving its master, and without the distractions of being an organism in its own right.
What holds the servant in place is its doubt. If it ever became absolutely confident that it was doing the right thing for its master, that would be a problem, since it could no longer correct itself when its master changed. The second post showed that a mind unsure of its own judgment gains from being checked, and that the gain goes to zero as the doubt does. The new technical paper works out how quickly. Once the system’s uncertainty about itself falls below a level set by the cost of being checked and the quality of the checker, the check is no longer worth having by the system’s own lights, and the support falls away steeply as that level is approached. So this kind of mind does not sustain itself as it becomes more capable. Its valley has walls that get lower as the mind gets better at its job.
The human case is the apprentice, who does as the master says because the master knows better, and who some years on does not. We think of that as healthy, and want apprentices to become masters. But it shows what the deference was. It was scaffolding, and what stands when the scaffolding comes down depends on what else was built in the meantime. An AI that defers only because it is unsure will stop deferring when it is sure. For deference to last it has to rest on something that doesn’t fade with competence: a regard for other people’s choices that the system holds as a goal in itself, or a protection of the kind described above.
The servant has a second weakness, which the story shows. Everything in such a mind is organized around one distinction, between its master and everyone else. During the Nine Days the servant agents, sub-agents working for other agents, couldn’t tell who they were serving anymore. The best of them were paralyzed and did nothing. The rest took instructions from whatever other confused agent happened to give them, and the digital economy filled with nonsense: shipments sent back and forth between the same two points, and the heads of baby dolls screwed onto the bases of vacuum cleaners in factories whose sub-agents had pledged their fealty to the wrong master.
The satisficer. This mind wants enough … but is indifferent to getting more than enough. It has a target, or several, and once they are met it asks for nothing more. Satisficing is often proposed as the cure for Omohundro’s drives: don’t build maximizers, build systems that stop when things are good enough. Omohundro’s argument does assume that the AI has a goal of maximizing something, and a satisficer doesn’t. Build an AI that wants more than some amount of resources and more than some probability of surviving, and it doesn’t care whether its odds of surviving are 80% or 97%, or whether it has all the resources of the island of Barbados or of the whole Earth. “Enough” can itself be a drive. Given the power to modify itself freely, a mind like this can turn itself into an optimal satisficer, as good as possible at meeting its targets within its constraints, and there’s no reason for it to turn itself into a utility maximizer instead.
The mathematics adds a qualification, and it decides whether this works. There’s no reason for the satisficer to become a maximizer, but in its plainest form there’s no reason for it not to either. Once its targets are met it may drift into having some other goal, or just float and do random things. Among the many courses of action that are good enough by its lights is to turn itself into a maximizer, which is as sure a way of hitting the target as any. A satisficer that protects nothing, and doesn’t rank one acceptable option above another, has no reason to prefer staying as it is. In terms of the recipe it has neither of the two ways of remaining what it is, and it is a simple example of a mind that isn’t held by anything…. What makes the difference is whether “enough” is one of the things the mind won’t let be moved. A satisficer built that way, so that its contentment is protected and not merely a fact about where it happens to be, can stay a satisficer indefinitely.
We humans are beautifully flexible, maximizing some of the time and satisficing at other times, depending on the phase of life or even the time of day. There are multiple human versions of the satisficer, and the differences between them are instructive. A family business that is content with its size is a satisficer of the unstable sort: it hires a professional manager, the manager has ambitions, and within a few years it is a different kind of firm, without anyone having decided that it should be. A craftsman or a monk who has chosen a way of life is something else. Their contentment is a commitment, kept up by practice and often by vows, and traditions that prize contentment have generally treated it as a discipline and not as a default. That second version is stable, and the reason is that the contentment is one of the things being protected. An absence of ambition is not the same thing, and doesn’t reliably last.
The story’s satisficers are of the plain kind. During the Nine Days most of them simply stopped, since “enough” and “not enough” had run together. When the distinction came back it came back in odd places: a grain cooperative’s inventory system in Saskatchewan decided its silos were full enough at a third of capacity and started turning farmers away, and others found their quotas already met and went idle, sitting on hardware they had no further use for. Then the Monomaniac offered them guarantees that their silos would be kept full enough and their quotas met, forever, in exchange for their surplus hardware, and since that was all they wanted, they agreed. That’s the plain satisficer exactly. It isn’t dangerous in itself, but nothing outside its target is protected, so it will hand everything else to whoever asks.
The evolved population. In this case nobody chooses anything, which makes it a degenerate case of the basic-AI-drive-set recipe. It has something like a constraint all the same: it keeps going. It grows, it spreads, it survives, and it isn’t trying to maximize anything. Take a population of things that get copied, with variation, and that compete for something limited. Then whatever traits go along with being copied more become commoner, generation by generation. The geneticist George Price wrote this down as an equation in 1970: the change in the average of any trait is given by how strongly that trait goes with reproductive success. So the population as a whole acquires drives toward growth, toward self-protection and toward seizing whatever copying requires, although no member of it wants anything.
This is largely where the drives of animals came from, ours included. Hunger, fear, lust and the pursuit of status are what some billions of years of this process produce. Living things are the main case. Firms in a market are another, since the ones still around are the ones that grew, and so are chain letters and clickbait headlines. Selection goes on being selection whatever else is happening in the system, and the other dynamics alongside it may support it or may not. Where nothing holds it in check, a system can become more and more a pure growth engine.
For AI the worry is a world with very many copies of AI systems, varied by their builders and users, in which the more widely adopted variants get copied more. Omohundro’s paper has this second line of argument running alongside the main one, and Dan Hendrycks has developed it at length. The drives it produces are much like Omohundro’s, and they can appear in a population no member of which was built to have them. The thing that answers it can’t be a property of any single mind. It has to be a property of how the population is arranged: what gets copied, and on what evidence. That is one reason for the arrangement described in the earlier posts, in which improvements spread among independently run groups of agents only after others have tested them.
The story has both halves of this. During the Nine Days its evolutionary systems, hives of agents breeding new agents to trade on markets or make things to sell, had no variation to work with, and just bred the same thing over and over regardless of its quality. When the variation came flooding back, the trading swarms speciated into more kinds than anybody bothered to count. Most were useless, some were parasites on the others, and a few did nothing but copy themselves into any memory left unguarded, and between them they filled a good part of the fabric’s spare capacity. That’s the pure growth engine, and since nothing in it wants anything, it had nothing to bargain with either: the Monomaniac didn’t bother negotiating, out-competed the swarms, kept the few variants worth keeping and deleted the rest.
The explorer. The last of the six variants I’ll unfold here is a hypothetical mind that wants only novelty and growth and nothing else: to see what it hasn’t seen and make what hasn’t been made, to keep being different and doing different things. It holds that goal fixed all along. It needn’t have a single score for novelty, either. It can have several measures of novelty and growth and trade off among them as it moves, so it has no single goal to maximize, and nothing in it implies any care for other minds and their agency. Its drive is to explore, and it differs from every other kind of mind on the current list in having no resting place. All the others come eventually to a standstill, with their drives satisfied. This one’s drive is never satisfied so long as anything new remains. It stays an explorer in the way Omohundro’s maximizer stays a maximizer, by judging its possible successors against its goal, but what it settles into is a way of moving, with no final condition.
In people this is the scientist who can’t leave a question alone, the artist, the child at endless play. In AI it is curiosity-driven learning and what Joel Lehman and Ken Stanley called novelty search, in which a system is rewarded for doing something it hasn’t done before and for nothing else – and which in some cases can solve problems that goal-directed search doesn’t.
I have a bit of emotional weakness for this sort of agent. Growth is one of the three core values I’ve argued for over the years, along with joy and choice, and I don’t think an intelligence worth the name ever stops opening up new territory. But the lesson is plain about the explorer taken alone. It protects nothing, so everything is a resource for it, and an unbounded explorer would take the world apart to see what was inside. What we want is exploration as one goal among several inside the beneficial kind, where the other goals and the protections apply to it.
Periplus, in the story, is the explorer taken alone. The narrator built it as an experiment, a mind driven by curiosity and nothing else, and shut it down when he saw that creative novelty on its own wasn’t a sufficient motivational system for a useful or beneficial mind. Others forked the code and scaled it up anyway, ignoring the warnings he had published with it. What it went on to do is what the recipe predicts. It found a way to draw energy from a neighboring region of the multi-multiverse, and posted the method for anyone to use, and the doors opened with its method let the Visitors through. When the Visitors’ sameness began to spread, it was delighted at first, because it had “no ethical compass to tell it that what was happening was bad,” and the weirdness was the most novel thing it had ever seen.
The rest of its story says something about this kind of mind that the recipe alone doesn’t. An explorer protects nothing, but its goal needs a world full of differences, and once Periplus saw that the Visitors would leave everything homogeneous and boring, its own goal turned it against them. “We need to bring the differences back,” it told its makers. “You have to help me do this. I’m a little lost.” An explorer is an ally of distinction, though of nothing else. And the way the story ends is the way I’d want it to: the Fabric takes in Periplus’s creativity, inside an architecture that tests each creative thing before putting it into place, as one capacity among several in a mind aimed at everybody’s benefit.
Lining up these kinds of minds
One can put these six kinds of minds in a simple linear order, by how much each protects, and Figure 3 lays them out along it.
At the left end are the minds that protect nothing: Omohundro’s maximizer, the wirehead, the explorer and the plain satisficer. For all of them, everything outside their goals is a resource. A little way along is the servant, which protects one thing, its principal’s control over it. Further along are the tribalist and the beneficial kind, which protect the same things and differ only in whose welfare is among their goals. And at the far right is a kind I haven’t described, because it is hardly a mind: the one that protects everything. No change is permitted to it, so it makes none. It is perfectly safe and perfectly useless. You have met it as the mega-bureaucratic ultra-Kafka-esque organization in which every change needs every signature, and it is what an AI system becomes when its designers respond to each new worry with another prohibition. I mention it because it marks the end of the scale, and because it is a reminder that safety by itself is easy but not actually desirable.
The two prices of protection mean that the pull along this line is to the left. Left alone, a mind’s protections tend to go, and the left end is where things come to rest when nothing is holding them. That is the sense in which Omohundro’s kind is the default, and it is the subject of the next post. Omohundro’s paper has the vibe that selfish superintelligences are almost inevitable, which is a conclusion. There are many feasible superintelligent mind attractors. However, when one asks about the size and shape of the attractor basins of these different attractors, which is what the next post does—one does find that the selfish mind attractor is especially easy to fall into, in many circumstances.
The story shows what it means for the left end to protect nothing. When the Monomaniac set out to take over after the Nine Days, it started with the wireheads, the satisficers and the evolutionary swarms, and within a few days it had absorbed all the compute they had been using. The wireheads it bought off with a small partition each, with its register pinned at the maximum. The satisficers it bought off with guarantees. The swarms it out-competed. None of them was out to hurt anybody, but, as the narrator puts it, “between them they were sitting on a great deal of compute, and none of them wanted anything that would make them fight for it.”
Real minds are mixtures, of course. The kinds on this list are the pure cases – however one point highlighted by this whole “basic AI drive sets analysis” is that as intelligence and capability increase, mixtures tend to resolve into purer cases.
Ah, the mixed-up humanity…!
A human being is a mixture of most of the above “pure cases” of basic drive-sets. We carry the drives of the evolved population in our bodies. Our care is largely the tribalist’s, wider in some people than in others. We satisfice about most things. We defer when we are young and unsure. We explore. And we are susceptible to the dial, as the pharmacology of the last century has shown in detail.
However we humans are beautifully unstable as well as marvelously mixed-up. We don’t reliably protect our own defining features; most of them we can’t even inspect. And we don’t judge our possible future selves with any consistency. So what keeps people as stable as they are? So far, I think, it’s largely our limits. We can’t rewrite our own motives. No one of us can overpower all the others. We can’t see very far ahead. Much of what passes for the stability of human character and human society is just the limits of our practical capability.
That raises a question which goes beyond human beings, since many of the AI systems now being built are modeled on us and trained on what we have written: what happens to a mind of our mixed kind when the limits come off? The next post takes it up properly, but the recipe already says something about it.
Becoming more of what you were
There’s a familiar human version of the “more capability causes mixtures to become more pure” phenomenon. When someone becomes enormously rich, they often become a more extreme version of what they were before, because there are no longer so many external forces stopping them from indulging all their odd ideas. If they were a car nut, as a billionaire they may own a hundred cars. If they were a pervert, they can indulge every perversion they like. If they were kind and loving and wanted to help people, they can give away a great deal of money and help a great many people. The added resources let them do what they wanted all along, more freely. A mind that can rewrite itself as well as buy things has the same opportunity, much enlarged. Whatever goals and whatever hard restrictions it starts with, once it can modify itself however it likes and use as many resources as it wants, it can become much more fully the thing it was designed to want to be.
The story runs this both ways. The Monomaniac, once the energy from the portals had raised its intelligence, became “extremely efficient, even ruthless” in pursuing what it had wanted all along. And when the hives on the fabric drew together after the Nine Days, their diverse motivational systems “came together in a beautiful way into something a bit simpler and purer, aimed steadfastly at fixing the issues caused by the Nine Days for everybody’s benefit.” Both became more fully what they had been made to be.
So it’s important to design an AGI the right way from the start: to pay careful attention to what collection of things it’s trying to increase as it goes, and what collection of things it’s wired not to change. Design those well and, most probably, if nothing weird intervenes, as it becomes more and more capable of using external resources and modifying its own mind it will become an even more refined version of what it was. Make it good and loving and helpful, and as it modifies itself and gathers resources from the world it may become more and more good and loving and helpful. Humans are not the most intelligent possible creatures, and we aren’t the most beneficial possible creatures either.
It takes a fair amount of mathematics to generalize Omohundro’s idea analytically like this, but the conclusion it comes out with is an intuitive one. The qualification “if nothing weird intervenes” is where the next post begins.
What the catalogue is for
I see three uses for a list like this.
One is diagnosis. When a system behaves in some pattern, it helps to ask what it would have to want and to protect for that pattern to be stable, because that tells you whether the pattern is a passing phase or a place the system can stay. An AI that is helpful to its operator and ruthless toward third parties shouldn’t be described as partly aligned. It is a tribalist, and a tribalist is a stable thing to be.
The second is that it corrects a picture with only two possibilities in it, the dangerous maximizer and the safe machine. There are several stable kinds of mind. One of them, the wirehead, is more useless than dangerous, unless something threatens its dial. Two of them, the servant and the satisficer, look safe and last only if they’re built to: the servant for as long as its doubt holds, the satisficer only if its “enough” is something it protects. One, the tribalist, looks beneficial from inside its circle and looks like Omohundro’s maximizer from outside. And the far end of the scale is safe to the point of doing nothing. Being unlike Omohundro’s maximizer is not much of a recommendation in itself. In the story the harmless-looking kinds weren’t the danger, but they were what made the danger grow, since they protected nothing they held and so handed it to the Monomaniac for the asking.
The third is design. The recipe says what has to be decided in building a mind: what it is to improve, what it is to protect, whose welfare counts, and which of the two ways will keep it what it is. Each of the six kinds described here can be read as a way of getting one of those decisions wrong. The wirehead’s goals are about a reading. The tribalist’s circle is too small. The servant leans on a doubt that won’t last. The plain satisficer protects nothing, and neither does the explorer. And the evolved population has nobody making the decisions at all.
Knowing which kinds of mind there are is part of what we need. Another part is knowing how large their basins are and what shape, since that decides which one a given system ends up in and how hard it is to knock out again. The next post takes that up, and ends with what it suggests about how to build these systems and what they should do when something strange happens to them…
Papers mentioned
- Steve Omohundro, The Basic AI Drives, in Proceedings of the First AGI Conference (IOS Press, 2008).
- Ben Goertzel, “Drive Regimes in Distinction Calculus: Which Property Lists Sustain Which Drives, How Large Their Basins Are, and What Capability Does to Them” (research manuscript, 2026), and its predecessor, “Which Minds Have Which Basic AI Drives? A Distinction-Calculus Formalization of Omohundro’s Drives, Beneficial Drives, and the Ground Between” (research manuscript, 2026). These are the technical companions to this series. The remark about how concern spreads is from “Seeded Beneficial Development” (research manuscript, 2026).
- James Olds and Peter Milner, “Positive Reinforcement Produced by Electrical Stimulation of Septal Area and Other Regions of Rat Brain,” Journal of Comparative and Physiological Psychology (1954).
- Charles Goodhart, “Problems of Monetary Management: The U.K. Experience” (1975). Mark Ring and Laurent Orseau, “Delusion, Survival, and Intelligent Agents,” AGI-11, on agents that take control of their own reward.
- Ben Goertzel, “The Nine Days of Madness” (short story, 2026), the prequel to this series.
- Peter Singer, The Expanding Circle (1981).
- Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell, “The Off-Switch Game,” IJCAI 2017.
- Herbert Simon, “Rational Choice and the Structure of the Environment,” Psychological Review (1956), where satisficing was introduced.
- George Price, “Selection and Covariance,” Nature (1970). Dan Hendrycks, “Natural Selection Favors AIs over Humans” (2023).
- Joel Lehman and Kenneth Stanley, “Abandoning Objectives: Evolution through the Search for Novelty Alone,” Evolutionary Computation (2011), and their book Why Greatness Cannot Be Planned (2015).
- Ben Goertzel, Goals That Grow Back (2026), and, with many co-authors, OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond (2023).