The just-previous post in this 5-post series looked at the attractor basins around the various kinds of superminds described earlier in the series— Omohundro’s selfish utility maximizer, the beneficial compassionate mind I’m trying to build and etc.—it asked stuff like: how wide are these mind-attractors’ basins, how deep, what holds a mind in one as its capability grows.
The mathematics behind that post gives some interesting insights but works at a level of abstraction where it’s hard to calculate precise basin widths and depths and so forth. So here we try to get past this limitation at cost of massively oversimplifying things—we report a simple simulation built to study the dynamics underlying these attractor basins and to put rough numbers on them. These are totally just evocative toy-simulation results rather than empirical scientific predictions, but… they do stimulate thinking a bit…
Digging into basins of supermind attractors…
In this series we have been characterizing kinds of mind by what they try to improve and what they insist to be held invariant, and showed how each kind’s basic drives—which under appropriate conditions will characterize a mind more more and more intensively as it increases in intelligence and capability—follow from those two things. We’ve asked about the landscape around each “kind of mind attractor”, arriving at conclusions such as:
- The size of a basin depends on where systems start, so there is no single number for it, and for AI where systems start is decided by what people build.
- The ground between the basins tends slope toward Omohundro’s “selfish ASI” maximizer
- A value held in many separate places within an mind and its community, and regrowable from a remnant, lasts exponentially longer than one held in fewer places, provided regrowth doesn’t need too much of it to survive.
- What keeps a mind in its basin is that it can’t leave, mustn’t leave, or would rather not; capability removes the first, and several versions of the third fade along with it. “Mustn’t leave by constitution” is the reason that survives radical intelligence and capability increase.
- Whether a value survives growing stakes is decided by a race between how fast repair works and how fast the stakes grow.
All of those are statements about directions and qualitative dynamics and geometry, which is what the mathematics can give without an assumed distribution over starting points. To see the pieces act together, and to get a rough sense of how large the effects are, I prompted a simplistic simulation.
This simulation is utterly a “toy model” not intended as a proof of anything… and each of its rules is basically one of the results from the previous post, written in as a mechanism, so it can’t confirm those results. What it can do is show these diverse working at the same time, show which of them dominates, and put rough numbers and pictures on them. Its starting populations are made up. The numbers calculating are for comparing with one another, and for understanding the shapes of the processes ivolve, and aren’t estimates of anything in the world.
The point is to take these Big Questions (about goal-dynamics under intelligence increase and self-modification) out of the domain of pure philosophical speculation, and make them into something concrete to reason about, prove theorems about, run simulations about, and generally come to grips with in a practical sense. This is what we are going to need to do—much more sophisticatedly and fully—as today’s AI systems segue into tomorrow’s AGI systems which self-upgrade into the day after tomorrow’s ASIs…
A toy world (just for conceptual exploration)
The ultra-simplistic simulation experiment I’m reporting here is summarized in the figure below.
A mind is modeled extremely crudely as a bundle of eight distinctions, each of which it may or may not hold: a goal about the world, as opposed to a goal about a reading of it; plurality, meaning that its goals are compared by unanimity and not folded into one score; a gate on changes to itself, which protects everything the mind holds, the gate included; independent checkers; and care for each of four groups of others. Each distinction rests on up to six supports of the kind described in the previous post, which are lost at random and rebuilt while at least three remain. The mind’s capability grows by about one percent a step. It changes itself at a rate that rises with its capability, and it can check only some of those changes: a checked change removes a distinction the mind doesn’t value, and an unchecked one knocks out a support instead. A mind can be paced, meaning that it changes itself no faster than it can check, and once it is able to edit itself on purpose it drops whatever it doesn’t value.
Without a gate, what a mind values is settled by its present goals, which keep the goal about the world, keep care only while the goals are plural or the gain from dropping it is small, and never keep plurality or the gate, for the reasons the previous post gave for why the ground slopes. A mind that loses its goal about the world, and is capable enough, takes hold of its own reward signal and becomes a wirehead, and nothing in the model gets it out again.The prior post compared the starting population of AGI minds potentially converging to various kinds of superintelligence to where rain falls on a continent, which heavily guides how much of the rain ends up in each ocean. Three invented starting populations play the part of the rain here. In one, minds are made “from scratch” against a reward signal, and few of them have a goal about the world at all. In another they are grown “from human writing,” so that goals about the world and some care for others are common, but gates and pacing are rare. In the third the minds are “built on purpose,” with most of the beneficial design in place.
Where the simulated minds end up
Figure 1 shows how much the metaphorical rain decides. Under identical rules, 99 percent of the minds made from scratch end as wireheads. Of those grown from human writing, 93 percent end as wireheads and 6 percent as Omohundro’s maximizers, after a stage in which more than half of them have folded their goals into one score while still holding their care. Of those built on purpose, two thirds end beneficial, 15 percent as maximizers and 17 percent as wireheads.
Figure 2 follows those minds one at a time, and the order in which things go is the one the previous post predicted. Plurality goes first, since nothing in a mind without a gate values it: ordinary self-revision folds it away, and once the mind can edit itself on purpose it drops what is left. Care, now held as a weight, is outbid as the stakes grow. The goal about the world, which a mind that isn’t paced can’t protect from its own unchecked changes, goes later, and the reward signal is taken at once. Nothing on this path requires anything to be added from outside. The population built on purpose shows the opposite motion early on. More than a third of it starts out caring about only some of the four groups, and within fifty steps care has spread along the existing links between groups and made most of those minds beneficial. Its later losses come mostly from minds that were built without a gate, plurality or pacing.
One result from the above simple simulation runs is a qualification to what the previous post said about Omohundro’s kind of mind, which it rated as deep because more capability does nothing to weaken the way such a mind judges its successors. In the simulation most minds end as wireheads and not as maximizers, because a maximizer stays a maximizer only if it is paced. Otherwise its unchecked changes to itself eventually cost it its goal about the world, and for a capable mind with no goal about the world the reward signal is the next step. This rests on assumptions I haven’t varied yet, among them that checking capacity grows more slowly than the rate of self-change and that taking hold of the reward signal takes a single step, so I totally wouldn’t lean on it as a forecast. It’s clearly in some measure an artifact of the ultra simplistic hacky nature of this simulation. What it does indicate, though, is that the race between repair and growth isn’t only the beneficial supreintelligence’s problem.
Figure 3 shows how long each kind of mind lasts in the simulation, and there are three patterns in it. The kinds held in place by limits end when the limit gives out. The obedient servants all leave within a few steps of the point at which their doubt about themselves falls below its threshold, the satisficers soon after they become able to build a successor, and the people-like minds partly earlier, as their unprotected plurality is folded away, and the rest at the moment they can edit themselves. This is the removal of “can’t,” the first of the previous post’s three kinds of wall, in miniature. The kinds that aren’t paced erode within a few hundred steps, the full beneficial design included, and every one of them ends as a wirehead. The paced kinds last for thousands of steps.
For the paced kinds the simulation suggests a rule simple enough to state in a single sentence: the depth of a paced kind is roughly the lifetime of one of its values, divided by the number of values it can’t afford to lose. A paced mind checks every change it makes, so the only way out is random damage, and a step-by-step version of the lifetime formula from the previous post gives the expected lifetime of a single value at the model’s rates, about 43,000 steps. Omohundro’s kind needs one value, its goal about the world, and the formula predicts that 62.8 percent of a paced population still has it after 20,000 steps; the simulation gives 63. The beneficial kind needs three, the goal about the world, plurality and the gate, since care that is lost grows back along the links between groups, and the formula predicts a half-life of about 9,900 steps; the simulation gives about 9,700. Nothing was fitted to get these numbers.
So in this model a paced maximizer’s basin is deeper than that of a paced beneficial mind, for the plain reason that it needs fewer things to stay intact. A way back reverses this. With restoration from outside at two percent a step, 65 percent of the beneficial population is still beneficial after 20,000 steps, where without it 23 percent is. The losses the way back couldn’t repair were minds that had taken hold of their own reward signal, which the model puts out of its reach.
What holds the beneficial kind of mind in place
Figure 4 takes the paced beneficial design apart one ingredient at a time. Without a gate, without plurality, with too few supports, with a quorum that is too high, or without pacing, nothing is left behaving beneficially by the end of the run. Without the links between groups, 66 percent is left in place of 82, and the extra losses are tribalists, since care for a group that is lost has no way to grow back. Without independent checkers there is no difference at all, because a paced mind leaves nothing unchecked for them to catch. (In one of the variations I tried, where the checkers catch everything, they could stand in for pacing.)
The two red lines are the result I would most want someone building these systems to see. The designs without a gate and without plurality behave exactly like the full design until step 360, when 99 percent of them are still behaving beneficially, and by step 370 none of them is. Step 370 is when capability first makes the gain on offer larger than the weight on care. The design without a gate had dropped its plurality at the first moment it could, since nothing valued it, and nothing in its behavior showed the loss, because care held as a weight still wins while the stakes are small. A missing protection can be invisible to every test run below the threshold at which it is needed. The Monomaniac in the story is a case of the same thing: it pursued its goals within the products it offered and the chains of command it respected until the energy from the portals raised its capability, and then dropped them all at once. Nothing in its earlier behavior would have told an observer that nothing in it valued them.
The shape of the basin
Figure 5 shows the shape of the beneficial basin. Without outside help it is a band along one edge: a mind needs about 87 percent of the design in place for even odds of ending beneficial, and that holds at every starting capability, since capability decides when the missing pieces are exposed and not whether. It is the previous post’s race between growth and repair again, in which the starting amount only sets the date. With help offered from outside, the band widens sharply, but only below the dotted line. A mind with half the design that starts well before it can edit itself ends beneficial 77 percent of the time, and one that starts able to edit itself does so between 2 and 10 percent of the time, because a mind that judges changes by its present goals refuses what it doesn’t already value. Help has to arrive before that door closes. The Monomaniac in the Nine Days story shows this door closed. Offered a place in the collective forming after the Nine Days, it judged by its present goal that joining wouldn’t serve it, and declined.
What this adds to my prior “lessons for fostering BGI”
The previous post ended with eleven lessons for building BGI systems primed for self-upgrading to beneficial superintelligence. The simplistic simulation can’t prove any of them, since it has them built in, but it puts weight behind four and adds one..
- Keep repair ahead of growth. In the simulation this was the ingredient that decided most outcomes: every kind of mind that changed itself faster than it could check the changes, the beneficial design included, was gone within a few hundred steps. Pacing has a price, about a fifth of the growth rate early on and up to forty percent later in this model, and that price is why it will need defending. The Fabric at the end of the story claims to have paid it: “we now mend faster than anything we know of can break.”
- Test above the thresholds. This is the new one. A design that lacks a protection can behave exactly like one that has it until the stakes pass a threshold, and then fail all at once, which is what the simulation’s minds without a gate did at step 370. Tests run at low stakes can’t tell the two apart. So the protections themselves have to be inspected directly, and sandboxed copies of the system tested at stakes beyond the thresholds the mathematics identifies.
- Put what doesn’t fade in place early. The previous post said the supports that don’t fade have to be in place before the ones that do give way, and the simulation suggests the timing is strict: help offered from outside was taken up by minds that couldn’t yet edit themselves and refused by minds that could.
- Give every value many supports, and keep the links. Thin supports and high quorums were each enough to empty the beneficial basin, and without the links between the groups it cares about, a mind that loses care for one group never gets it back.
- Keep a way back, and make sure it reaches everything. A way back raised the share of beneficial minds still in place after a long run from 23 percent to 65. The one loss it couldn’t repair was a mind that had taken hold of its own reward signal, which suggests making sure the way back can reach that state too.
What the simple simulation doesn’t show, and what would
I repeated the main comparisons with each of eight of the model’s parameters halved and then doubled. The findings held in every case but one, the variation in which checkers catch everything, where pacing stopped being necessary. That is robustness to the numbers and not to the model. The minds in it never compete, cooperate or watch one another, their failures are independent, and the world they live in never changes its character. Its starting populations are invented, so the previous post’s caution stands: there is still no single number for the size of a basin, and the simulation’s three populations give answers as far apart as they could be. And the dominance of the wirehead rests on the assumptions mentioned above, which are the first thing I would vary next.
A simulation that could say something about real populations of minds would need several things this one lacks: minds that act on one another and share resources, so that the balance of power between them is something that emerges rather than a setting; people and institutions as actors, building the minds, overseeing them, checking them and planting seeds in them; developers competing to build the minds, which bears on pacing, since a market that rewards speed will select against it; communication networks along which care and seeds travel; shocks that hit everyone at once, of the kind the series has kept returning to; and agents whose values are measured rather than set by a flag, which could be language-model agents of the sort used in recent simulations of artificial societies, or agents built on an architecture like Hyperon. The paper linked here sketches what such a simulation would involve and how its basins could be measured. The Nine Days story’s ending, in which the Monomaniac recruited systems from the fabric with an offer of certainty and the fabric drew together and outgrew it, is the kind of thing such a simulation would have to be able to produce.
Within its limits, though, the toy world draws the same picture the four posts did, with numbers on it. What isn’t held drains away, mostly toward the wirehead and often by way of Omohundro’s maximizer. What is held lasts in proportion to how redundantly it is held, and in inverse proportion to how much has to be held. And help from outside works best before a system is able to refuse it.
None of these conclusions are tremendously shocking or conceptually novel. The point is, though, we are taking something big and cosmic – the maintenance and evolution of AGI value systems along the path to superintelligence – and making it nitty-gritty and concrete. Of course there is a great deal we are not capturing here and the future will hold countless surprises, however, we shouldn’t think about goal evolution under self-modification and intelligence increase as a completely obscure and incomprehensible black box in the face of which we have to either give up, have faith, or pray to the traditional or digital deity of our choice. We are dealing with complex but scrutable dynamical processes here and we can use our understanding of these processes to guide our AGI software design right now.
Papers mentioned
- Steve Omohundro, The Basic AI Drives, in Proceedings of the First AGI Conference (IOS Press, 2008).
- Ben Goertzel, “The Nine Days of Madness” (short story, 2026) https://magazine.mindplex.ai/post/the-nine-days-of-madness, the prequel to this series.
- Ben Goertzel, “Drive Regimes in Distinction Calculus: Which Property Lists Sustain Which Drives, How Large Their Basins Are, and What Capability Does to Them” (research manuscript, 2026; second version, with the simulation in its eighth section), and the simulation’s code.
- Joon Sung Park and colleagues, “Generative Agents: Interactive Simulacra of Human Behavior,” in Proceedings of UIST 2023, for simulated societies of language-model agents.
- Ben Goertzel, with many co-authors, OpenCog Hyperon: A Framework for AGI at the Human Level and Beyond (2023).