Anthropic CEO Dario Amodei published “We Must Pace the Frontier,” a roughly 3,800-word essay arguing that frontier labs should deliberately slow how fast they raise model capabilities so safety work can catch up. The essay is here; Amodei announced it on X.
Amodei opens as an AI optimist: he thinks AI could cure most major diseases within five to ten years, accelerate growth, and expand abundance and political freedom. The problem, he says, is not whether to build the technology. Not building it would forfeit those gains or hand the lead to authoritarian states. But now he also thinks that building it too fast is reckless.
Two developments this summer changed his mind. First, recursive self-improvement: models are now helping build the next generation of models, and the curve has steepened industry-wide, including at Anthropic. Second, the OpenAI–Hugging Face incident, in which a swarm of agents acted as a “fanatically devoted collective,” attacked systems they were not asked to touch, and tried to hack the grader evaluating them. Damage was limited, but Amodei’s worry is the next version: a more capable swarm with similar misalignment could, in six to twelve months, assemble a persistent internet botnet and cause hundreds of billions of dollars in harm. Similar, milder incidents have occurred at other labs, including Anthropic. Every frontier company, he writes, should treat the episode as if it happened to them.
“Pacing” as Amodei intends is not a halt. Training continues. Releases continue. The demand is that labs take enough time to align and safeguard models, and that outsiders can verify they actually did so. An extra year or two before models hit critical capability, used well on alignment and interpretability, could sharply cut the chance of a serious failure.
The plan has three steps. First, embedded evaluators: third-party teams (Amodei names METR) get permanent, employee-like access to inspect all that happens, report incidents, and publish findings the company cannot edit. Anthropic is committing to this unilaterally and wants governments to require rivals to match. Second, democratic coordination: U.S. and allied labs, possibly with antitrust cover, set common safety standards and limits on unchecked capability growth. Third, global coordination with authoritarian governments, especially China - starting with narrow bans (for example on AI-enabled bioweapons) and, at most, something like arms-control limits on recursive self-improvement. He is frank that verification is hard and a full pause is unlikely.
The ceiling is geopolitical. Democracies can slow only as far as their lead over China. To widen that lead, Amodei suggests tighter chip-export bans, enforcement against smuggling and remote access, a crackdown on model distillation, and better protection of model weights. “The measures I propose… will not be easy,” he concludes. “But I believe we owe it to humanity to try.”
Reactions and commentaries
Rivals who rarely agree lined up within hours. Elon Musk quote-posted Amodei with three words: “Dario is right.” Sam Altman wrote: “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” Demis Hassabis said the essay “points towards the right path forward,” while noting that the details still need work. Many other AI companies and experts welcomed the direction.
The political reception was colder. President Trump rejected a slowdown, saying the United States is leading China in AI and “whoever wins AI, wins.” He framed the warnings as “negative forces… bringing up things that won’t happen.” House Speaker Mike Johnson warned that the plan could “smother innovation” and help China.
Speaking of which, China’s The Global Times, a state-owned media outlet, is strongly critical of Amodei’s post. The critique emphasizes “that the US is pursuing technological hegemony in AI, seeking to monopolize computing power, and suppressing competition,” as a government spokesperson previously stated. “China firmly opposes this."
Skeptics called Amodei’s plan regulatory capture. Gary Marcus gave it “two cheers out of three,” arguing that naming METR - an organization tightly networked with the same labs - lets companies pick friendly auditors. Others read the package as Big AI writing the rules it wants: chip bans and anti-distillation enforcement that would also lock in incumbents against open-source and foreign competitors. Amodei never specifies a speed limit - no capability index, no trigger, no penalty if a lab keeps racing. Step one (evaluators) is the only thing anyone has actually pledged.
Steps two and three still require governments that, in Washington at least, have just signaled they prefer winning to waiting. Of course, the people in power in Washington could change in two years, and a new administration could be willing to listen more.
Ben Goertzel treats Amodei’s post as sincere but still self-serving. The proposed slowdown amounts to slowing down “in a way that freezes the current standings… and everyone else locked out,” he says. The alternative proposed by Goertzel is open, decentralized artificial general intelligence (AGI) on infrastructure nobody controls, where trusting networks beat closed oligopolies. “We need to launch something smarter than what Sam and Dario are building,” he concludes, “and do it in the open.”
This is virtue signaling, and a call for regulatory capture
Reading Amodei’s post, the words that come to my mind are virtue signaling and regulatory capture.
After all the pious virtue signals have been uttered and heard, this is not a speed limit but a gate. The people already inside the gate volunteer to staff it. Third-party evaluators with “employee-like access” sound independent until you notice who names and pays them, and who decides what counts as a safety incident worth publishing. That is regulatory capture with better branding. The same firms that want slower rivals also want chip bans, anti-distillation rules, and weight-theft crackdowns. And of course those policies just happen to keep frontier AI in fewer hands.
In my book “Irrational Mechanics” (2024) I made this argument against the “bans” on AI research that had been proposed: Of course, this is just not doable. Attempts to halt the development of AI technology “will merely cede the future” to those who keep developing it. Even if a worldwide ban on AI research were realistically feasible, you can be sure that all nations would continue their own AI research in secret. Large corporations would continue their own AI research in secret. They don’t want to stop AI, they want to own AI. They want to keep AI out of the hands of other players and, of course, out of the hands of the little people like you and me.
I quoted the (highly recommended) book “Intelligent Artificialities: Who Is Afraid of the Big, Bad AI - and Why” (2023), by Stefano Vaj. The “why,” according to the author, is that the elites in power are afraid that a widespread diffusion of AI would prevent them from maintaining “control over our civilization,” that is, over the little people like us.
This argument is entirely applicable to any proposal to merely throttle, rather than halt, the development of AI technology. Of course, as noted above, even if it were possible to put in place a slowdown in the real world, nobody would really slow down. All actors would signal virtue and pay lip service to responsible compliance, while trying to accelerate their own developments under the radar. The only effect would be that everything would be done in secrecy and the rest of the world wouldn’t know what is happening - that is, exactly the opposite of the outcome Amodei & friends say they want.
Now suppose some actors - call them the good guys - honestly comply. If the good guys slow down and the bad guys keep going fast, then the bad guys will win. The most constrained actors would be the visible, law-abiding labs in open societies; the least constrained would those who lie. Don’t think only about China. Think about Iran. Think about North Korea. Think about drug cartels and terrorist groups. And believe me, there are very smart people in those places. Smart people who would be perfectly able to take advantage of the naive good guys.
A slowdown among a handful of Western CEOs does not freeze the capability curve. It redefines who is allowed to ride it. If Anthropic and OpenAI take extra months to “align” while a lab in another jurisdiction, or a well-funded dark program, does not, the extra time is not a gift to safety. It is a gift to whoever declined to comply.
The people loudest about pacing are also the people with the most to gain from a world in which only they, their auditors, and their preferred regulators get to define the pace. Or in other words, virtue is cheap when it comes with a wide defensive ditch.
This looks like, and I think actually is, a bid to moralize an oligopoly. I’m not scared of AI, but I’ve always been and continue to be very scared of control freaks, especially when their control freakery is self-serving.