When AI Builds Itself: Governance, Power and Competitive Risk

Anthropic recently published an important article, When AI builds itself, about recursive self-improvement: the scenario in which sufficiently capable AI systems begin to contribute more and more autonomously to designing, developing and improving their own successors.

The claim is not that we are already living inside a science-fiction dystopia. Anthropic explicitly says that full recursive self-improvement is not here yet and is not inevitable. But the signal is clear: AI is already accelerating the development of AI, and that acceleration may arrive before institutions, companies and governments are prepared.

This matters because the warning is not coming from an outsider. It comes from one of the most advanced AI labs in the world. When a frontier lab argues that society may need the option to slow or temporarily pause frontier AI development, provided that pause is global, coordinated and verifiable, it is not making a cinematic argument. It is describing a problem of industrial, geopolitical and technical governance.

1. Are we already at the point where AI needs limits?

It depends on what we mean by limits. If it means shutting everything down tomorrow, that is not realistic. AI is already embedded in software, cybersecurity, research, customer support, product development, public administration, healthcare, education and business operations. Stopping everything is not a technical switch. It is a global economic and political decision.

If limits mean defining more serious boundaries for the most powerful systems, then yes: we are already in that phase. Not because AI is sentient, but because the combination of capability, operational autonomy, tool access, iteration speed and economic incentive creates concrete risk.

The weak point is not an isolated model in a lab. It is the model connected to infrastructure, code, cloud environments, agents, tools, sensitive data, workflows and competitive pressure. The more AI can act, the more governance stops being a policy document and becomes a control architecture.

2. Did Terminator and The Matrix simply anticipate reality?

Science-fiction dystopias are useful as metaphors, but dangerous as analytical models. The most likely risk is not a machine with narrative will deciding to conquer the world. It is a sequence of increasingly capable systems, distributed across different organisations, optimised for partial objectives, operating inside competitive markets with insufficient controls.

Terminator and The Matrix are stories about loss of control. That part remains relevant. But real loss of control may not look like a war between humans and machines. It may look like markets moving too fast to regulate, critical infrastructure defended by automated agents against other automated agents, personalised propaganda at industrial scale, scientific research accelerated without adequate validation, or companies no longer fully understanding the systems they produce.

Science fiction imagines an enemy. Reality may give us something harder: an incentive system that makes it rational to run even when slowing down would be wiser.

3. Will companies actually stop?

Probably not spontaneously, and probably not unilaterally. Anthropic is pragmatic on this point: a pause would make sense only if it were coordinated, credible and verifiable across multiple frontier labs and countries. A single company pausing would mostly change who leads the race.

This is the core problem. If Anthropic slowed down alone, OpenAI, Google DeepMind, Meta, xAI, Chinese labs or other actors could continue. If only one country imposed strict limits, other jurisdictions could become more attractive for capital, talent and compute.

The result is a classic arms-race dilemma: everyone might benefit from more time for safety and governance, but every individual actor fears losing advantage if it slows while others keep moving.

4. The competitive advantage of not stopping

If AI really accelerates AI development, the advantage of continuing is not linear. This is not only about shipping a product earlier. It is about accumulating more experiments, more automation, more operational data, more research capacity, more infrastructure integration and more capital.

Anthropic reports that a very large share of its internal code is now written by Claude and that technical productivity has increased significantly. Even with methodological caveats, the signal is strong: when AI systems improve the work of people building AI systems, the curve can accelerate.

That makes governance difficult. A weak rule penalises the actors that follow it. A strong but unverifiable rule creates incentives to defect secretly. A national rule can move the problem elsewhere. What is needed is coordination that is technical, legal and geopolitical at the same time.

5. What role should governments play?

Governments cannot remain spectators. But we should not pretend that public control is automatically more ethical than private control. Private companies seek profit, market power and competitive advantage. States may seek public safety, but also surveillance, social control, geopolitical influence and military capability.

The risk is not only "Big Tech versus citizens". It is also "state versus citizens", "states versus states", "companies and governments intertwined without transparency", and "a technology too powerful to be concentrated in very few hands".

The idea of governments taking equity stakes in frontier AI companies is interesting, but not sufficient. Public ownership may create visibility and influence, but it can also create conflicts of interest or turn the AI race into a national strategic asset. The real question is not only who owns the company. It is who can verify what is being trained, with which data, under which security levels, with which deployment limits and with which accountability in case of harm.

Where does an ethical, pro-human world begin?

An ethical world does not emerge by choosing between absolute free markets and total state control. It emerges from a sufficiently balanced distribution of power: innovative companies, public rules, independent research, civil society, technical audits, international standards and transparency proportional to risk.

In practice, some measures seem unavoidable:

• registration and classification of frontier training runs;
• independent audits of security, cybersecurity, capabilities and alignment;
• mandatory incident reporting for high-risk deployments;
• pre-release evaluations for models with advanced agentic capabilities;
• controls around access to compute, data centers and chips for frontier training;
• international verification mechanisms, not only voluntary statements.

None of this is simple. But the alternative is governance arriving after an incident, as often happens with critical technologies.

Benefits, defects, possibilities and risks

The possibilities

AI that accelerates research and development could produce enormous benefits: faster drug discovery, safer software, scientific progress, better educational tools, more efficient public services, proactive cybersecurity and more productive companies.

If governed well, this technology could amplify human capability rather than blindly replace it. It could make rare, expensive or concentrated expertise more widely available.

The defects

Models make mistakes, hallucinate, optimise poorly, can be manipulated, can generate vulnerable code, can amplify bias and can produce outputs that are persuasive but false. When models become agents, these defects do not remain text on a screen. They enter processes.

As automation grows, humans also risk losing operational understanding. If the human role becomes only reviewing systems that move faster than humans can inspect, review can become a bottleneck or a ritual.

The risks

The major risks are not one thing. They include cybersecurity, economic concentration, disinformation, surveillance, infrastructure dependency, displacement of skilled work, regulatory capture, military escalation and technical loss of control.

Recursive self-improvement makes these risks more delicate because it changes the time available to react. If improvement cycles become too fast, the slower rhythms of politics, culture, education and law may fail to keep up.

Is a pause possible?

A useful pause would need at least four characteristics: it must be multilateral, verifiable, bounded by clear criteria and accompanied by real work on safety and governance. A symbolic pause does not help. An unverifiable pause can make things worse. A pause imposed only on some actors can shift advantage to the least cautious ones.

This is why the most important part of Anthropic's argument is not simply "let's stop". It is "let's build the conditions that would make a credible slowdown possible if it becomes necessary". That is a very different proposition.

Conclusion: the question that remains

Perhaps AI will never become capable of improving itself to the point where the human role becomes marginal. Perhaps the curve will slow. Perhaps we will discover technical, energy, economic or scientific limits stronger than we currently imagine.

But if even part of the scenario described by Anthropic moves closer to reality, the problem will no longer be only building better models. It will be building institutions, companies and cultures capable of withstanding the speed of what they are creating.

The real question, then, is not whether Terminator or The Matrix were right. The more uncomfortable question is this: can we govern a technology that rewards acceleration, even when the wiser choice may be to slow down?

---