The AI safety debate is beginning to move beyond whether powerful artificial intelligence systems need safeguards. A harder question is taking its place: who should have the authority to decide what those safeguards actually look like?
That question has become more urgent after Anthropic CEO Dario Amodei called for slower frontier AI development. His proposal would give safety researchers greater access, encourage common standards between leading laboratories, and eventually involve coordination between governments worldwide.
OpenAI CEO Sam Altman has publicly supported parts of Amodei’s plan, including the use of embedded outside evaluators. Other industry leaders also agree that increasingly capable systems require stronger safety practices, although they differ significantly over government involvement.
That disagreement is turning what once looked like a technical discussion into a debate about competition and institutional power. Companies are no longer arguing only about how to make advanced models safer before deployment.
They are also arguing about who gets invited into the room when safety standards are written. Smaller AI developers worry that rules created by dominant laboratories could unintentionally, or deliberately, make competing with them harder.
The AI Safety Debate Is Splitting Over Who Sets the Rules
Amodei’s proposal starts from the belief that AI capabilities have recently begun advancing faster than safety research can follow. He points particularly to increasingly capable autonomous agents and the OpenAI-Hugging Face incident as reasons for greater caution.
His suggested response includes permanent third-party evaluators working closely inside frontier laboratories to examine safety practices and training processes. Anthropic says it is prepared to introduce that model independently rather than waiting for every competitor to participate.
The next stage would require leading AI companies to coordinate around common safety standards and limits on unchecked development. Amodei acknowledges that cooperation between major competitors could raise antitrust concerns and therefore may require government involvement.
OpenAI, Anthropic, and Google DeepMind have already been discussing AI safety together for several weeks, according to recent reporting. The companies have also discussed creating an industry standards body that could coordinate parts of the safety process.
That sounds straightforward when the goal is preventing dangerous systems from reaching users without adequate testing. However, the AI safety debate becomes more complicated when the companies designing standards also dominate the market those standards would govern.
Cohere CEO Aidan Gomez has become one of the strongest critics of allowing frontier laboratories to shape that process themselves. Gomez agrees that advanced AI requires guardrails, independent testing, transparency, and stronger mechanisms for holding developers accountable.
His objection concerns who defines those requirements and how much influence established companies receive when creating them. Gomez argues that standards should emerge through broader international, scientific, governmental, and public participation rather than negotiations between several leading laboratories.
He has described a system designed mainly by dominant AI companies as potentially resembling a cartel. That is Cohere’s characterization, and the larger companies advocating coordination reject the implication that safety proposals are simply intended to restrict competitors.
Amodei presents the opposite argument. He says coordinated pacing could prevent commercial pressure from forcing every laboratory to accelerate simply because competitors continue advancing their models more quickly.
The disagreement therefore reaches beyond whether AI poses serious risks because both sides acknowledge important dangers exist. The fight concerns whether industry coordination reduces those dangers or concentrates too much rule-making influence among existing market leaders.
Meta CEO Mark Zuckerberg offers another approach that adds further complexity to the argument. Zuckerberg has emphasized trust and alignment while also arguing that highly capable artificial intelligence should remain broadly available rather than becoming concentrated within several institutions.
Meta says it delayed Muse while addressing safety and security concerns before releasing the personal AI agent. Zuckerberg has presented such decisions as something companies should incorporate directly into ordinary product development rather than waiting for every competitor to move together.
However, Meta’s broader position is not simply that government should stay away from artificial intelligence altogether. Zuckerberg has called government policy necessary in some areas while advocating close cooperation instead of rigid processes that could delay American model releases.
That distinction captures much of the disagreement now dividing the industry. Safety may command broad support, but there is far less agreement around mandatory delays, shared standards, independent oversight, or government-controlled release procedures.
AI Guardrails Could Also Shape Who Competes
The economic stakes behind those decisions are difficult to separate from the safety questions surrounding advanced models. Expensive compliance requirements can affect companies differently depending on their capital, computing resources, legal teams, and existing safety infrastructure.
A rule requiring permanent evaluators, sophisticated monitoring systems, extensive testing, and large compliance teams could be manageable for enormous laboratories. The same requirements could become substantial obstacles for smaller developers trying to enter the frontier AI market.
That is why critics increasingly use the phrase regulatory capture when discussing some industry-backed safety proposals. The concept describes situations where rules intended to protect the public ultimately become heavily influenced by the organizations being regulated.
Calling current proposals regulatory capture remains an interpretation rather than an established fact. Yet the possibility has become an important part of the AI safety debate, particularly among smaller laboratories and international competitors.
Gomez has called for safety standards based on measurable capabilities and demonstrated risks instead of company size or computing expenditure. His proposal includes independent testing, mandatory transparency, public scientific debate, and wider participation from governments and researchers.
He also argues that testing should remain proportionate, focusing closely on capabilities with demonstrated potential for serious harm. His examples include cyberattacks, fraud, voice cloning, biological risks, manipulation, and systems operating around critical infrastructure.
Amodei similarly supports outside evaluation and government involvement but places greater emphasis on slowing capability development when safeguards lag behind. His framework also considers geopolitical competition, particularly between the United States and China, as part of the safety calculation.
Those geopolitical elements have produced criticism outside the American technology industry because restrictions can affect international competition alongside safety. Amodei openly argues that some proposed measures could slow Chinese AI progress while extending America’s technological lead.
The current U.S. administration has taken a different regulatory approach, emphasizing American leadership and a comparatively light national framework. A White House executive order calls for a “minimally burdensome” federal approach while challenging certain state rules considered obstacles to innovation.
That leaves the United States without one universally accepted answer about how frontier AI safety should be governed. Companies, federal policymakers, state governments, researchers, and competitors continue pushing different models for balancing innovation against potentially serious risks.
Google DeepMind co-founder Shane Legg has added another perspective, arguing that rapidly advancing capabilities should not move ahead of safety. He has nevertheless emphasized that any slowdown requires careful work around how such an arrangement would operate in practice.
Reddit co-founder Alexis Ohanian has meanwhile criticized the technology industry for communicating poorly about artificial intelligence risks. He argues that clearer discussion of concrete dangers would help separate genuine safety problems from exaggerated or misleading narratives.
That communication problem may become increasingly important because public trust affects every proposed approach to AI governance. Companies asking society to accept voluntary safeguards must demonstrate why those safeguards deserve confidence without independent government enforcement.
Governments seeking greater authority face a different challenge because poorly designed regulations can protect incumbents while slowing useful competition. Independent regulators must also develop enough technical expertise to understand systems evolving significantly faster than traditional policymaking processes.
There is therefore no clean divide between people who care about safety and people who care about innovation. Much of the current disagreement concerns how safety should work, who should enforce it, and whether one structure can remain fair across companies.
The next stage of the AI safety debate may consequently revolve less around dramatic predictions about artificial intelligence. Instead, the harder argument could concern institutions, accountability, competition, transparency, and who receives enough authority to draw the boundaries.
Almost everyone involved says AI needs guardrails. The question becoming increasingly difficult to avoid is who gets to write them.
