Who Gets to Define the Rules for AI? | Cohere

Here is the question nobody is asking clearly enough: should a handful of select, market-dominant AI companies from Silicon Valley get to define the rules and safety standards of a generational technology for the entire world? All while simultaneously determining how fast this technology progresses? We have tried that before with very poor results. Once again using fear under the pretext of protecting the public, these oligopolies are now requesting to bend competition rules and be permitted to dictate the terms for everyone else. A wolf in sheep’s clothing, a cartel by any other name.

AI needs guardrails. That is not the dispute and never has been. The dispute is over who writes them, who gets to participate and whose interests the rules are protecting. The question is truly about whether we should have the freedom to choose based on scientific evidence or if we should hand the reins of the most consequential technology in human existence to a few Silicon Valley executives.

In 1975 the Securities and Exchange Commission needed reliable bond ratings for its capital rules. It designated three firms as Nationally Recognized Statistical Rating Organizations and never published criteria for how anyone else might earn the designation. These were government-blessed outside evaluators, paid by the very issuers whose securities they graded, sitting behind a barrier the regulator itself had built. Twenty five years later there were still only three of these evaluators. Then they rated subprime mortgage securities triple-A and nearly took the global economy down with them.

Europe ran the experiment again in 1985. Car manufacturers lobbied for a sweeping antitrust waiver, the Motor Vehicle Block Exemption, arguing that modern vehicles were complex, safety-critical machines and that manufacturers therefore needed control over who was qualified to sell and service them. The subsequent regulation let manufacturers set the standards for premises, equipment and staff training, explicitly in the interest of safe and reliable vehicles. But what followed wasn't safer cars. It took the European Commission roughly twenty five years of reforms to unwind, and to establish what should have been obvious at the start: it is possible you can hold strict safety standards without handing the incumbents a monopoly on meeting them.

I want to be clear about what we agree with. Independent review of highly capable AI systems is a good idea and we support it. However, many aspects of the proposal raise fundamental questions: who writes the standard those reviewers apply? Who conducts or oversees the review? Who gets to participate in the conversation that sets the rules?

On these questions, the proposal is clear. A handful of the most powerful labs based in one country would agree on shared standards and the limits to how fast the technology should advance. And here’s the key point: because it’s normally illegal for competitors to agree to limit what they produce, the plan asks governments for a narrow antitrust waiver to make that coordination lawful. And it also asks governments to require every other AI developer to blindly follow whatever the participants settle on – despite those other developers and wider society not having an opportunity to voice the impact or share their perspective on the science.

It promises that a coordinated approach would give developers time to do this safety work without sacrificing commercial advantage. But to whose advantage? The firms drafting the framework are the firms sitting at the top of the market today. A mechanism that slows everyone down while explicitly preserving existing commercial advantage does not make AI safer. It risks entrenching today’s dominant AI companies by turning their current advantages into baseline for what it takes to compete safely.

Convince a government that AI is an existential threat and you can convince it to outlaw your competition. The intention is clear and it does not create a safer world.

So what will enable safe, responsible AI development? To be clear, I don’t believe I have all the answers - nor do I think I should get to make the rules instead. Rather, I will try to propose practical and effective ideas that can be considered alongside those of many others by governments and lawmakers as they use their democratic powers to set the direction of travel.

An evidence-based risk framework.

Mandatory transparency.

Testing, scoped by the evidence.

Real assurance mechanisms.