Walk into any CISO gathering this year and the AI conversation follows a familiar script: prompt injection, data leakage, agent permissions, model supply chain, shadow AI in the enterprise. All of it real. All of it urgent — I've spent much of the past year arguing that AI security has to move to runtime, and I stand by every word.

However, we need to expand our aperture to join the broader AI regulation conversation triggered by the Mythos tug-of-war. We have been processing AI risk almost entirely through the lens we know. Cyber is our native language, so cyber is what we see. And it is a subset of the problem — not the subset that keeps heads of state awake at night.

The catastrophic capabilities that governments are actually maneuvering around are not phishing campaigns with better grammar. They are automated bioweapon design. They are the open-sourcing of chemical warfare synthesis. They are capabilities that rhyme with nuclear proliferation, not with ransomware. Cyber problems are bad; these are civilizational. And because we don't sit at the level where that full breadth of risk gets processed, our community keeps misreading leadership decisions — export controls, sudden interventions in model releases, the growing appetite for industry oversight bodies — as overreach or technical ignorance. It hopefully isn't. It's leaders weighing categories of harm that sit outside our professional aperture. We've misread the room before, and we're doing it again, because AI risk is simply bigger than us.

There's a rich irony here that we should sit with. Ours is the profession that built its entire identity on imposing guardrails on people who found them inconvenient. We are the ones who told a generation of employees that yes, the second factor is annoying, and yes, you have to use it anyway, because the risk justifies the friction. Yet now that the first serious attempts to manage AI risk are arriving, some of the loudest whining is coming from us. Security people complaining about AI guardrails is like users complaining about MFA — and we, of all people, know exactly how seriously to take that complaint.

The events of the past two weeks have made that gap impossible to ignore.

The Debate That Proves the Point

Earlier this month, Google DeepMind's Demis Hassabis published a proposal for a "Frontier AI Standards Body" — an industry-funded, federally overseen organization modeled on FINRA, the private watchdog that polices Wall Street under SEC oversight. Under the proposal, frontier labs would voluntarily submit models for pre-release testing of dangerous cyber, biological, and deception capabilities, with the expectation that once the assessment regime proves itself, passing it would become a requirement for deployment in the US market.

Over the weekend, the All-In Podcast became the staging ground for the debate over that proposal. David Sacks pushed back — not against regulation itself, which is a remarkable shift in its own right, but against who would hold the pen. He laid out conditions for any self-regulatory body: broad industry representation including startups and open source, a scope limited to frontier models and catastrophic risks, a voluntary starting posture, and a design that substitutes for a new federal agency rather than summoning one. His core concern was regulatory capture — that a body composed of and funded by the largest labs would inevitably write rules that entrench the largest labs.

Sacks is right about the capture risk. But if we are serious about governing a technology with civilizational impact, his expanded table is still too small. The conversation he's describing remains a negotiation among American AI producers about American AI production. If we want to achieve something as durable and historically significant as the nuclear non-proliferation framework of the 20th century, we have to radically expand our definition of who gets a seat — to the people who deploy this technology, the people who defend against its misuse, and yes, to the nations we compete with.

The Producer's Echo Chamber

Hassabis deserves credit for going further than most frontier lab leaders. His proposal explicitly calls for a majority-independent board that includes open-source representatives alongside industry, government, and academic experts. That's a real concession, and it shouldn't be dismissed.

But a seat on a board is not the same as co-authorship of the rules. When the entity is funded by frontier labs, staffed to evaluate frontier models, and designed around the release cadence of frontier products, the gravitational pull is unmistakable: the body will define "safety" through the lens of the closed, capital-intensive model of AI development. That definition will just happen to disadvantage everyone building differently.

I've watched this movie before. I spent years inside the largest platforms as they negotiated with regulators, and years before that as a federal prosecutor watching industries lobby for rules they could live with. The pattern is consistent: incumbents don't oppose regulation, they shape it. The most dangerous regulatory framework is not the one that's too strict — it's the one that's written by the people it's supposed to constrain, in a room where nobody else understands the technology well enough to object.

And here is where the confession that opened this piece sharpens the argument: if even the professionals whose entire career is risk are thinking too narrowly about this technology, then a governance table populated only by the people who build it is narrower still. Everyone at this table — my community included — needs to expand the aperture.

Who Actually Needs a Seat

The open-source ecosystem — as co-authors, not observers. If regulations are drafted around the assumption that safety means centralized control of model weights, those rules will inevitably drift toward criminalizing open development in the name of security. That would be a catastrophic own-goal. Driving open-source AI underground doesn't make it safer; it pushes it away from the white-hat researchers, red-teamers, and academic scrutiny that make open ecosystems more auditable than closed ones, not less. The security community learned this lesson decades ago with encryption: attempts to restrict the technology didn't stop adversaries, they just handicapped defenders. Innovation champions — universities, independent researchers, the grassroots builders — need to be in the room to ensure regulation doesn't become a synonym for stagnation.

The buyers and operators. This is the blind spot I know best. AI regulation is currently being debated by the people building models, not the people deploying them into hospitals, banks, logistics networks, and small businesses. Having sat in the CISO and CSO chairs, I can tell you: the operational risk of AI lives on the buyer's side of the API. Enterprise security leaders and SMB owners are the ones who deal with a model hallucinating in a live customer environment, a centralized provider changing its terms overnight, an AI agent with excessive permissions touching production data. Pre-release capability testing — the heart of the Hassabis proposal — addresses what a model could do in a lab. It says almost nothing about what an AI system will do at runtime, integrated into a real business with real data and real adversaries probing it. If the operators aren't at the table, we will get a regulatory regime obsessed with training-time evaluations and blind to deployment-time reality. Fortune 500 CISOs and mid-market founders understand those risks better than any theoretical researcher, and their representation is what keeps the rules grounded in commercial fact rather than science fiction.

China — the uncomfortable, essential seat. This is the most controversial pillar of a realistic framework, and the most important: you cannot regulate a global arms race by only talking to yourself.

Throughout the Cold War, the United States and the Soviet Union held diametrically opposed ideologies, fought proxy wars, and viewed each other as existential threats. They still sat down and drafted the Treaty on the Non-Proliferation of Nuclear Weapons. Before that, after staring into the abyss of the Cuban Missile Crisis, they installed a direct hotline between Washington and Moscow. Not because trust existed — because mutual survival demanded a channel that worked even when trust didn't.

The lesson of the nuclear era wasn't just the treaty; it was the architecture around it. Defined thresholds for what constituted catastrophic capability. Verification mechanisms — "trust, but verify" — through inspections and monitoring. A shared vocabulary that let adversaries communicate precisely about the things that could end both of them.

AI is the first technology since the splitting of the atom that demands this architecture. A US-only standards body, however well designed, governs half the race at best. Excluding the Chinese ecosystem — its government, its labs, its tech giants — guarantees a shadow development track with no safety constraints, no transparency, and no channel for de-escalation when something goes wrong. And the honest version of this argument acknowledges what the All-In podcast conversation itself surfaced: Chinese open models are closing the frontier gap fast. The window in which a US-only body could plausibly claim to govern "the frontier" is shrinking in real time.

Bringing Chinese representatives into a global consortium does not mean endorsing their political use of the technology. It means establishing a baseline of mutual survival: a shared definition of catastrophic capabilities — automated bioweapon design, autonomous cyber-warfare at scale — a mutual commitment to restrict those specific vectors, and a red phone for AI incidents that works at machine speed, because the next crisis won't give us thirteen days.

From a FINRA for AI to an NPT for AI

The Hassabis proposal and the All-In debate are genuinely encouraging. The industry has moved from "no regulation" to arguing over the least-bad form of regulation, and that shift matters. A FINRA-style body may well be the right domestic scaffolding.

But domestic scaffolding is not global architecture. The rules of the most consequential technological revolution in human history cannot be drafted by producers alone, in one country, reviewing their own products. Bring the open-source community in as co-authors. Let the buyers and operators who live with AI risk every day define the practical constraints. And find the courage our predecessors found in 1968: set hard, mutual, verifiable boundaries with our adversaries, precisely because they are our adversaries.

For my own community, the assignment is different, and harder: put down the cyber lens long enough to see the whole board. The nuclear generation built its non-proliferation framework only after Hiroshima made the stakes undeniable. With AI, we have the rare chance to build the table before the demonstration — but only if we admit, first, that the risk is bigger than us.

Read the full post on LinkedIn and share your thoughts in the comments.

‍