.svg)
In "Trust Us, We're the Experts," Joe Sullivan argues that AI safety is repeating a pattern he's watched for two decades — at eBay, Facebook, and in his own federal case at Uber: powerful companies deciding behind closed doors what a technology is allowed to do, until governments step in and ask who gave them that authority. His warning is that "trust us, we're the experts" has never been a sufficient answer — but neither is a panicked, global off-switch. Sullivan makes the case that AI risk is already spread across many hands — chipmakers, cloud providers, governments, the labs, developers, and defenders — and shouldn't be defined by the companies selling the product. The path forward, he argues, is defense in depth, independent oversight, real transparency, and government-led coordination between the public and private sectors.
Last week one of the biggest AI companies in the world shut off its two most powerful models for every customer on the planet. Not slowed down. Not blocked in a few countries. Turned off everywhere, after the U.S. government ordered it to cut off access for foreign users on national-security grounds. The company said the order didn't follow any fair or fact-based process. A few days before that, it had apologized for quietly weakening the same model for outside researchers without telling them. And a few days before that, it had asked Washington for more power to police AI, including the power to pull dangerous models off the market.
It looks like chaos. It isn't. It's a wrestling match that has played out again and again between technology companies and government. Every time, the innovators act like they're smarter than the rest of the world; every time, the government looks heavy-handed when it finally steps in to impose safety.
Here's how it starts. A private company ends up making decisions that affect millions of people. It makes those calls based on its own values and its own read of the technology. Then the government shows up and asks the only question that really counts: who said you get to decide?
The first time I saw this up close was in 2002 at eBay, long before "content moderation" was a phrase anyone used. Part of my job was deciding what people were allowed to sell. That sounds simple until you remember what the internet did: it erased distance and borders. Suddenly someone could sell a car, a gun, a bottle of pills, or a protected animal part to a buyer two thousand miles away, in a state or country with completely different rules. A sale that was perfectly legal where the seller sat could be a crime where the buyer lived. Regulators who had spent decades policing what crossed their borders woke up to a marketplace that ignored them — with a private company in California writing the rules. Cars were one of the first real fights, along with guns and alcohol and anything else a government had reason to watch. We were setting commerce rules that used to belong to the states, and we were doing it on our own.
Years later I ran security at Facebook while the same fight moved from goods to speech and data access. The question changed from "what can you sell" to "what can you say," or "tell us who said it," but the shape was identical. A private company was deciding, for billions of people, where the line sat. One side of Washington said we took down too much. The other said we took down too little. Both were sure we were picking winners. We were making rules that used to belong to the public square, and we were doing it on our own.
My own time at Uber ended in a federal courtroom, so I've felt the sharp end of this personally. I'll let others argue the details of my case. But stripped down, it came from the same tension: a company making decisions about how to handle a security problem, and a government that believed it hadn't been kept in the loop the way it should have been. Whatever you think of how that played out, the friction underneath it was the one I'm describing. The decision got made inside the company. The government felt it was on the outside looking in.
Now it's AI, and the pattern is back, bigger than ever. The companies building these models are deciding what the technology can do, who can use it, and where the limits are — based on their own values and their own belief that they understand the risks better than anyone else. And the government has shown up again with the same question it asked me about cars, asked us about speech, and asked me again in court.
Four things repeat in every one of these fights:
With cars and speech, the government wanted the companies to change their rules — sell less, take down more, take down less, do more to protect kids. This time, in one context, the government wants the company to drop a limit it put on itself — its refusal to support certain military uses — and the company is saying no. So it's the company holding a moral line against the government, not the government reining in the company. That flips the usual story on its head, and anyone who reuses the old script without noticing will get the politics backwards.
This time, the government actually has the legal power. It was always hard for the government to touch what people said online. But AI models can be treated like other powerful technology with obvious military uses — the kind the government is allowed to control when it crosses borders. So when it pulled the lever last week, it was, at least on paper, using real authority. It could never do that to a Facebook post.
This time, the stakes are bigger, and that cuts both ways. Bad content is real, but you can usually recover from it. The risks the labs talk about — helping someone build a weapon, or automating cyberattacks — are on a different scale. That's the best reason to take their caution seriously. It's also the best excuse for "only we can be trusted to decide," which is the oldest line in the book.
And this time, the companies have a money problem the platforms didn't, three ways at once. A frontier lab sells the dangerous product, asks the government to regulate the dangerous product, and — in at least one case — is about to go public on the strength of both stories. Asking the government to build the fence around your own yard, weeks before your IPO, is a hard look to pull off.
So where's the hubris? Not in being confident about the technology. Most of the time these companies really do understand their own systems better than their critics. The hubris is the next step — the quiet jump from "I understand this best" to "so I should be the one who decides." Those are two different things. Knowing how something works doesn't make you the right person to decide what it's allowed to do, and to whom. That call belongs to a lot more people than the engineers.
Last week handed us a perfect example. The company quietly made its model worse for certain outside users and didn't tell them. The thinking was: we know better, so we'll decide for you, and we won't even mention it. That's not safety. That's deciding alone and hoping no one notices. The tell is how fast they reversed it and apologized once people did. Deep down they knew the problem wasn't the judgment. It was making the call in the dark.
That's the lesson I keep relearning, all the way back to eBay. The thing that gets you in trouble isn't building the technology. It isn't even understanding it best. It's deciding alone — making calls that affect everyone while keeping the public and its government out of the room. I've made versions of that mistake myself. It keeps happening because every new technology makes a small group of people feel like they're the only ones who really get it. That feeling is powerful, and it's wrong.
We can't let the government off the hook either. The shutdown was abrupt and without precedent, and that makes life impossible for business. Regulators need to set objective standards companies can understand and follow. Right now nothing about the rules for AI development and distribution is clear. That's the other powerful player making its own call and skipping the process.
So the real picture isn't a humble government against arrogant tech. It's two powerful sides, each sure it's acting for the public good, each unwilling to follow the rules it wants the other one to follow. The perceived hubris runs both ways.
So here's the harder question underneath all of this. How do you keep a powerful technology from helping someone build a weapon or break into critical systems, when the people who understand it best have every reason to decide alone, the government's power is real but easy to abuse, and there's a fortune riding on the answer? "Trust us, we're the experts" has never been the answer. But neither is throwing one global off-switch in a panic.
The good news is that there's no single moment where you either stop the harm or you don't. The risk can be lowered at many points along the way — from the chips a model is trained on to the machine that would actually have to build the dangerous thing in the real world. And here's the part that matters most: most of those points aren't controlled by the AI company at all.
That's exactly why we have to be careful right now. The product companies have every reason to tell us the place to manage this risk is inside the model, during training, using guardrail methods only they understand. Of course they do. That's the part they own, and the part where "we're the experts" keeps the decision in their hands. But that's a choice about where to look, not a fact about where the risk lives. So let me walk the whole chain.
It starts before a model exists. Training a frontier model takes a huge, rare amount of computing power and specialized chips. The chipmakers, the cloud providers who rent the computing power, and the government that controls exports are all chokepoints — and none of them is the AI lab. There's also a choice, right at the start, about what goes into the training data. The most dangerous step-by-step material doesn't have to be in there.
While it's being built, you test the model as it gets more capable, to see whether it's actually getting better at the things you're afraid of, and you bring in outside teams to attack it before it ships. And there's a quiet, unglamorous control that matters more than almost any other: protecting the model itself. A trained model is, in the end, a file. If that file is stolen or leaked, every other safeguard here disappears at once. Securing it is a safety control, even though it looks like ordinary security work.
Then comes the decision everyone is focusing on: how you release it. A model behind a locked door, where the company can see who's using it and cut off bad actors, is a completely different risk than one whose full inner workings are posted publicly for anyone to download and change. (Let's save the debate over open source models for another day.) That choice can't be undone — once it's out, it's out, and almost every later control stops working. It's the decision that most deserves an outside check, and the one labs most want to keep for themselves.
While the model is running, you can make people prove who they are before they reach the most sensitive capabilities. You can watch for the patterns that actually signal danger — not one bad question, but someone working their way toward a real recipe or a working attack. And you can cut off specific users fast. Notice the contrast with last week: a precise, targeted shutoff is a control; flipping the whole thing off for the planet is what it looks like when you never built the precise version.
For the cyber risk especially, the real chokepoint is what you let the model do on its own. A model that explains a concept is one thing. A model wired up to act by itself — to scan, write code, run it, and move from one system to the next with no human in the loop — is something else entirely. Keeping these systems boxed in, limiting which tools they can reach, and making a person sign off on serious actions is where cyber risk actually gets contained.
Then there's the last mile, outside the model entirely — the part the AI companies can't help with. For biological risk, the real-world chokepoint isn't the chatbot; it's the handful of companies that synthesize custom DNA to order. If they screen what they're asked to make and check who's asking, the dangerous path is blocked no matter what the model said, and whether or not it was jailbroken or stolen. That's a more reliable control than anything inside the model — and no AI lab runs it. For cyber there's no such physical gate, so the last mile is defense itself: patching, sharing the holes that get found, and catching intrusions. Knowing how an attack works isn't the same as pulling it off against a target that's actually defended.
That difference should change how we treat the two. For biology, the model mostly lowers the knowledge barrier, but the physical world still has natural gates — materials, equipment, the synthesis step — so the most dependable fixes are downstream, off the model. For cyber, the raw materials are everywhere and there's no physical gate, so what the model adds is speed and the ability to act on its own — which puts the controls that count at the point where it acts, not where it talks. One model-centered playbook for both is the lab's instinct, because it's the playbook they own. It's the wrong instinct.
First, the principle that should govern all of it is the one every security person already lives by: defense in depth. No single layer is the gate. You assume any one control will fail, and you make sure another is standing behind it. That's my point in laying the chain out above — it shows the model company is one control among many, not the control.
Second, someone outside the seller has to be able to check the work. Independent testing, so a company isn't grading its own homework. Real transparency about what the safeguards are and how well they hold, so the rest of us can see where the gates are. And a way to act after the fact when something slips through. None of that depends on trusting a company's good intentions, which is the only kind of system worth building. The company in this story even said the right words — it wants oversight that's clear, fair, and based on facts. It just broke that rule with its hidden limit the same week the government broke it with a rushed ban.
So my recommendation is simple to say and hard to do: don't let the product companies define this moment. Not where we step in, and not who decides whether it's enough. The intervention they'll point to is the one inside the model, judged by them, because that's the version that leaves them in charge. But the real answer is spread across chipmakers, cloud providers, governments, the labs, the developers who build on top, the companies that deploy, the DNA-synthesis firms, the insurers, and the whole community of defenders. The decision is already shared out across all of them. The only question is whether we admit it, or let the people selling the product convince us they're the only ones who can hold the line. We'll only reach a mature solution — one that acts holistically — if government drives it, with less bickering and hubris on both sides and a real commitment to partnership between the public and private sectors.
