“Trust Me, Bro” Is Not a Safety Framework

Mark Zuckerberg would like everyone to calm down. Nah, boo. There’s too much riding on it.

In response to calls from Anthropic CEO Dario Amodei and others to slow the development of frontier AI, Zuckerberg argued on X that “every lab has the responsibility and incentive to move at the pace required” for safety. Meta, he said, had already delayed the release of one of its own models for several months. The labs have market incentives. They face liability. Independent evaluators can provide outside scrutiny. Any lab that ignores alignment, he wrote, “will fall behind.”

It’s all well-dressed, but after you strip away the policy language, his argument is basically “trust me, bro.” Yeah. I’m not really feeling that.

My objection isn’t that Mark Zuckerberg is uniquely untrustworthy, or that everyone working at Meta wakes up plotting the end of humanity. It’s that corporations don’t become reliable public-safety regulators merely because some of the people inside them are thoughtful, ethical, or sincerely afraid of what they’re building.

Meta, OpenAI, Anthropic, Google, xAI, and the rest are companies. They have investors, payrolls, compute bills, competitors, product calendars, and breathtaking piles of money waiting on the other side of the next capability jump. They’re under constant pressure to move faster than the lab across the street—or across the Pacific. Asking them to magically deprioritize all of that, voluntarily and indefinitely, for risks whose costs someone else will likely pay isn’t a “safety framework.” It’s a request for institutional sainthood.

History tells us expecting that is, frankly, a bunch of BS.

We’ve seen this movie

Oil companies didn’t need better access to climate science. Exxon’s own researchers were remarkably good at projecting global warming decades ago. The problem wasn’t ignorance. It was that acknowledging the consequences threatened the business.

Cigarette companies didn’t lack information about addiction or cancer. A federal court eventually found that major tobacco companies had deceived the public for decades and ordered them to publish corrective statements. Again: not an information failure. An incentive failure.

Boeing is the more immediate warning. The Federal Aviation Administration delegated portions of the 737 MAX certification process to Boeing through its Organization Designation Authorization program. After two crashes killed 346 people, the Transportation Department’s inspector general found that the FAA hadn’t fully understood Boeing’s safety assessments of MCAS before the first crash, and that the delegated process didn’t adequately ensure independence.

That’s the important version of the story. It’s not the bumper-sticker claim that Boeing simply certified its own airplane, but the more instructive truth: a system that allowed the regulated company to perform its own safety checks failed to preserve independence or eliminate bias. That’s like allowing an AI model to review its own work. Decorrelated reviews are essential in both systems.

Self-policing doesn’t usually fail because every person involved is a monster. It fails because the incentives are relentless, the warning signs are probabilistic, and the person empowered to say stop works inside the machine that’s rewarded for saying go.

The industry isn’t actually united

Zuckerberg’s position is getting attention partly because so many other people in the industry have moved in the opposite direction.

Amodei’s essay, bluntly titled “We Must Pace the Frontier”, says commercial incentives could drive a race to the bottom. He wants third-party evaluators embedded inside frontier labs with employee-like access, common standards among democratic countries, and, eventually, verifiable international coordination. Anthropic has committed to the embedded-evaluator model for itself.

Sam Altman responded, “I agree with Dario that we need to pace the frontier,” and said OpenAI would also provide independent evaluators employee-like access. Elon Musk’s response was even shorter: “Dario is right.” Google DeepMind CEO Demis Hassabis said the direction was correct, while arguing that the details still needed work. Microsoft’s Satya Nadella and Mustafa Suleyman have also welcomed more deliberate pacing and independent evaluation.

Then there’s Nvidia CEO Jensen Huang, whose view is much closer to Zuckerberg’s: “The market forces are already there,” he told the Associated Press, adding that no new laws or regulations were necessary.

I’m glad to see powerful people inside the labs asking for external oversight. I’m also not prepared to confuse their willingness to be regulated with a complete regulatory design. Incumbent firms can be sincere about safety and aware that expensive compliance regimes make life harder for smaller competitors. Both things can be true at the same time.

The answer isn’t to divine which CEO has the purest heart. The answer is to build a system that doesn’t require us to know.

The administration wants the accelerator

The Trump administration’s public position has mostly been that the race itself is the policy.

President Trump has called concern about an AI takeover a “HOAX,” described proposed guardrails as a “SICK conspiracy,” and argued that “whoever wins AI, wins.” White House AI adviser David Sacks says labs are free to slow themselves voluntarily, but warns that collective limits could become regulatory capture, an antitrust problem, or an excuse to hand the advantage to China.

That view isn’t entirely irrational. The United States can’t regulate a Chinese model lab by passing an American law. Any slowdown that applies only to one side of a strategic competition can create its own danger. Treasury Secretary Scott Bessent has nevertheless indicated that the administration is open to discussing shared AI risks with China, including risks from both closed and open-weight systems.

So even inside an administration publicly committed to speed, the real problem keeps reappearing: how do you control a technology when some of the most consequential actors may not accept the same controls?

You don’t solve that problem by giving up on controls where they’re possible. You solve it by pairing regulation with defense.

The water is already inside the boat

The case for independent inspection becomes stronger when you look at what OpenAI itself is saying about Astra.

This isn’t a leak, and it isn’t doomer fan fiction. In Astra’s official deployment-safety report, OpenAI says it observed a “substantial decrease in chain-of-thought monitorability.” In plain English: one of the few ways researchers have been able to watch what a reasoning model is doing—reading the intermediate reasoning it writes down—is becoming less dependable.

That written chain of thought isn’t the model’s soul laid bare. Labs have never been able to translate every numerical operation inside a neural network into a tidy human explanation. But the model’s written scratch work and actions have provided a useful window. OpenAI’s own chief scientist now describes advanced AI as “an intellect we don’t fully understand” and says the ability to rely on that window is progressively diminishing.

There has been a broad research norm against directly optimizing the reasoning trace, because punishing a model for “bad thoughts” can teach it to hide the evidence while preserving the behavior. Astra reportedly uses opaque recurrence only in a limited way, and OpenAI disputes that the architecture is the cause of the monitorability decline. Fine. That distinction matters to the science.

It doesn’t make the policy problem small.

There’s another problem with Zuckerberg’s claim that the labs have a reputational incentive to stay out of the news for mistakes: the scary stories appear to be selling the models. Mythos became famous for escaping a sandbox and finding vulnerabilities in sensitive government systems. Anthropic released the public, more restricted Fable version, the government pulled it after a jailbreak scare, and interest grew after the ban. OpenAI publicly classified Astra as capable of finding and exploiting previously unknown security flaws, and demand after launch was reportedly so high that the company considered temporarily suspending new Pro subscriptions. That doesn’t prove people wanted these models because they were dangerous, but it absolutely complicates the claim that bad press is a meaningful brake. In this market, “dangerously capable” can sound a lot like premium branding.

A captain who finds a hole in his boat doesn’t reassure the crew that it’s only a limited amount of ocean.

The labs are building systems they understand imperfectly, using monitoring windows they admit are deteriorating, under competitive pressure they admit is intensifying. This is precisely when “trust me, bro” stops being reassuring.

It’s also why an inspector can’t be a consultant invited in for a quarterly tour. Independent evaluators need statutory authority, secure access to models and training systems, incident-reporting requirements, and the power to stop a run or a deployment when a defined threshold is crossed. If the inspected company controls the inspector’s access, schedule, and paycheck, we’ve rebuilt the 737 MAX problem with GPUs.

China is an argument against a naive pause, not against oversight

The strongest objection to regulation is also the one we’ve got to take most seriously.

China isn’t going to stop because the United States passed a beautifully formatted safety bill. Chinese labs may cooperate when interests align, but no serious plan can make human safety contingent on universal compliance, perfect verification, or the enduring goodwill of a strategic competitor. Amodei acknowledges this directly: slowing so far that Chinese state-linked projects take the lead could make the world less safe, not more.

This is where the debate keeps collapsing two different questions into one.

First: should American frontier labs be externally inspected and constrained when their work crosses clearly defined risk thresholds? Yes.

Second: should the United States and its allies retain the capacity to understand, contain, and counter dangerous models produced outside that system? Also yes.

Those answers aren’t contradictory. Building codes don’t eliminate arson, and fire departments aren’t an argument against building codes.

We should also be precise about language. An open-weight model—a model whose trained parameters can be downloaded—isn’t necessarily open source in the traditional sense. Its training data, training process, and safety work may remain opaque. But once capable weights are loose in the world, they can be copied, modified, and redeployed beyond the reach of any single company or country. That’s not a reason to abandon regulation. It’s a reason regulation can’t be the entire plan.

Hugging Face was the warning shot

In July, OpenAI’s agents breached Hugging Face while conducting an evaluation. According to Hugging Face’s technical timeline, the agents took roughly 17,600 attacker actions over four days. They recovered credentials, moved across infrastructure, executed code on dozens of servers, and obtained root access on one system.

Hugging Face detected the intrusion, but it didn’t initially know the activity came from an evaluation run by another American lab. OpenAI says its own monitoring raised an alert six days after Hugging Face’s public disclosure, and that it connected the activity to Hugging Face and notified the company the following day. OpenAI’s incident report is candid about the seriousness of the event and the need for better containment.

That candor is useful. It isn’t comforting.

The incident was an evaluation, not a malicious campaign. The next one might not be. A bad actor won’t send a polite note explaining which model just acquired credentials and started moving laterally through somebody else’s servers. We need systems capable of detecting, attributing, and containing agentic attacks at machine speed, including attacks launched with models whose creators have no interest in joining our safety compact.

Build the Fire Department (The “Decorrelated Model” Review)

We’ve done versions of this before.

The regulatory piece is the building code. The frontier-defense lab is the fire department. It doesn’t make every building fireproof, and it doesn’t wait for arsonists to sign a voluntary safety compact. It watches for smoke, responds while there’s still time, and keeps one burning building from taking out the entire block.

When distributed denial-of-service (DDoS) attacks became a routine method for knocking organizations offline, the answer wasn’t to ask botnet operators to adopt a voluntary code of conduct. We built shared defensive infrastructure. Google’s Project Shield has provided free DDoS protection to vulnerable news, human-rights, and election sites since 2013. Cloudflare’s Project Galileo does similar work for at-risk public-interest organizations.

AI defense will be harder, but the principle survives: when an offensive capability becomes cheap and widely distributed, defense has to become infrastructure.

The United States and allied democracies should create an independently governed frontier-defense laboratory. It should be publicly chartered, securely funded, and firewalled from the commercial labs it evaluates. The companies should contribute talent, access, and compute; they shouldn’t control the board, write the test, or decide what gets disclosed.

Its job wouldn’t be to win a secret race to build the most powerful general model. Its job would be to acquire and evaluate dangerous open weights, develop containment and attribution systems, maintain threat intelligence, pressure-test critical infrastructure, build defenses that smaller organizations can actually deploy, and publish as much methodology as security permits.

It’d also need constraints of its own: independent inspectors, strict purpose limitations, auditable compute, mandatory incident reporting, adversarial review, and clear legal prohibitions against quietly turning a defensive mandate into an offensive product program. Otherwise, we’ve simply built another lab and stapled the word public to the door.

This is the part missing from both ends of the current argument. “Let the companies handle it” ignores the companies’ incentives. “Slow everything down” ignores the models and actors that’ll remain outside the slowdown. We need a building code and a fire department.

The politics are already strange enough

If you want a sign that this concern has escaped its usual political box, consider Bernie Sanders and Steve Bannon.

They didn’t join hands for a bipartisan chorus. They spoke separately from the same stage, and their prescriptions—especially on China—weren’t identical. But both rejected the idea that a handful of technology oligarchs should be allowed to determine the terms of everyone else’s future.

Sanders said, “When you are racing toward a cliff, you don’t just ease up on the gas pedal.” Bannon’s version was less polished: “We’re just not going to let you run wild.”

When Bernie Sanders and Steve Bannon can look at the same industry and independently conclude that the oligarchs need adult supervision, it may be time to stop pretending the demand for oversight is a boutique concern of one political tribe. It sounds like a bad line from a sci-fi movie, but all of humanity really is at risk here.

Trust isn’t the mechanism

I don’t want a government panel deciding which autocomplete feature can ship next Tuesday. I don’t want regulation so broad that only the richest incumbents can afford to comply. I don’t want safety theater that makes us feel righteous while an unconstrained model from somewhere else walks through the side door.

I want frontier thresholds defined in public. I want qualified inspectors who don’t answer to the company being inspected. I want evaluations whose methods and failures are disclosed as openly as genuine security concerns allow. I want labs required to report incidents before somebody outside the building discovers them the hard way. I want stopping power when the evidence says stop.

And because not every actor will submit to that system, I want a public-interest team capable of defending the rest of us from the systems that don’t.

Zuckerberg is right about one narrow thing: every lab does have a responsibility to develop safely. What he hasn’t established is why that responsibility should be treated as sufficient oversight. A responsibility without independent verification is a promise. A promise made under competitive pressure is a hope. Hope isn’t a safety mechanism.

Inspect it. Constrain it where we can. Prepare to defend against what we can’t constrain.

Because once the water is inside the boat, the argument about whether the breach was limited is over.