On Tuesday, an Anthropic researcher resigned and said the people building AI earnestly believe it could kill us all by the end of the decade.
Anthropic’s own alignment lead responded that he puts the odds above ten percent, while adding that he considers the risk from today’s models low. His worry is superintelligence that improves itself.
That distinction is the whole ballgame, and almost everyone will miss it.
The AI in your organization today is dangerous in an ordinary way. Autonomous agents with real credentials, write access, and thin operator training can delete things, move money, and act on bad inferences faster than anyone catches it. These are the questions I write and speak about. I also participate in collaborative workgroups with the Council of Chief State School Officers’ Community of Innovation. This is not abstract to me. It means agents near student records and eligibility decisions, in organizations not staffed to supervise them.
I expect serious damage. Serious, not existential.
And it is the kind of failure we already know how to govern. Bounded uses. A named owner. Review that matches the consequence. A stop button that works. None of that is exotic. Most organizations simply have not built it yet.
What Coxon describes is a different category, and the shape of it is familiar.
In June 1945, a committee of Manhattan Project scientists warned that using the bomb on cities without warning would precipitate an arms race and prejudice any chance of international control. They were right on both counts. The report was seized by the Army, classified, and never reached the President. A petition from about seventy scientists weeks later never reached him either. It stayed classified until 1961.
Then came the horrible demonstration.
Hiroshima on August 6, Nagasaki three days later. The treaty that finally set the terms did not enter into force until March 1970. Nearly twenty-five years.
Now hold that next to last week. On September 2, at a G20 meeting in Chapel Hill, US officials said all twenty members including China endorsed a framework urging governments to avoid standing up AI-specific regulators and to reserve new rules for genuinely novel situations. Non-binding. No fines. No deadlines. Six days later, Coxon resigned.
And the nuclear regime had grips this one will never have. Fissile material is scarce and countable. Inspectors walk into buildings. That is written into the treaty. Model weights are a file. Compute is a rental agreement. Nobody walks into a data center and counts the superintelligence.
That is not an argument for panic. It is an argument for two things at once. Govern what you have, seriously, this quarter. And stop assuming the adults are handling the rest.