Anthropic researcher resigns, warns self-improving AI is 'gambling with our lives'

Anthropic researcher Jacob Coxon has resigned, warning that the race to build self-improving AI risks catastrophe. In a public statement, he said labs are "gambling with our lives" by pushing systems that can rewrite their own code without agreed-upon safety limits. Coxon is the latest in a string of high-profile departures from major AI companies over existential risk concerns.

Coxon, who worked on alignment at Anthropic, argued that current technical safeguards are insufficient against what he calls "recursive self-improvement"—AI that improves itself faster than humans can audit. He specifically called for "pacing agreements" between leading labs, a proposal that would commit them to slow releases or capability thresholds when risks spike. The idea has circulated in safety circles for years but has gained no formal traction among competitors.

His resignation follows a pattern of internal dissent at frontier labs. Former OpenAI researchers have publicly criticized that company's safety culture, and several Google DeepMind alumni have signed open letters demanding stricter oversight. Coxon's move is notable because Anthropic has positioned itself as the safety-first lab, often citing its 'responsible scaling' policy as a model for the industry.

Anthropic did not dispute Coxon's account but emphasized that it has "paused training of frontier models" when risks exceeded set thresholds, a practice it says is not common across the industry. The company also stated it supports "broader coordination" but declined to commit to any binding external pace agreement.

The challenge is that pacing agreements require trust. Labs are locked in a commercial race, and any unilateral slowdown risks losing market share to rivals—like OpenAI, Google, or Meta—who may not follow suit. "It's a classic prisoner's dilemma," said one AI policy researcher who asked not to be named. "Until there's a credible enforcement mechanism, talk of pacing is academic."

Coxon's exit may not change lab behavior, but it adds pressure on regulators who are already scrutinizing frontier AI. The EU's AI Act and US executive orders have focused on transparency and testing, not on limiting self-improvement. If more researchers resign with similar warnings, the debate will shift from whether to slow down to how, and that is where pacing agreements might find a foothold.

For now, Coxon says he will work with advocacy groups to push for binding pacts. His message to lab leaders: "You don't get a second chance with extinction." Whether that lands in boardrooms remains the open question.