Automated Assurance: Anthropic's Claude Generates Proof of Code Safety

A new push by Anthropic to automatically generate mathematical proofs of software correctness is raising questions about the cost and feasibility of guaranteeing that code is safe. The company's Frontier Red Team, in a collaboration with partner Theorem, used its Claude model to mathematically verify existing software, a process known as formal verification. The goal is to make this level of rigorous checking, which has traditionally been prohibitively expensive and reserved for critical systems, a routine part of software development.

Formal verification has long been considered too costly for widespread use. For example, a team of specialists required 20 person-years to formally verify a small operating system kernel, a level of effort that has confined the technique to high-stakes applications like avionics software. Anthropic's approach aims to change that by automating the proof generation process.

Working with test programs from 123 open-source libraries, Claude produced roughly 180 million lines of mathematical proof code in just a few weeks. The model translated crash tests into mathematical problems, analyzed the compiled machine code, and generated proofs of its correctness. An independent verification program called Rocq then automatically checks those proofs, a process the researchers describe as fully autonomous. While generating these proofs, Claude identified and fixed hundreds of bugs that conventional testing had missed. In one case involving a widely used PDF reader, the effort uncovered as many security bugs in a week as are typically reported for that software in a year.

Anthropic acknowledges significant limits to this method. The proofs are only valid under specific conditions, including a particular processor architecture, single-threaded programs, and inputs under 64 MB, which does not reflect how these libraries typically run. The compute cost is also steep, with some proofs running up hundreds of millions of tokens, and the company did not report total project costs. However, the company notes that a smaller test run showed proofs can be reused, and newer models are becoming more efficient at generating them.

If AI can write code and prove its safety, the economics of software development could shift. Anthropic expects formal verification to become as practical as standard testing, and Theorem's roadmap already targets critical infrastructure like glibc, OpenSSL, and cURL. The question for the industry is moving from whether AI can program to whether it can guarantee the code it writes is safe under defined conditions.