AI Explained: What Does "Formally Verified" Mean? Lean, Proofs and Why a 6-Million-Line Check Matters
This week Google DeepMind announced AlphaProof Code. According to AI Weekly's summary of the announcement, AI agents wrote more than 6 million lines of a formal proof language called Lean in about four weeks to mathematically verify selected cryptographic guarantees in the Signal messaging protocol's real-world shipping code.
The AI agents flagged only about 1% of their own generated proofs for human experts to review, and DeepMind's team reported that when they deliberately re-introduced two previously fixed bugs into the repository, the system caught both. To see why this marks a milestone in computer science, you need to understand the fundamental difference between testing software and mathematically proving it.
1. Testing vs. Proving: Why Example Inputs Aren't Enough
In traditional software engineering, testing runs a program on specific sample inputs and observes the output. As computer scientist Edsger Dijkstra famously observed, "Testing can show the presence of bugs, but never their absence." If you test an absolute-value function on 10,000 numbers, the 10,001st input (such as integer minimum overflow) might still crash the application.
Formal verification, by contrast, constructs a rigorous mathematical proof showing that a piece of software satisfies a precise logical property for every possible input that could ever exist. It represents the monumental difference between:
2. The Four Parts of a Formal Verification
- Specification: A precise mathematical statement describing what the code must guarantee. For example: "For all valid cryptographic keys k and messages m, decrypt(k, encrypt(k, m)) equals m."
- Implementation: The actual source code executed in production (e.g. C, Rust, or assembly).
- Proof: A rigorous chain of deductive logical steps proving that the implementation strictly satisfies the specification across all states.
- Proof Assistant (such as Lean 4): Specialized compiler software that verifies every logical step mechanically. Its core verification engine, known as the kernel, is compact enough that human mathematicians can inspect and trust its foundational axioms.
In the example above, Lean accepts the theorem only if the logical simplification holds for every possible natural number n across the entire number line.
3. Why AI and Proof Assistants Fit So Well Together
Proving properties about commercial software systems requires writing millions of tedious, excruciatingly detailed logical steps. That represents the exact kind of high-volume, structured synthesis that modern LLM agent teams excel at.
Most importantly, this architecture solves the AI hallucination problem through a concept known as untrusted generator, trusted checker:
AI agents brainstorm, iterate, search, and output vast candidate Lean proofs. The AI itself does not need to be trusted.
The Lean compiler checks every single deduction step. If a model hallucinates a false step or makes an invalid logical leap, the kernel instantly rejects it.
If Lean's kernel approves the proof, it is mathematically verified regardless of whether it was authored by a Fields Medalist or an automated neural network.
4. What Formal Verification Does NOT Guarantee
While formal verification provides the highest standard of software assurance in existence, it is not magic. Engineers must remain vigilant regarding four critical limitations:
5. Quick Reference Glossary
6. Check Your Understanding
Because testing only exercises the finite set of scenarios and inputs that you try. An untested corner case can always exist.
Because Lean's proof-checking kernel verifies every logical step mechanically. Hallucinations and invalid deductions are rejected outright.
A flawed or incomplete specification. If the mathematical statement of what the code should do is wrong, the code will flawlessly execute the wrong behavior.
7. Try It Yourself: The 30-Minute Natural Number Game
- Search for the Natural Number Game created by the Lean academic community (a free, interactive in-browser game teaching formal proof tactics).
- Complete Level 1 and observe how the Lean theorem prover displays your current proof goal and the exact obligations remaining.
- Intentionally introduce an incorrect proof step (e.g. invalid rewrite) and read how the compiler rejects the deduction.
- Write down one property formal verification guarantees and one limitation it cannot protect against.
8. A Note for Your Career & The Key Takeaway
Formal methods are rapidly transitioning from academic curiosity to mandatory industry infrastructure across cybersecurity, aerospace avionics, cryptographic protocols, semiconductor design, and automated smart contracts.
As AI models make writing millions of lines of Lean economically viable, the bottleneck is shifting. The most valuable skill of the next decade is not memorizing syntax, but mastering mathematical logic and precise specification writing.