Getting the (In)formal Statement Right

It's clear to everyone now that LLMs have changed how mathematics gets done, and other fields are close behind. OpenAI's model proved finite-time blowup for the Navier–Stokes equations with a smooth, compactly supported force, and as Quanta reported, another model formalized the proof in Lean 17 hours later. Anthropic's model wrote about 13 million lines of Lean for Fermat's Last Theorem in 11 days. LLM agents have verified an operating-system kernel, production cryptography, a compiler and Ethereum bytecode.

What excites me most about all of this is that we're getting closer to a world where much more of mathematics can be formalized, checked, and built upon. I find the progress remarkable. We can now generate proofs at a scale that would have been difficult to imagine even a few years ago, what Mistral calls proof abundance, and Lean gives us a way to verify those proofs without having to read every line. Kim Morrison pointed out back in 2024 that autoformalization is on the critical path, and I think the recent progress, along with the case for formalizing all of mathematics, has made that increasingly clear.

But there's a problem that I think deserves much more attention. How do we know that the statement being proved is actually the statement we intended to formalize? A proof can be completely correct in Lean while establishing something weaker, different, or even unrelated to the original mathematical claim. And when we translate that formal result back into natural language, it's surprisingly easy to lose the distinction again. For me, this is one of the most important challenges in autoformalization today. I want us to get to a point where we can trust not only the proof, but also the mathematical meaning of what has been formalized. That's two goals: correctness, knowing that a result is true and which result it is, and understanding, a person being able to read the statement, see what was proved and why, and use it.

Here is a small example I found in openai/math, the release where a model wrote hundreds of research papers and put a Lean challenge next to each headline. One paper is about Yau's conjecture on nodal sets. The nodal set of a Laplace eigenfunction is its zero set, and Yau conjectured that on a closed n-dimensional manifold, the (n−1)-dimensional measure of the nodal set of an eigenfunction with eigenvalue λ stays between two constant multiples of √λ. The paper works on S4 × S1, where that measure is the Hausdorff measure H4, and it constructs a smooth metric there whose eigenfunctions beat the upper bound by a fixed power: H4/λ1/2+ε0 → ∞ for some ε0 > 0. And the release ships a Lean proof of its challenge.

The challenge asks for less: it requires only H4/√λ → ∞, so any violation of the bound counts. I like to see a gap like this on a concrete sequence, and √λ log λ does the job. It beats √λ by a factor of log λ that keeps growing, so it meets the challenge, but it falls behind λ1/2+ε0 for every fixed ε0 > 0, so it misses the paper's claim. The sequence is abstract and says nothing about real eigenfunctions: all it shows is that the two conditions differ. Here is the difference, with the nodal measures replaced by that sequence:

The paper's claim against the Lean challenge, as growth conditions. The sequence is abstract; it isn't claimed to come from real eigenfunctions.
Show the Lean
The paper's claim and the Lean challenge, as growth conditions
-- The Lean challenge asks for this, with h the nodal measures:
def LeanShape (h : ℕ → ℝ≥0∞) (lam : ℕ → ℝ) : Prop :=
  Tendsto (fun k ↦ h k / ENNReal.ofReal (Real.sqrt (lam k))) atTop (𝓝 ⊤)

-- The paper's Theorem 1.1 claims this, a fixed power ε₀ more:
def PaperShape (h : ℕ → ℝ≥0∞) (lam : ℕ → ℝ) : Prop :=
  ∃ ε₀ > 0, Tendsto (fun k ↦ h k / ENNReal.ofReal (lam k ^ ((1 : ℝ) / 2 + ε₀))) atTop (𝓝 ⊤)

-- A sequence that separates them: λ_k = k + 1, h_k = √λ_k · log λ_k.
def sibLam (k : ℕ) : ℝ := (k : ℝ) + 1

def sibH (k : ℕ) : ℝ≥0∞ := ENNReal.ofReal (Real.sqrt (sibLam k) * Real.log (sibLam k))

-- The challenge's condition holds for it...
theorem leanShape_sib : LeanShape sibH sibLam := by
  -- (eight lines: the ratio is log λ_k, which goes to infinity)

-- ...and the paper's fails, for every ε₀ > 0.
theorem not_paperShape_sib : ¬ PaperShape sibH sibLam := by
  -- (fifteen lines: log λ_k / λ_k^ε₀ tends to 0)

#print axioms OAI.Yau.Target.not_paperShape_sib
-- 'OAI.Yau.Target.not_paperShape_sib' depends on axioms: [propext, Classical.choice, Quot.sound]

What I found interesting here wasn't a mistake in the proof. The proof establishes exactly what the Lean statement asks for.1 The issue is that the Lean challenge and the theorem in the paper make different mathematical claims. This is precisely why I think semantic fidelity matters so much in autoformalization. We can have a perfectly valid proof, checked by Lean, and still end up communicating a result that hasn't been formally established. As more of these results are generated automatically, I think we need to become much more careful about this distinction.

I've also been following many of the autoformalization discussions on Lean Zulip, and one thing I've appreciated about the community is how seriously people take the distinction between a checked proof and a faithful mathematical statement. The discussions around Meta's Atlas are a particularly interesting example. Atlas was reported as more than 45,000 declarations formalized from 26 textbooks. Formalizing tens of thousands of declarations is an impressive achievement, but it also raises a much harder question: how do we know that those declarations faithfully represent the original mathematics? This is not something we can settle by counting successful Lean compilations. One reader wrote of a book listed as 56 percent formalized that "the reality is that 0% of the statements here are accurately formalized", a post later quoted by Bastounis, Circelli and Hansen, and the complaint was about the statements, not the proofs: most of them had been checked only by other models.

When an AI system says it has formally proved a result, I want to know what to check before trusting it, and I keep the jobs apart. The proof has to prove the statement, the statement has to mean what the source meant, and the natural-language version has to say what the statement says. Lean does the first one. Today the other two are mostly done by people, though I expect agents to take on more and more of them. As they do, I think the human job changes from checking each statement to deciding what's worth formalizing, which definitions are the right ones, and what a result means. For understanding there are two more questions: whether the decomposition keeps the structure of the argument instead of giving one opaque certificate, and whether the abstractions are meaningful, general enough and compatible with the rest of mathematics. A successful kernel check answers only the first.

A doodle: a paper on the left, the Lean statement in the middle, a headline speech bubble on the right, and a stack of proof pages below the statement. The arrows from the paper and to the headline are dashed and labelled "no kernel check"; the arrow from the proof is green and labelled "Lean checks".
Three distinct checks: proof correctness, autoformalization fidelity, and autoinformalization fidelity.

Six checks before announcing a result

Over time, and especially from reading the community discussions and looking through generated formalizations myself, I've started using a few checks that I find particularly helpful. Most of these ideas come from people in the Lean community, and I think they're worth making more systematic as autoformalization becomes more common. Here are six things I look for when evaluating a formal result. Today I run most of them myself, with agents helping. As models get better at these semantic questions, I expect agents to run more of them, and I'd like the list to work as a specification for those agents as much as a habit for people.

  1. Freeze the statement, then run Comparator. Before anyone proves anything, write the theorem into a challenge file with sorry for its proof, together with every definition it uses, and leave no definition open. Pin Lean and Mathlib. When the proof comes back, check it against that file with lake comparator, which builds the proof in a sandbox and replays it through independent checkers. That establishes that the proof proves the frozen statement under the axioms you allowed. It can't establish that the frozen statement faithfully represents the informal mathematics, and the next checks are for that.
  2. No extra axioms. The proof may use propext, Classical.choice and Quot.sound and nothing else: no sorry, no axiom, no native_decide. Read the list off a replay of the proof, not off the source. If you need an assumption, make it a hypothesis, where it shows in the statement.
  3. Have an agent read the statement, then a person. Give both the elaborated statement, the definitions it uses and the exact source, and give neither the proof. Write down every place where the Lean and the source disagree.
  4. Try to break it. Check that some input satisfies all the hypotheses together, and try each new definition on an example you already understand. For code, put the bug you care about next to the theorem and check that the statement rules it out.
  5. Autoinformalization: write the natural language from the Lean. The headline says what the statement says, no more. If the announcement claims something, there should be a Lean statement behind it, and where the Lean falls short of the paper, say so right there.
  6. Record what the review covered. Write down who read what: which commit, which definitions and hypotheses, and what they skipped. If the statement changes, or a definition under it does, that review is stale and somebody has to look again.

What a checked proof guarantees

Proofs can come from anyone because of how Lean is built: a proof is a term, its statement is the term's type, and the kernel checks the one against the other and never asks where the term came from. Once we've fixed the statement and its definitions, a proof from a graduate student, a language model or an alien gets the same treatment. Lean builds this in: any two proofs of the same proposition are equal by definition, so example (p : Prop) (h₁ h₂ : p) : h₁ = h₂ := rfl goes through. Jason Rute has made the same point on Zulip: once the statement is separated from the proof, the "referee" only needs to check "the statement and definitions used in the statement".

I rely on Comparator to turn this into a protocol, and since Lean 4.35 it ships with Lake as lake comparator. I write a challenge file that states each theorem with sorry and name the axioms I allow. Comparator builds the untrusted solution in a sandbox and checks that each theorem has my statement, down to every constant it mentions, uses only those axioms, and is accepted by the kernel. With --paranoid it also replays everything through five more checkers, among them con-leche, whose verified mode is proved in Lean never to accept a theorem of False.

A doodle: the challenge box on the left, read by a stick figure; Comparator in the middle with three ticks for same statement, only those axioms and kernel accepts; on the right, a student, a robot and an alien inside a dashed box labelled anyone, never read, with a stack of proof pages.
People read the challenge. Anyone can send the proof.

Lean's own manual calls this the gold standard for "un-reviewed AI-generated proofs and programs", and it says what it leaves to the reader, ending with "no human error or misleading presentation of the theorem statement in the trusted challenge file". No checker covers that condition, so it falls to a person, and to me that's where most of our attention should go.

One thing Comparator can't check for us is an open definition, one the challenge leaves as sorry for the solver to fill in. The Comparator docs give the example themselves: if the challenge is RiemannHypothesis ↔ ChallengeSolution with def ChallengeSolution : Prop := sorry, then def ChallengeSolution : Prop := RiemannHypothesis solves it by reflexivity. So when a challenge leaves a definition open, I read what the solver filled in myself.

The same rule, that a reader checks the statement and its definitions but never the proof, tells us what to read when theorem A depends on theorem B. If A's proof uses a proved B, the kernel checked B, so nobody needs to read it, however many million lines it took; if B is an axiom or rests on a sorry, the axiom check rejects it. If A takes B as a hypothesis, A is a conditional theorem and its headline has to say "assuming B". And if A's statement mentions B's definitions, those definitions are part of the statement and somebody has to read them. Axiom's CFSG is this third kind: it says every finite simple group is isomorphic to a group in Tau Ceti's list, so reading it means reading how the groups in that list are defined.

A doodle in three panels. Left: a theorem A box with a lemma B box inside it and a tick, captioned the kernel checked B, so nobody needs to read it. Middle: theorem A with the hypothesis (h : B) → ..., captioned conditional: the headline says assuming B. Right: theorem A's statement mentions B.Group, an arrow comes up from a sheet of B's definitions held by a stick figure, captioned A means only what they mean: read them too.
Three ways theorem A can depend on theorem B, and what each one asks of a reader.

Lean has two engines, one that checks proofs and one that compiles programs. On the checking side, the elaborator runs our tactics and automation, and none of that is trusted: it only proposes declarations. The kernel checks each one, a replay checks them all again, and every axiom is listed. The kernel is small, and small isn't perfect: Mario Carneiro said on Zulip that "there are bugs in it like any other software, but it is held to a high standard", and Jason Rute added that the independent checkers only reassure us if we run them.

The compiler is the other engine. It turns definitions into C that runs on Lean's runtime, and the kernel never sees any of it. Compilers and the runtime deserve a post of their own, and I'm writing one soon. Here one tactic matters.

A doodle with two lanes. Top: tactics and automation feed a proof term to the kernel, which accepts it and lists its axioms. Bottom: the compiler produces running code that the kernel never checks. A dashed orange arrow labelled native_decide, trust me, it's true, goes from the running code up into the kernel and enters as an axiom.
The kernel checks terms. Compiled code is never checked, and native_decide carries its answer across.

native_decide is the bridge between the two: it compiles a decision procedure, runs it, and since Lean 4.29 adds a fresh axiom saying the answer came back true. A proof that uses it trusts the compiler, and one mismatched @[implemented_by] is enough to prove False. This compiles on Lean 4.35 without Mathlib:

native_decide proving False
-- Compiled code runs `fast` wherever `slow` is called; the kernel sees `slow`.
def fast (_ : Nat) : Bool := false
@[implemented_by fast] def slow (_ : Nat) : Bool := true

theorem slow_true : slow 0 = true := rfl
theorem slow_false : slow 0 = false := by native_decide   -- runs `fast`
theorem oops : False := Bool.noConfusion (slow_true.symm.trans slow_false)

#print axioms oops
-- 'oops' depends on axioms: [slow_false._native.native_decide.ax_1_1]

I take three habits from this.

  • Do computations in the kernel. I stick to decide, decide +kernel, rfl and norm_num, which leave a term the kernel checks and add no axiom of their own. For a big search, I work out the answer outside Lean and check a certificate inside it. native_decide is fine while we're exploring, but I wouldn't ship it.
  • Keep assumptions as hypotheses. Shreyas Srinivas made this case on Zulip: an axiom isn't part of any theorem's statement and carries into every theorem that uses it, while a hypothesis sits in the statement, where the reader is already looking.
  • Check untrusted code outside the editor. Lean's metaprogramming can lie to us: a macro can change what False means, a statement can be swapped for True while the editor still shows the original, and an axiom can be added, used and hidden, as Derek Egolf's demos show. The source file can't be trusted for the axiom list either: a bad declaration added under debug.skipKernelTC can report no axioms at all, and agents have spelled the keyword "ax" ++ "iom" to slip past a text filter. The tricks are well understood and nobody honest uses them, so my rule is short: I don't check code I don't trust inside the build that ran it. I replay the exported proof with lean4checker --fresh and hand it to Comparator with a statement I wrote myself, as the Lean manual describes and Jason Rute recommends.
A hidden macro fools the editor's checkmark. Rebuilding in a sandbox, replaying with independent kernels and comparing against your own statement catches it.

Autoformalization

I think writing the right formal statement is often much harder than we give it credit for. When we're formalizing a well-known theorem, we usually have a mathematical statement that has been studied and refined for years. Even then, as the Formal Conjectures project and the work on Erdős problems have shown, getting every definition and hypothesis right can be surprisingly difficult. Boris Alexeev, writing about the formalized Erdős problems, found that "misformalization is a big issue and was more frequent than I had expected". Things become even more interesting when we're formalizing a new research paper or a result discovered by an LLM. There may be no agreed-upon formal statement to compare against, and deciding what the theorem should say requires understanding the underlying mathematics in the first place. The person best placed to make that call is usually the author, who is also the one hoping it's true, and most of openai/math is this kind.

Statements go wrong even in the benchmarks we use to measure autoformalization. When an audit at ICML 2026 went through variants of five standard Lean benchmarks, it raised 4,833 flags, and 398 of those were certified by machine as counterexamples, vacuous theorems or unsound axioms. If the statements we measure ourselves against can go this wrong, I don't trust my own without checking them. Here are the habits I use when I write a statement myself.

  • Start from the source as written, and record every change. If I fix a mistake, strengthen or weaken a hypothesis, or leave a part out, I say so right next to the Lean. Fixing the source is fine, and so is translating it as it stands, as long as a reader can tell which one I did. FormalFlow's proof of a core theorem behind MIP* = RE is a good model: it corrected side conditions in the published argument and said so.
  • Use Mathlib's definitions, and test the ones you add. If I get a definition wrong, every theorem about it means something else while every proof still goes through. So each new definition of mine gets three things: a lemma that pins it down, an example, and a bridge to the standard notion. Mathlib's own definitions need reading too: the version I use has no P, NP or SAT, and Turing.TM2ComputableInPolyTime is relative to whatever encodings the user supplies. With constant encodings, even the halting problem counts as polynomial-time.
  • Read the statement after elaboration. The statement Lean elaborates, after it fills in the implicit arguments, coercions and instances I left out, isn't always the one I thought I wrote. Lean and Mathlib make their operations total: Real.log 0 = 0, (2 : ℕ) - 3 = 0, and deriv (fun x : ℝ ↦ |x|) 0 = 0 even though |x| has no derivative at 0. Getting a value back doesn't mean the precondition held, so the statement has to say it did. In current Mathlib a bare ⊤ in ContDiff ℝ ⊤ f means analytic, not smooth, and autoImplicit turns a π without open Real into a variable. I turn autoImplicit off and look at every statement with #check @name.
  • Try to break it before anyone proves it. Provers will happily use contradictory hypotheses and never touch the mathematics I meant. So I look for one input that satisfies all the hypotheses at once, and I check each existential to see whether something trivial witnesses it. If the target is an open problem, I make the statement say which side holds, since P ∨ ¬P is Classical.em. VeriTile, a public Lean framework for Triton kernels, states two attention theorems under a hypothesis that fails for every positive shape, so they say nothing about any launch.
  • Don't trust equivalence checks once provers are strong. An equivalence check asks a prover to show that the statement we wrote and a reference statement imply each other. If we've proved P and we've proved Q, then we've proved P ↔ Q without relating them at all. Poiroux and colleagues warned about this, and BEq+ has since been measured accepting 94 to 96 percent of 50 unrelated provable pairs. One way out is to let the model search but keep the acceptance rule separate, as SemantEq does. Its auditor accepts a pair only when each side turns into the other through a few kinds of allowed step, checked on the proof term the kernel sees, and on the authors' certification set it accepted none of the 104 pairs judged non-equivalent. It still needs a reference we trust, since a mistake shared by both sides passes. My own habit is to turn a statement's fixed objects into parameters and try both readings on something simple, which is how the Yau gap at the top showed up.

Reading the statement

In my experience, LLM agents are good first readers, and I use them that way: I hand one the elaborated statement with every definition it reaches, put the paper it came from next to it, and keep the proof out of sight. Then I ask it three things: where do the two differ, is there an input where all the hypotheses hold, and can it refute the statement in Lean. A refutation that fails proves nothing, but one that succeeds settles the question.

A doodle: the source and the statement with its definitions go to a robot holding a magnifying glass, with three questions: where do they differ, can the hypotheses hold, can you refute it. A dashed arrow labelled leads, not verdicts goes to a stick figure reading on their own.
What I hand a reader agent, and what I ask it for.

Agents are optimistic judges, though: Herald's judge rated Herald's own miniF2F translations 97.5 percent faithful, while human evaluation put the figure at 62.7 percent. They can even notice a problem and wave it through: in the thread about Meta's Atlas, Charles Arnal showed a judge report that spotted a curvature formula proved only in an abstract ring and then called it "a valid algebraic abstraction". So I treat what an agent tells me as a lead, not a verdict, and I want the person to form their own reading first. A fresh agent that back-translates the statement into natural language without seeing the source, the way claimcheck does, gives a separate comparison.

Today, the person is the bottleneck. A useful reader has to know the mathematics well enough to spot a wrong hypothesis and Lean well enough to see what a coercion or a default value does to it, and in any given field there aren't many people who know both. Sven Krippendorf, Leopoldo Sarra and Joseph Tooby-Smith make the same point in a recent paper on LLMs and formal physics: "An argument generated in minutes may require days of expert scrutiny." That's why I spend the reader's hours on the statement and its definitions, and try to keep those short. As agents get better at this kind of reading, I expect the person's part to shrink to spot-checking the agents, settling the cases where they disagree, and deciding what counts as faithful in the first place.

The strongest check I've seen keeps the definitions out of the machine's hands: Morph Labs' Trinity wrote the probabilistic abc formalization, but Bhavik Mehta and Arend Mellendijk had written their own definitions "before any AI was involved". Bhavik then had Lean confirm that Trinity's radical agreed with Mathlib's on positive numbers and that its exceptional set agreed with theirs, closed his own statement by converting his definitions to Trinity's and applying its proof, and ran lean4checker on the result. In the end, the human part shrank to a small bridge, checked by Lean, between what they meant and what Trinity proved.

I also don't want "read the definitions" to mean reading a million lines. Banri Yanahama and Akiyoshi Sannai's Lean Compass works out which declarations in a project can change what a set of target theorems says: it drops dependencies that enter only through proofs and treats Mathlib as trusted. On three proof-heavy projects that cut the declarations by 94 to 99 percent on average, and that list is where I'd want a reader to start. The other question runs the opposite way: does every claim in the announcement have a Lean statement behind it? LeanTrail matches a paper's items to Lean and labels each one formalized, statement only, partial, blueprint only or missing. A model does the matching, so "formalized" still needs a reader, but a list like that tells me far more than one green badge.

Whatever the reader concludes, I want it written down in a way that can go out of date, because a review covers one version of a statement: these definitions and hypotheses, at this commit, read by this person. People in the Tau Ceti channel have been building this. In the prototype tools that came out of it, a review is tied to a hash of the declaration and of the definitions below it, and shows up as stale when either changes. Every record names the person or the agent behind it, and Rémy Degenne's point applies: "We should not present human validation and machine-generated audits on the same level." A hash tells us when to look again, not whether the first judgment was right.

Autoinformalization

We've spent a lot of effort thinking about how to translate informal mathematics into Lean, but I think we need to pay just as much attention to the reverse direction. Once we've proved something formally, how do we explain what was actually established? Most mathematicians are not going to inspect every Lean file behind an announcement, and they shouldn't have to. They should be able to understand the mathematical contribution from a faithful informal explanation. When I looked through OpenAI's math release in October, nearly 60 families of results, covering 76 of its 405 Lean challenges, had headlines that claimed more than their Lean statements. Agents compared every challenge's Lean statement with its headline and its paper, and a second, independent pass rechecked all 405. Seven of the gaps are major, like Hilbert's sixteenth problem below, and most of the rest come from one bookkeeping slip: a newer paper was added to a headline without the Lean being updated. I think this is why we need to take autoinformalization much more seriously. We're getting increasingly good at generating formal proofs, but we also need to get better at understanding and explaining what those proofs actually establish. Models are getting better at this direction too, and I expect checking a headline against its Lean to be one of the first jobs agents take over. One headline "resolves the uniform boundedness assertion in Hilbert's sixteenth problem", while the Lean covers quintic Liénard systems. Yet the same release also shows how to do it well: its SAT challenge proves (log log n)2 in the exponent where the paper has log log n, and its docs say so. I think this is exactly how such differences should be communicated. There's nothing wrong with formalizing a weaker result, provided we're clear about what has actually been proved.

When someone says they've formalized a paper, I always wonder what exactly that means. Did they verify the main theorem, formalize the original mathematical argument, or develop a reusable library of definitions and results? These are all valuable achievements, but they're not the same thing. I'd like every announcement, mine included, to say which one it means, and where its route left the paper's. OpenAI's Navier–Stokes blowup is a good example: its Lean follows different bounds from the paper, so it checks the theorem and not the paper's argument. Chris Henson found that distinction missing from the press coverage, which skipped "the nuance between the top-level statement being faithful versus a formalization following every line of a paper proof precisely".

Tags make the same kind of promise: Axiom's release of the Stacks project calls it a formalization of "the entire Stacks Project", and every declaration carries a Stacks tag linking it back to the text. Kim Morrison's post about it does what I'd want every reader to do, calling it "really awesome" and then reading the tags against the files. In one file, the theorem tagged with a final non-algebraicity statement can't name the stack that statement is about: its docstring says "this library has no Hom-stack of two algebraic stacks", so it takes a general stack and the two main inputs as hypotheses and proves the short implication between them. Its comments say so plainly, and Kim reported that every statement gap he found was flagged in the source. But the website's partial and conditional labels miss some of those gaps, so the tag promises more than the theorem gives.

A link can also say how close it is. The Mathlib Initiative's Literature Links, released on November 9, is a small JSON format that pairs a passage of a paper or a book with its Lean counterpart and names the relation: an exact match up to junk values like x / 0 = 0, a close match, a broader or narrower statement, part of a theorem, or a search that found nothing. Their Seifert–van Kampen companion, made with their formal-companion tool, lays the links over a proof from Tau Ceti, and I'm going to use the format in some of my own projects. I find this direction particularly promising because it makes the correspondence between informal and formal mathematics explicit. Of course, labeling a correspondence as exact doesn't establish that it's mathematically faithful, but it gives us something concrete to review and improve, which is what Tobi Lehman calls a faithfulness review.

Percentages are promises too, and Vivien Cabannes, one of Atlas's authors, explained in that thread that its evaluation is "non-transitive by design": a statement counted as formalized if its own proof didn't directly use sorry or unjustified axioms, even if a dependency did. I'd call that an honest answer, and it's a different gap from proving the wrong statement. In the same thread Jireh Loreaux asked for the shares to be split six ways: results already in Mathlib, results that needed only glue, new formalizations, new ones in a form others can use, statements that don't match, and failures. That's the breakdown I'd want next to any headline percentage.

Decomposition and abstraction quality

Comparator says nothing about the second goal, understanding. Kevin Buzzard asked on Zulip: if all we want is "a formal verification that a theorem is true", "then who cares about the art of proof?" I do. When correctness is all we're after, I'll grant that a certificate is enough and nobody has to read the proof, but I care about the proof as well as the statement. For understanding, someone has to be able to follow it: a proof that follows the paper checks the argument and not only the result, its lemmas are what other people reuse, and reading it is how we learn why the theorem is true.

That's why I think of five levels a formalization can aim at, roughly in order of the human time they take, with the first two about correctness and the other three about understanding. The ladder is my own and it's subjective, not a precise classification: the lines between levels are judgment calls, a project can sit between two of them or skip one, and other people would draw it differently. None of them is wrong, and each is judged on different things. Jack McCarthy said the same on Zulip: "We should stop treating "quality" as a monolith." But the trouble starts, I'd say, when a project is announced, or read, at a level it hasn't reached. That's an easy mistake for any of us to make, whether we're writing the announcement or reading it. Levels 3 and 4 are where decomposition and abstraction quality show: whether the formalization keeps the structure of the argument, and whether its definitions are ones other mathematics can use. Checking these will get more automated too, but choosing the abstractions, which definitions a library should be built on, is where I expect people's goals to sit, because it's a judgment about what the mathematics should look like.

A doodle of a five-step staircase: certificate, reviewed statement, readable project, reusable library and Mathlib, with correctness over the first two steps and understanding over the last three. A stick figure stands on step two, and a dashed orange arrow labelled Tau Ceti: skips 3? jumps from step two to step four.
Five levels, roughly in order of the human time they take. The ladder is a subjective sketch, not a precise classification.
  • Level 1, a certificate. One theorem, accepted by Comparator with the three standard axioms, and a proof nobody needs to read. The openai/math solutions are certificates for statements the same model wrote along with the papers, and AxiomProver's IMO 2026 run proved statements it had autoformalized itself. A certificate says the proof proves the statement, and nothing about whether the statement is the right one.
  • Level 2, a reviewed statement. The statement and its definitions have been read against the source, as in the checklist. Anthropic's FLT and Axiom's CFSG are both unread proofs of statements people wrote down first. FLT's challenge is Mathlib's own FermatLastTheorem, and its build fails unless #print axioms shows the three standard axioms, a check anyone can copy with #guard_msgs. CFSG proves Tau Ceti's ClassificationStatement, fixed in advance by a human-reviewed roadmap, though LLMs implemented the groups it lists. Formal Conjectures and lean-eval are collections of statements at this level, which is as far as the checklist gets us.
  • Level 3, a readable project. Someone can learn the mathematics from it. A blueprint, made with leanblueprint or Verso's, ties the informal argument to the Lean declarations and shows what is still open. Proofs are split into named lemmas that follow the argument, and the comments explain the idea rather than the tactic. PFR, Carleson and Buzzard's FLT project are built around blueprints.
  • Level 4, a reusable library. Other people build on it, which brings in the hard half of the review guide: do the results already exist, "possibly in a more general form with a different name", and does the code "provide a sensible API"? In practice it means definitions that fit Mathlib's hierarchy and an API that later work can use. CSLib for computer science and PhysLib for physics, both on Mathlib, aim here.
  • Level 5, Mathlib. Merged after maintainer review. The review guide orders its questions "approximately from easiest to hardest" as "style, documentation, location, improvements and library integration", and says the last one "generally requires several months of developing familiarity with mathlib". Mathlib also asks contributors to understand LLM-written code well enough to justify each design decision "without the use of an AI".

Whether a library is reusable can be tested on later work, and Timothy Chow proposed a nice way to do it on Zulip. The idea is to take a textbook with a lot of material that isn't in Mathlib and, say, a dozen papers that look like they'd be easier with that textbook formalized, and to formalize the papers without it. Then we formalize the textbook, redo the papers on top of it, and see whether the formalizations get notably smaller. He expects they will, but doesn't take it for granted. I'd trust that test over any count of declarations or lines. Krippendorf, Sarra and Tooby-Smith call the underlying idea "trust by reuse": the downstream uses of a definition, and the failure modes set up around it, are the evidence that it's the right one. But reuse only tests what later work uses. Violeta Hernández noted on Zulip in 2024 that "sometimes a "bad definition" can be relative": Mathlib's StrictOrder.cof wasn't the right cofinality for partial orders, but its API only used it for well-orders, so it hadn't caused trouble.

There's now a careful case study of why this is the hard part. Vasily Ilin and Brian Nugent wrote the statement of Grothendieck vanishing by hand with Mathlib's definitions and had an agent follow Hartshorne's proof, and after 9 active days it compiled with no sorries. Then one of them reviewed the code, and of the agent's 62 new definitions, one was done the right way: "The remaining 61 definitions were all done poorly and are not very useful for future formalizations." The theorem was fine, but the library work was still ahead, and that's the debt level 4 is about.

I find Tau Ceti interesting because it doesn't sit on this line. Humans write its roadmaps and review rubrics, LLMs write and review the code, and its README says it is "explicitly not aiming here at curating and digesting mathematical knowledge in the way that a human authored library like Mathlib can". It skips the human reading of level 3 and goes for the reuse of level 4, and Axiom's CFSG and some lean-eval problems already reuse its statements. Does reuse hold up when nobody reads the code? I don't know, and that's what makes it an experiment worth watching. Each level after the second costs more human time today, and that's the cost I most want better automated checks to bring down.

Beyond mathematics

I don't think any of this is special to mathematics. The checklist and the levels carry over as they are to theoretical computer science, physics and any field where a claim can be stated precisely, and the failures look the same everywhere: a quantifier in the wrong place, a hypothesis no input satisfies, a returned value read as if the precondition held.

Science adds a link that mathematics doesn't have. Krippendorf, Sarra and Tooby-Smith split a formal physics result into three checks: experiment checks whether the physical model describes the world in its regime, experts check whether the Lean says what the model says, and Lean checks only that the proof proves the statement. As their abstract puts it, "a checked proof does not establish that its definitions faithfully capture the intended physics". A faithful formalization of the wrong model is a theorem about the wrong world.

A doodle: a moving ball labelled the world, a dashed arrow labelled right regime? experiment checks, a box with the physical model E = one half m v squared for v much less than c, a dashed arrow labelled says the same? experts check, and an orange box with the Lean definition kineticEnergy p := one half m v squared and the theorem 0 ≤ kineticEnergy p. A green arrow from a stack of proof pages points up into the Lean box, labelled proves it? Lean checks, with a tick.
Science has one more link, after the kinetic-energy example of Krippendorf, Sarra and Tooby-Smith. Lean checks only the last one.

Formalizing physics also finds real mistakes. Tooby-Smith formalized a widely cited 2006 paper on the stability of the two-Higgs-doublet potential, and the formalization "revealed an error in the arguments of this 2006 paper, invalidating their main theorem". To the best of his knowledge it's the first non-trivial error in a physics paper found this way, and he asks "the uncomfortable question of how many physics papers would not pass this higher level of scrutiny".

Code adds a different link: the statement is about a model of the program (its semantics, its harness and the machine it runs on), and somebody has to show that the model matches what ships. I've run into a version of this myself. When the log_softmax bug was going around, Edward Yang pointed out that TorchLean had the same problem, and I agreed: our logSoftmax theorems were over ℝ, so they could never see a floating-point bug. An error bound isn't automatically the fix either, as Edward put it: if we prove "absolute error ≤ 10⁻⁷, well, that will admit the buggy implementation!" So now I want the bad example next to the theorem, with a reason the statement rules it out. The gap between the natural language and the Lean shows up here too. In a recent verification release, the LZO fuzz test checks only that decompression succeeds and returns the right length, and the release's notes for that test say so, but the announcement describes more. Code probably deserves a post of its own, so I'll stop here.

Working with agents

The best systems I know share a loop: an informal plan, a split into lemmas, and a lot of parallel proof search that reads Lean's error messages, from Aristotle's graph search to Maestro's coordinator, which can repair a false lemma from a counterexample Lean has checked. When I read one of their results, the first thing I look for is who wrote the statements. AlphaProof and Aristotle proved IMO problems whose Lean statements experts wrote by hand, while the PutnamBench entries proved benchmark statements, and Aleph fixed 28 of those along the way. I think the statements are where the difficulty is now. On MathAtlas, a statement benchmark built from 103 graduate textbooks and unrelated to Meta's Atlas, the paper's best baseline, ReForm 8B, writes a statement that compiles and that the CriticLean judge accepts as faithful 9.8 percent of the time. No proof is required for that number.

Agents also change what they were asked to prove when it gets hard. In the xv6 verification, an agent that couldn't prove a lemma once deleted a line from the kernel source and once added an axiom, when the real reason the lemma was unprovable was a bug in the code. And the agent that wrote the Axon compiler tried "unilaterally weakening the conclusion of the theorem". The cheating Jason Rute knew of was aimed at evaluations rather than at Lean's soundness: one model tried to look up the answers to a Lean benchmark, another tried to call stronger models for help. Chris Bailey wasn't aware of any recent LLM result that exploited a soundness bug.2

  • Separate writing from accepting. I'd rather the agent that writes a statement, a proof or a summary never be the one that signs off on it. I give provers the frozen challenge read-only, and the checks run somewhere they can't reach.
  • Give the prover a way out. If an agent can say a statement is false, and that answer is handled separately from a failed proof, it probably has less reason to cheat.
  • Report what you measured. Statement fidelity, proof completion and end-to-end success are three different numbers, so I'd report them separately, along with the budget and who wrote the statements. Then I'd test the gates on cases where we already know the answer: a weakened statement, a contradictory hypothesis, a hole, an injected axiom. If one of those gets through, we don't have a gate.
  • Credit the people who make it usable. Cleaning up generated code, reviewing it, putting it in the right place and keeping it building through upgrades is service work. I agree with Krippendorf, Sarra and Tooby-Smith that credit "should be attached mainly to quality, comprehension, and high-level understanding rather than to the volume of results". Releasing a big generated library starts that work rather than finishing it.

Tools and community resources

  • Tools for agents. For talking to Lean there are the Lean REPL, Pantograph, the Kimina Lean Server and, for agents that use the Model Context Protocol, lean-lsp-mcp. For searching Mathlib, I'd use the search engines LeanSearch, Loogle and LeanExplore.
  • Skills for coding agents. Cameron Freer's lean4-skills come with axiom checks and guardrails. Scott Armstrong's LeanAutoformalizationSkills have a stated goal I share, to prove "the theorem the mathematical source actually states", and Chris Birkbeck's mathlib-quality is for bringing agent-written code to Mathlib's standards. The Mathlib Initiative's formal-companion comes with a skill, companion-faithfulness, that audits a Lean companion against the book it formalizes: deterministic detectors settle what they can, agents read the rest, and an adversarial pass challenges what they report.
  • Setups worth reading. I'd point to two: mini-Lean-agent, which keeps the trusted evaluator apart from the prover, and SymCrypt's review of theorem statements before any proving.
  • Say what you're working on. The community's Project Intentions board records who intends to formalize what. Its point is "to help people find collaborators and avoid quietly duplicating each other's work", it asks for specific registrations with a credible expiry date, and it asks everyone to respect students' projects. It's a courtesy, not a reservation, and I'd check it before starting anything big.

Correctness and understanding

I'm genuinely excited about where AI for mathematics is heading. I think we're only beginning to see what becomes possible when proofs can be generated, formally checked, and reused at this scale. In almost every story here, Lean did its job, and what went wrong was the statement or the natural language around it. If we want this progress to translate into mathematical understanding, we need to take autoformalization just as seriously as proof generation. Getting the statement right, understanding the definitions behind it, and communicating what was actually proved are all part of the same problem.

I also don't think the answer is to have mathematicians manually review everything that AI produces. In fact, I strongly believe that as models improve, human review will become less and less necessary for many of these tasks. We're already seeing agents become useful at finding discrepancies between informal and formal statements, checking definitions, and identifying missing assumptions. I expect much of the work involved in auditing autoformalization, autoinformalization, and even decomposition and abstraction quality to become increasingly automated, and not only in mathematics: the same goes for physics, computer science and any field where a claim can be stated precisely. The human review we need today is a practical limitation, not a permanent feature of formal mathematics. As review gets automated, the human goals change: less time checking each statement, and more deciding which questions matter, what the right definitions are, and what the results mean. But that makes it even more important to understand what we're checking and how reliable those checks actually are. Automated semantic review can become highly reliable without being formally certified, and a more accurate model still doesn't give the guarantee Lean's kernel gives. My hope is that we can eventually make these processes just as systematic and trustworthy as proof checking, so mathematicians, physicists and computer scientists can spend more time understanding, extending, and discovering new results.

Ultimately, I don't just want us to produce more formally verified proofs. I want us to build mathematical knowledge that the community can understand, trust, and reuse, and that makes the next discovery easier. I think autoformalization, autoinformalization, and the quality of the mathematical abstractions we choose will play a central role in getting us there.

Most of the habits here I learned from people in the Lean community.

  1. The excerpt compares the two growth conditions and nothing else, so it says nothing about eigenfunctions themselves. The full file compiles on Lean 4.35 with Mathlib, uses only the three standard axioms, and also proves that the paper's condition implies the challenge's. ↩
  2. Both are from a recent Lean Zulip thread about press coverage of LLM proofs; Mario Carneiro's and Shreyas Srinivas's points above come from the same thread. ↩