Thousands of candidate code patches scored by proof: each one certified sound, proven unsound, or left undecided
On models of each check, 2,673 candidate patches of unstated origin are certified sound and 3,077 proven unsound, and two independent solvers agree with every verdict.
A patch that adds a bounds check can pass every test and still let bad inputs through. The lab scores candidate patches by proof instead. For each patch, a decision procedure that needs no outside solver either certifies it sound, proves it unsound with an input that breaks it, or says plainly that it cannot decide. Two independent solvers check its verdicts.
Why now
AI coding tools now write patches at scale, and tests plus fuzzing can pass a patch that is still unsound. The lab pointed the same scorer at patches written by one such model and proved some of them unsound.
What it shows
Many security fixes come down to one check: before reading or writing, make sure an index or a length is in range. A patch that gets the check slightly wrong can pass every test a developer writes and still accept inputs it should reject.
The lab scores such patches by proof. Each candidate check is modelled as arithmetic over whole numbers, and a decision procedure that needs no outside solver gives one of three answers: certified sound, proven unsound with a concrete input that breaks it, or undecided.
The numbers
The served files do not say how the candidate patches were produced; only the patches from the AI model, below, are described as real patches. Out of 6,036 candidate patches, the procedure certified 2,673 and found 3,077 unsound; both solvers agree with every one of those verdicts, and each unsound one has a breaking input inside the stated range. Two independent solvers, Z3 and cvc5, each agree with all 5,750 of those verdicts.
The 286 patches the procedure leaves open were handed to both solvers in the same input. They agree on every case: 38 are certified sound, 195 are proven unsound by an input that replays in exact arithmetic, and 53 stay undecided. That brings the decided count to 5,983.
Patches written by an AI model
The lab also pointed the scorer at 24 real candidate patches written by one AI coding model. The claim records that 10 of them are proven unsound.
Why it matters, and to whom
This is for teams that accept patches they did not write: maintainers of widely used libraries, security teams reviewing fixes, and anyone running AI tools that propose code. A test suite says a patch handled the inputs someone thought of. A proof of soundness covers every input in the model, and a proof of unsoundness comes with an input that shows the problem.
How it was checked
Every verdict the procedure reached was re-checked by two solvers built by different teams, given the same standard input. Behind every unsound verdict there is a breaking input inside the stated range that replays in exact arithmetic (for one library it comes from the solvers, not from the scorer's own record, as the next section explains), and every certified verdict from the solvers also carries a certificate that can be checked on its own. The lab's limits, in the same file, name the solver versions.
A flaw the lab found in its own scorer
The limits also record a flaw the lab found in its own scorer: for one library's patches, many of the breaking inputs it recorded lie outside its own stated range of inputs. The solvers supply a breaking input inside the range for every one of those verdicts, and the scorer's summary flag that every decided verdict is a proof is, by the lab's own account, not true of those recorded points.
What it does not claim
- It does not claim anything about the libraries' own code, or about any copy of them in use. The verdicts are about models of the checks, over a stated range of inputs, as the limits say.
- It does not decide every patch. A small share stay undecided because the answer depends on numbers wrapping around at a machine word.
- It does not claim that checking patches by proof is new. Solvers such as Z3 and cvc5 are standard tools for this kind of question. What the lab adds is a scorer that decides most cases without a solver, and two solvers that confirm every verdict it reaches.
How to reproduce it
The evidence file names the command that re-runs the check and the commit of the code it ran. It needs the lab's code, Python, and the two solvers named in the limits.
Prior art
- Z3 Theorem Prover (Microsoft Research): one of the two solvers that re-check the verdicts
- cvc5, an SMT solver: the second, independent solver
- SMT-LIB, the standard input language for SMT solvers: the shared format both solvers are given
Evidence
- The evidence file for this result
- Our claim file, with its limits
- All our published files, each with a fingerprint you can check
Related results across the group
- Six slightly weakened bounds checks pass random fuzzing, and two coverage-guided fuzzers find all six
- Random crashes found unclean recoveries in our AI agent's transaction layer; every crash point we chose recovered cleanly
- Three agent tools can leak a secret together even when every pair of them is safe