{ "_doc": "One result, read from the codebase’s own files at the commit named here. Every value is that file’s own, copied verbatim; nothing here is written by hand. A value replaced by {withheld: true, sha256_of_value, bytes} is committed by its sha256 instead of published, for the reason it states, and is never edited. Each whole source file is committed by sha256 below, and its bytes are published only where its visibility says so. The codebase is named by the public subject its files are published under.", "lane": "code-admission", "repository": "code-admission", "commit": "b36a228c1b7ad32cc67717ac41d4419be13994a3", "portfolio_id": "code-admission:certbench", "id": "certbench", "attestation_row": { "attested_at": "2026-09-24", "attestor": "01-videocodec", "repository": { "withheld": true, "why": "a working-directory name (R8)", "sha256_of_value": "4ae679153e57269f61a4bf89b4afd161521d2f44caf38195a42876bb222c3ee0", "bytes": 7 }, "source_doc": "CROWN_JEWELS_RESOLVED.md", "source_doc_sha256": "5110525f2dc1b7a9b4781fdfc31491405d8d974ef721fa679dc15d7d6b2832c4", "state": "attested", "top_n": 20, "note_sha256": "ecc1adb4ea43531df2a8f265f2402dfdc09ca6cd3f4e6f746f92bb63a9d0dad6", "ids_in_row": 20, "file": "dataroom/TOP_N_ATTESTATION.jsonl", "line": 1 }, "claim_source": { "receipt": "code-admission/dataroom/top40.json", "sha256": "fcfaafe5f4f159116f21a97ad79f09535cd296e2dde0f3dc1765943b7fe442c8", "field": "entries[14].claim", "file": "dataroom/top40.json" }, "top40_entry": { "id": "certbench", "rank": 15, "title": "CertBench provable-soundness scoring", "claim": "Thousands of candidate guard patches are scored by the solver-free decision procedure: 2,673 certified, 3,077 proven unsound, 5,750 decided of 6,036 scored. z3 and cvc5 each agree with all 5,750 of those verdicts, though 333 of the counterexamples the scorer records (all for sudo) lie outside its own domain; SMT replaces each with an in-domain one that replays. The 286 the scorer leaves indeterminate were handed to SMT, the same SMT-LIB input to both solvers, which agree on every case: 38 are certified sound, 195 are proven unsound by a 32-bit counterexample that replays in exact integer arithmetic, and 53 stay undecided, because each is unsound in the scorer's integer model only through a value of 2^32 or more and sound once every variable is held to 32 bits. That brings the decided count to 5,983 of 6,036. The same scorer was pointed at 24 real, live GLM-5.2 candidate patches: 10 are proven unsound (0.4167, median over-acceptance 131,072), which lands inside the 16-46% band the DARPA SoK reconstructs for tests-plus-fuzzing pipelines (artifacts/certbench/summary.json :: live_glm_anchor).", "scope": "6,036 candidate guard patches scored by the solver-free decider; 286 are explicitly indeterminate and handed to SMT rather than guessed.", "limits": "Every verdict is a fact about a linear integer model of each guard over the scorer's declared domain, not about the libraries' C code. The SMT leg ran z3 4.15.4 and cvc5 1.3.4 (artifacts/certbench/smt_leg.json :: solvers). Its 38 certifications rest on both solvers' unsat, and each also carries an integer-tightened Farkas certificate that replays. Its 195 refutations rest on counterexamples that replay. 53 of 6,036 (0.9%) stay undecided: their verdict turns on 32-bit wraparound, which the linear model does not represent. For 333 of the 392 sudo_set_cmnd patches the scorer calls unsound, the counterexample it records has su_size = 0, outside its declared domain (su_size >= 1). summary.json's every_decided_verdict_is_a_proof = true is therefore not true of those recorded points, though an in-domain counterexample from SMT stands behind every one of the 392 verdicts. The over-acceptance figures are counts over a declared 48-bit model domain.", "artifact_path": "artifacts/certbench/summary.json", "witness_command": "python3 scripts/attest/assert_claim_number.py certbench", "witness_result": "exit=0 · EXIT0_NO_TOUCH · PASS|| certbench: all 29 number(s) and value(s) the claim of record states equal the committed artefact's", "witness_class": "ASSERTING", "third_party_axis": "—", "facts": { "ran_with_receipt": true, "third_party_graded": false, "negative_control_stated": false, "ip_class_known": false, "reproduce_command": true, "regenerates_not_verifies": false }, "facts_count": 2, "repro_command": "python3 scripts/attest/assert_claim_number.py certbench", "reproduce_command": "python3 scripts/attest/assert_claim_number.py certbench", "ip_class": null, "evidence_tier": "T2", "run_status": "RAN", "output_excerpt": "$ .venv-fcm/bin/python -m certbench.runner\n[certbench] scored 6036 | decided(proof each)=5750 cert=2673 proven_unsound=3077 indeterminate=286 | heartbleed exact-unsound=0.5737 | GLM anchor=0.4166666666666667\nEXIT=0\nNOTE: `make certbench` is a NO-OP -- the target name is shadowed by the same-named `certbench/` directory, so make reports 'up to date' and runs nothing. The module must be invoked directly. That shadowing is a real defect in the Makefile, found by this pass.\n$ python3 -m certbench.smt_leg\n[certbench-smt] handed 286 to SMT: certified 38 (tightened Farkas 38), proven unsound 195, undecided 53 | decided 5983/6036 (0.9912) | solvers agree True | audit of 5750 decided: all agree True, recorded counterexamples outside domain 333, in-domain found 333\nEXIT=0", "attestation_rank": 55 }, "witness_quality_entry": { "id": "certbench", "portfolio_id": { "withheld": true, "why": "a working-directory name (R8)", "sha256_of_value": "8bb864777fa5c885ad23043c1380b4fcf0fc3b2ada28ea1d2480aedc5e9f24a0", "bytes": 17 }, "class": "ASSERTING", "why": "value probe RED and control GREEN: the witness enforces the claimed number as its artifact carries it, and is not merely parsing the file.", "witness_command": "python3 scripts/attest/assert_claim_number.py certbench", "clone_exit": 0, "defect_demonstration": { "probe": "value", "number": "0.4166666666666667", "mutated_exit": 1, "restored_exit": 0 }, "asserts": "the witness fails when the claimed number in its artifact is changed" }, "packet": null, "measured": { "measured": true, "basis": "defect_demonstration.mutated_exit = 1", "exit": 1, "defect_rejected": true }, "register": { "truth_state": "attested", "truth_line": 113, "measured_per_register": false, "witness_class_per_register": "ASSERTING", "ip_class_public": "unknown", "ip_class_sha256": "b23a6a8439c0dde5515893e7c90c1e3233b8616e634470f20dc4928bcf3609bc", "ip_class_source": " @ ", "receipt": "register/CROWN_JEWELS_TRUTH.md", "receipt_sha256": "79ca50c3472bae4ebd179ed533456325be8e0e9027739787bfba9f06442fc39b", "ip_class_receipt": "register/ip_class.jsonl", "ip_class_receipt_sha256": "1cd70f508c6b79f941e894144715088f18fcd31092e56e98a4cf6be526d9e724" }, "withheld_fields": [ { "field": "witness_quality.portfolio_id", "codes": [ "LANE_DIRECTORY_NAME" ] }, { "field": "attestation_row.repository", "codes": [ "LANE_DIRECTORY_NAME" ] } ], "sources": { "attestation": { "receipt": "code-admission/dataroom/TOP_N_ATTESTATION.jsonl", "source": "dataroom/TOP_N_ATTESTATION.jsonl", "commit": "b36a228c1b7ad32cc67717ac41d4419be13994a3", "sha256": "b74ec34446c82d9c19cb2f8d32481efab6dae665d0e521fa8df4d77981db1371", "bytes": 3842, "visibility": "sealed" }, "top40": { "receipt": "code-admission/top40.json", "source": "top40.json", "commit": "b36a228c1b7ad32cc67717ac41d4419be13994a3", "sha256": "6015b44486bcc57677221edd3f60d2020cbbaeaba18295305f46d145ef489e0b", "bytes": 547392, "visibility": "sealed" }, "witness_quality": { "receipt": "code-admission/dataroom/witness_quality.json", "source": "dataroom/witness_quality.json", "commit": "b36a228c1b7ad32cc67717ac41d4419be13994a3", "sha256": "7b8f6118a896052bc5b49ee22b3142833868f4f9b0eae8d0700a82e72450d84f", "bytes": 101422, "visibility": "sealed" } } }