reverify: deterministic tools judge AI claims

2akouwu/reverify is an MIT-licensed Python tool that stops AI agents from making things up. The model proposes a claim about a binary; deterministic tools check it against the actual bytes; the result comes back as VERIFIED / REFUTED / INCONCLUSIVE with evidence (GitHub, 2026-09-05). On 71 real Windows system files, the AI’s textbook answer was wrong 97% of the time — reverify caught every one and never accepted a wrong claim.

The project also ships reverify rollover, a lossless context hand-off that replaces summary-based compaction with a .reverify ledger so a fresh session starts from grounded facts instead of a recap.

What happened

On 2026-09-05, 2akouwu/reverify sits at ~911 stars, 194 forks, MIT license (GitHub, 2026-09-05). Created 2026-08-31, it has 71 commits and 15 tagged releases in five days. Latest release: v0.11.0 (2026-09-04), adding lossless context rollover across Claude Code, Codex CLI, Gemini CLI, and OpenCode (v0.11.0, 2026-09-04). PyPI latest: 0.10.0 (Python ≥3.8) (PyPI, 2026-09-05).

Why it matters

The “deterministic tool as judge” pattern is a concrete alternative to trust-and-summarize agent loops. Where context-mode (AIN-546, 2026-07-29) attacks context-window bloat through compression and session continuity, and DeepSeek dsh (AIN-609, 2026-08-19) provides a plugin-first agent harness, reverify tackles a different root problem: the agent’s claims are not grounded in reality. The model proposes; the tools decide. Only what survives verification becomes a fact.

For binary and structural claims it directly attacks hallucination and context rot. reverify equiv extends the same pattern to Python and C rewrites: an AI’s refactor is tested against shared inputs, not trusted on authority alone (README, 2026-09-05).

How the propose-and-verify loop works

A claim is any hypothesis about the binary. The deterministic tools check it against the actual bytes and return VERIFIED, REFUTED, or INCONCLUSIVE with evidence. Only VERIFIED claims survive into the agent’s working state.

Install: pip install reverify (pure-Python core), pip install "reverify[full]" (capstone + unicorn + lief), or pip install "reverify[angr]" (function/call analysis). The MCP server lets Claude Code and Cursor call the tools directly (README, 2026-09-05).

The benchmark

The headline result: 275 binaries, 4 formats, 0 false VERIFIED (pooled 95% upper bound ~1.4%) (BENCHMARK.md, 2026-09-05).

CI gate: any wrong claim marked VERIFIED fails the build. Each run records SHA-256 hashes of every binary, every verdict, and tool versions as a benchmark-<os> artifact. The corpus control confirms the textbook push rbp; mov rbp, rsp prior is verified where compilers emit it (gcc/clang -O0 with frame pointers) and refuted where they don’t (-O2, MSVC, AArch64) (BENCHMARK.md, 2026-09-05).

Lossless context rollover

reverify rollover install touches hook config for Claude Code, Codex CLI, Gemini CLI, and OpenCode — with backups and reversible uninstall (v0.11.0, 2026-09-04). The .reverify ledger records every grounded result (ESTABLISHED / KNOWN FALSE). When context fills up, the session is replaced — the fresh context opens from the ledger, not a model-written recap. This replaces model-written compaction, not augments it. Refuted claims come back as KNOWN FALSE, so the next session skips the same wrong prior (README, 2026-09-05).

The MCP server covers Claude Code and Cursor. Caveat: review before running on sensitive repos — the install modifies CLI hook config.

Risks and caveats

• Project age. Created 2026-08-31, one primary author (62 commits), plus one helper. Bus factor = 1 (GitHub, 2026-09-05).

• Release velocity. 15 tagged releases in ~5 days; expect breaking changes. The README explicitly warns about compatibility.

• Supply-chain lag. PyPI latest is 0.10.0, GitHub latest tag is v0.11.0. Pinning pip install reverify==X.Y.Z is mandatory, not optional (PyPI, 2026-09-05).

• Dual-use. RE / malware-analysis tool. README and SECURITY.md restrict use to authorized analysis: malware, CTF, interoperability, vulnerability research on owned or permitted artifacts (SECURITY.md, 2026-09-05).

• Benchmark scope. The “0 false VERIFIED” result is over 275 binaries / 4 formats using specific priors. It is evidence, not a universal guarantee. The 95% upper bound is ~1.4%, not zero.

• Optional MCP install. Rewrites CLI hook config — back up before installing.

Sources