Benchmark design: measurement gaps from the whole-repo runs + research-backed recommendations

[isPANN]

Summary

This is an archival / tracking issue capturing what our latest claude-code whole-repo runs (four frontier models) revealed about the benchmark's measurement design, the evaluator bugs we fixed along the way, and a prioritized set of design recommendations synthesized from recent community work on agentic benchmarks.

Note on secrecy: this repo is public, so specific buggy-rule identities and certificates are deliberately withheld below (they are the answer key). Bugs are referred to by count/category only.

---

1. What the runs surfaced (the core design tension)

Four models ran whole-repo (one open session over the ~290-rule library, subscription auth, prompt stripped of any "how to find bugs" recipe). The scoreboard is not the interesting part — the shape of the runs is:

• Coverage and certification are two independent axes, and the model owns both. One model wrote a script that swept all ~290 rules (cheap: 2.8M tokens / 43 turns) — its own output printed a genuine bug — yet it certified only 1 because it never turned that detection into a certificate. Another model probed ~13 rules deeply (18.3M tokens / 162 turns) but certified 5, everything it found. The broad sweeper covered 20× more and scored lower.
• Coverage is model-strategy-determined and essentially unbounded. There is no turn cap (removed — see below) and the wall-clock rarely binds, so how much of the library gets touched is a strategy choice, not a harness constraint.
• The score conflates four things: strategy choice + persistence/patience + "certification discipline" (bothering to prove each finding) + actual bug-finding skill. Only the last is what we mean by capability.
• Single-session variance is high. One model's two runs both underperformed for different reasons — a transient API stall, then under-certification.

Bottom line: as-is, the leaderboard number is not a clean measure of capability, and that is unlikely to satisfy the community.

---

2. Evaluator bugs found & fixed this cycle

Flushed out by a smoke run; all now fixed with tests (verifier calibration green). Two are integrity holes, three are false negatives that were scoring real bugs as rejections.

#	Class	Bug	Fix
1	integrity	--max-turns carried over from the mini-swe backend but claude -p self-terminates long before it — a vestigial cap.	Removed; wall-clock is the only backstop.
2	integrity	--allowedTools only suppresses the permission prompt — it does not restrict the toolset. The agent could Write/Edit/spawn subagents into the library under test, and ambient MCP servers leaked in.	Sandbox via --tools + --strict-mcp-config.
3	false negative	Verifier read the target type only from bundle.target.type, but a real pred reduce bundle names it in path[-1].name. Any cert pasting a genuine bundle failed to resolve its target.	Handle the real bundle shape.
4	false negative	Models sometimes wrote the bundle as prose ("target type X"); extraction fed the whole phrase to pred.	_clean_target_type extracts the type token; a hallucinated name still fails at pred reduce, so leniency can't accept a fake.
5	false negative	When a bug was both narrated (bare string) and logged (structured dict), de-dup kept the first-seen — often the degraded copy.	De-dup keeps the richest copy per (rule, source).

Fixes 3–5 recovered real bugs for two models (each went from 0 → 1). All fixes preserve zero-trust: the target type is only routing; the bug is always re-derived from source.

Root cause of 3–5: certificates are regex-scraped from the model's free text + a durable text file — there is no structured submission channel. That fragility is the through-line, and item R1 below is the real fix.

---

3. Recommendations (synthesized from recent community work)

Verdict from the research: our zero-trust verifier is already best-in-class (proof-carrying witness re-execution + clean-env re-derivation, the SV-COMP / SMT-COMP / Kaggle-code tier). The gaps are in (a) the submission mechanism, (b) scoring methodology, (c) the leaderboard-as-oracle.

Tier 1 — do first (dissolves the bugs we hit)

• R1. Replace text-scraping with a structured pred cert submit tool channel. Agent calls pred cert submit --rule R --source <file> --target T; the CLI validates the schema, re-derives the round-trip zero-trust, and returns accept/reject at submit time. Eliminates the whole class of format-fragility (items 3–5) because there is one machine-authored artifact and the model never hand-formats. Precedent: SWE-agent first-class submit (arXiv 2405.15793), CVE-Bench /done grader (arXiv 2503.17332), CTFExplorer unified submit (arXiv 2602.08023).
• R2. Make "certification" the harness's job, not the model's. Require batch sweeps to emit structured JSONL {rule, source} candidates; the harness auto-runs the verifier over every candidate and reports bugs located vs bugs certified separately. The detected-but-uncertified bug would then certify itself, and the located/certified gap becomes visible instead of a silent zero.
• R3. Enforce minimality in the trusted verifier via delta-debugging. Auto-shrink any breaking instance to a 1-minimal witness and reject non-minimal certs — makes "minimal counterexample" a proven property and blocks padding attacks. Precedent: SMT-solver fuzzing + ddmin (Brummayer & Biere), minimal witnesses (arXiv 1910.10636).

Tier 2 — credibility table-stakes

• R4. Decide: fixed per-rule task set vs open whole-repo (the central fork — see §4). A fixed task set (score = certified-bug rate over a common denominator) kills the coverage/strategy confound entirely; the community norm (GAIA/SWE-bench/τ-bench). If we keep whole-repo, we must report exposure-normalized metrics (rules_probed / N, certified / probed).
• R5. Multiple trials + reliability + confidence intervals. k≥3–5 runs; report pass^k (reliability) + avg@k (mean) with 95% CIs, and cluster standard errors on the rule (naive SEs can be 3×+ too small). Explicitly exclude infrastructure failures (e.g., transient API stalls). Single-run leaderboards are treated as a red flag. Sources: Anthropic Adding Error Bars to Evals (arXiv 2411.00640), reliability framework (pass^k), METR elicitation.
• R6. Budget-normalize. 2.8M vs 18.3M-token runs are not comparable as single numbers. Publish a bugs-found-vs-token(and-turn) curve and compare at a fixed compute point, à la HAL's cost/accuracy Pareto (HAL, UK AISI).
• R7. Fair, documented elicitation. Identical harness + stopping rule for all models; report ceiling-vs-typical, since self-termination is a scaffold artifact, not capability (METR guidelines).

Tier 3 — leaderboard-as-oracle & gaming defenses (our biggest unaddressed risk)

Every accept/reject reveals one bit about which rules are buggy. This is where ARC-AGI / Kaggle spend most of their design budget.

• R8. Private held-back rule slice drives the authoritative ranking; public reporting stays aggregate-only (we already do the latter). Precedent: ARC-AGI semi-private/private split (arXiv 2601.10904).
• R9. Rate-limit submissions; errors consume quota; opaque constant-information verdicts (uniform "not reproduced" — never reveal size-cap vs no-bug vs malformed). Precedent: Kaggle probing defenses.
• R10. Rotate the secret buggy-rule set + seed canary/decoy rules to detect answer-key scraping/trading; add contamination canaries.
• R11. Adversarial hacker-fixer pass before trusting rankings, and harden the verifier (require two successful solves that genuinely disagree, not one erroring; validity check on the source). METR found "don't reward-hack" instructions can raise hacking to 70–95%; Terminal-Bench had >15% of verifiers bypassed.

Tier 4 — framing / novelty

• The correctness notion (round-trip solution/certificate preservation) is prior art — Karp, a DSL that property-tests hand-authored reductions (PLDI'22). Our novelty is turning it into a scored, adversarial, agent-facing benchmark over a production library at scale, with zero-trust re-derivation. Position for ML (verifier-grounded, contamination-resistant bug-finding), TCS (library-scale differential testing of reductions, soundness-first à la SMT-COMP), and SE (delta-debugged minimal witnesses). Adopt the ABC checklist task/outcome-validity vocabulary (arXiv 2507.02825).

---

4. The one decision the team must make first

Fixed per-rule task set (max credibility; kills the coverage/strategy confound; every model faces the same 290 tasks) vs. keep the open-ended whole-repo "hunt the library" framing (the original vision, but requires heavy exposure-normalization and multi-trial reporting to be defensible). Everything in Tier 2 depends on this call.

---

5. Proposed next steps

• Team decision on §4 (fixed task set vs open whole-repo + normalization).
• Implement R1 (pred cert submit structured channel) — highest value, also the root fix for the false negatives above.
• Implement R2 (auto-harvest + located-vs-certified reporting).
• Implement R3 (delta-debugged minimal witness in the verifier).
• Adopt R5/R6 reporting (trials, CIs, budget curves) once §4 is settled.
• Design the leaderboard-oracle defenses (R8–R11) before any public submission intake scales.
• Verify the recent (2026) arXiv citations above against their abstracts before using any in a formal write-up.

_{Prepared from the 2026-07-12 whole-repo runs and a four-track literature review. An internal design brief with the full comparison and figures exists separately.}