Skip to content

README.md

Latest commit

 

History

History
141 lines (106 loc) · 7.4 KB

README.md

File metadata and controls

141 lines (106 loc) · 7.4 KB
Raw

diffbisect

Your agent (or your long day) left a big uncommitted diff and a test now fails — diffbisect tells you exactly which hunks to revert.

git bisect needs commits; it cannot see inside one, let alone into a dirty working tree. diffbisect delta-debugs the diff between a base rev and your working tree, splits it into change regions, and reports the hunks that — reverted — make your test pass again. It reasons below commit granularity, where git bisect cannot reach at all.

Install / quickstart

npx @precisionutilityguild/diffbisect -- npm test

diffbisect only reads the exit code of the command after --, so vitest, jest, mocha, node --test, pytest, and cargo test all work unmodified. One npx command — no Python, no config, no daemon, no API key, no LLM. Deterministic given a deterministic test command.

A real run (trimmed)

Real captured output. The demo repo carries nine working-tree edits — seven benign refactors and one seeded two-region conjunctive breakage: primary() and backup() were each flipped to false, and route() === "ok" only fails when both are present. That is the hard case — interference, not a lone bad line — and the default mode isolates it exactly. Trimmed to the verdict; the full transcript (all nine hunk diffs) is in pkg/README.md.

$ diffbisect -- node check.mjs
diffbisect · "node check.mjs" · base HEAD
  9 regions · 26 probes (59 logged incl. memoized)

regions
  H1  math.js:1-4  [region]
  ...                              (H2–H7 elided; all nine print in the full run)
  H8  math.js:35-39  [region]
  H9  math.js:40-45  [region]

culprits (revert these): H5 H8

causes
  #1  H5 H8  (1-minimal witness)

complete: reverting the culprits from the full diff passes (confirmed by a passing probe).

complete: ... is not a claim — diffbisect ran the command on a worktree with H5 and H8 reverted and observed exit 0.

What this reports

  • culprits = the revert set. Remove exactly these hunks and your tree passes; when complete is true that was confirmed by an actual passing run, not inferred.
  • causes = the independent breakages, each with a witnessMinimal flag — one 400-line diff can carry more than one unrelated regression.
  • cluster = residue diffbisect could not carve. Hunks that interfere beyond the budget land here rather than being force-fit into a false "minimal" answer — treat it as your suspect list.

What this is NOT

  • Not git bisect. That bisects commits and cannot look inside one or into a dirty tree.
  • Not git-bifurcate. Its --strategy hunk stays commit-anchored and it is Python; diffbisect takes the dirty working tree as first-class input and treats interference residue as a result.
  • Not lockbisect / depbisect. Those bisect lockfile or dependency versions across commits; diffbisect bisects the source hunks in your working tree.
  • Not CI coverage / patch-coverage tooling. Codecov and diff-test-coverage gate lines changed between commits; diffbisect isolates which uncommitted hunks broke a specific test.

Test command in a fresh worktree (the #1 stranger trap)

Each probe runs in a fresh git worktree, which shares the object database but not untracked files: node_modules/, build output, and anything .gitignored is absent. A bare npm test fails to start, so the base-precondition probe FAILs and diffbisect exits 2 with the base tree does not PASS — before any bisection begins. The fix is the wrapper git bisect run users already write: symlink the untracked artifacts into each probe's worktree first.

diffbisect -- sh -c '[ -e node_modules ] || ln -sfn /path/to/repo/node_modules node_modules; exec npm test'

/path/to/repo is your real checkout; the [ -e ... ] || guard makes it idempotent across probes. The same applies to any generated artifact the tests need — a dist/, a build cache, a binary.

Flags

Flag Default Meaning
--json off Emit the machine-readable contract.
--base <rev> HEAD Base rev to diff the working tree against.
--target <rev> — Bisect the committed range <base>..<rev> instead of the working tree.
--max-probes <n> 500 Probe budget. Exhaustion is a clean exit 3, not an error.
--confirm <n> 1 Confirmation reruns for the first FAIL and revert-confirming probes (0 disables).
--minimal off Classic delta debugging end-to-end: a 1-minimal witness per cause, guaranteed but slower.
--fast off A single fast pass only; may return a residue cluster (explicitly labelled).
--jobs <n> 1 Speculative parallel probes; wall-clock only, byte-identical log to --jobs 1.
--keep-worktrees off Leave probe worktrees on disk (debug).
-- <cmd> required Everything after -- is the test command; its exit code is the oracle.

Exit codes

The command after -- is the oracle, using git bisect run's convention (0 PASS; 1–127 except 125 FAIL; 125 UNRESOLVED; ≥128 signal death → abort). diffbisect's own exit codes are distinct — branch on these, not on output text:

Code Meaning
0 Culprit set isolated, cluster empty.
1 Runtime/internal failure: not a git repo, worktree setup died, or the command was killed by a signal.
2 Usage/precondition error: bad flags, no command, or the base tree does not PASS / the full diff does not FAIL.
3 Residue: the probe budget was exhausted or an interdependency cluster remains. The partial result still prints.

Limitations

  • Interdependent hunks can exhaust the budget into a cluster. Non-monotone interference is the normal case for real diffs; violations degrade minimality, never correctness.
  • Flaky tests degrade to UNRESOLVED. --confirm protects the search from an inconsistent probe; it is not a flake detector. A truly flaky suite will not isolate cleanly.
  • Binary, rename, and mode units are atomic, as are untracked new files. Windows is unverified.
  • Your tree and index are never touched. Every probe runs in a fresh git worktree.

More

  • Full contract — --json field-by-field, the probe exit-code convention, the mocha wrapper caution, exit-2/exit-3 transcripts: pkg/README.md.
  • Release: v0.1.0.

Lineage: Zeller & Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE TSE 2002 (https://doi.org/10.1109/32.988498).

Part of a family

Three tools, one idea: answers grounded in what your tests actually execute — evidence an agent can't hallucinate. Pick by the question you're holding:

Your question Tool
"A test is failing — which line is the bug?" culprits — spectrum fault localization (npm)
"Where is feature X implemented?" recon — feature location by coverage diff (npm)
"My uncommitted diff broke the tests — which hunks?" diffbisect — delta debugging below commit granularity (npm)

MIT © Precision Utility Guild