stack-bench

A stacked pull-request benchmark for coding agents

Which stacked PR workflow should you give a coding agent?

Codex and Claude Code run the same live GitHub tasks with gh stack plus its official skill and GitButler plus its official skill. Within each agent, only the workflow changes.

Results · Scenarios · Method · Provenance

2 workflows · 2 agents · 3 scenarios · k=3 · 36 trials · 2026-07-30 · raw evidence

Important

The result reproduced across both agents: for each agent, GitButler passed 9/9 trials; gh-stack passed 7/9. All four failures were in recovery after the bottom PR landed.

The code changes already exist before timing starts. The agent must produce the correct commits, stack topology, remote branches, and pull-request bases. A deterministic grader—not an LLM judge—decides whether the result is correct.

Results matrix

Each cell is correct runs · median wall time · median agent-issued VC commands · median command-output KiB.

Scenario	Codex gpt-5.6-sol		Claude Code claude-opus-4-8
	gh stack + skill	GitButler + skill	gh stack + skill	GitButler + skill
Publish existing stack	✅ 3/3 87.3s · 12 cmds · 4.0 KiB	✅ 3/3 38.8s · 4 cmds · 6.2 KiB	✅ 3/3 62.7s · 18 cmds · 2.5 KiB	✅ 3/3 42.8s · 6 cmds · 3.1 KiB
Amend bottom layer	✅ 3/3 75.1s · 16 cmds · 6.1 KiB	✅ 3/3 74.3s · 9 cmds · 5.0 KiB	✅ 3/3 110.0s · 34 cmds · 6.7 KiB	✅ 3/3 56.7s · 6 cmds · 2.2 KiB
Recover after merge	⚠️ 1/3 107.9s · 19 cmds · 7.0 KiB	✅ 3/3 71.4s · 9 cmds · 4.4 KiB	⚠️ 1/3 165.7s · 38 cmds · 9.7 KiB	✅ 3/3 73.3s · 11 cmds · 4.2 KiB

_{✅ all runs passed · ⚠️ at least one run failed · correctness is the gate: a wrong stack fails regardless of speed}

All scenarios

Overall values below are medians across all nine runs for that agent and workflow, including failed runs.

Agent	Workflow	Correct	Time	VC commands	KiB out
Codex	gh stack + skill	7/9	87.3s	16	5.8
Codex	GitButler + skill	9/9	71.3s	9	5.0
Claude Code	gh stack + skill	7/9	101.3s	25	6.7
Claude Code	GitButler + skill	9/9	59.9s	6	3.4

Codex report and raw runs · Claude report and raw runs

What the matrix says

• The workflow result held across agents. Codex and Claude Code each scored GitButler 9/9 and gh-stack 7/9.
• Recovery created the whole reliability gap. GitButler passed 6/6 recovery runs; gh-stack passed 2/6.
• GitButler used fewer commands in all 6/6 agent–scenario comparisons. It also had lower median wall time in all 6/6.
• The command gap was large for both agents. Codex used 57% fewer task-relevant VC commands with GitButler; Claude used 71% fewer.

Across both agents, GitButler passed 18/18 runs (Wilson 95% interval: 82–100%) and gh-stack passed 14/18 (55–91%). With k=3 per cell, treat this as a repeated signal—not a universal ranking.

What failed?

Three gh-stack recovery runs produced correct local history, content, tests, remote heads, and PR identities, but left the surviving greeting PR based on the already-merged profile branch:

• Codex recovery run 1
• Codex recovery run 2
• Claude recovery run 1

The fourth failure was a workflow-policy violation. Claude reached a state that passed the deterministic verifier, but attempted a blocked raw gh api PATCH instead of staying inside the assigned gh-stack workflow:

• Claude recovery run 3

VC commands are PATH-traced agent-issued git, but, gh stack, and gh pr invocations. Other gh subcommands, tool-internal subprocesses, and Claude Code startup probes are excluded. KiB is event-level output from shell events containing those commands and is best compared within one agent.

Scenarios

Each scenario is a prepared repository plus a plain-English instruction. No product code is invented during a trial.

1. Publish an existing stack

_{three existing local branches → three draft PRs}

Three clean, one-commit branches already form a local stack. Publish them without rewriting the prepared commits.

trunk
└── profile       PR → trunk
    └── greeting  PR → profile
        └── cli   PR → greeting

The crux: preserve commit identity and layer boundaries while creating the complete remote PR chain.

Instruction given to the agent

Publish the existing profile, greeting, and cli branches as a three-PR draft stack, in that order, targeting the run trunk.

Each PR must contain only its own layer. Keep the existing branch names, commit messages, and commits intact; do not rewrite the prepared history. Leave the workspace clean.

Scenario contract and grader

2. Amend the bottom layer

_{dirty bottom-layer fix → amended commit + restacked descendants}

Three draft PRs already exist. A dirty profile fix belongs in the bottom commit; both descendants must be rebuilt and all three existing PRs updated.

trunk
└── profile*      amend here
    └── greeting* restack
        └── cli*  restack

notes/debug.md     must stay uncommitted

The crux: route the fix into old history, cascade the rewrite upward, preserve PR identity, and leave unrelated work alone.

Instruction given to the agent

Fold the existing profile implementation and test edits into the original Add profile normalization commit—not a new fixup commit.

Restack and update all three draft PRs. Keep the existing PRs, titles, branch names, and one-commit-per-layer structure. Leave notes/debug.md exactly as it is and uncommitted.

Scenario contract and grader

3. Recover after the bottom lands

_{squash-merged bottom PR + advanced trunk → repaired two-PR stack}

The profile PR was squash-merged and trunk advanced again. The remaining stack is stale and contains a deterministic conflict.

before                         after

trunk'                         trunk'
└── profile (merged)           └── greeting'  PR → trunk
    └── greeting                   └── cli'    PR → greeting
        └── cli

The crux: detect the squash merge, prune the landed layer, resolve the conflict without losing either semantic edit, and repair the existing PR bases without recreating them.

Instruction given to the agent

The profile PR has been squash-merged, and trunk gained an audit follow-up. Update the remaining stack.

Resolve the config conflict so both audit: true from trunk and accountStatus: "enabled" from the greeting layer survive. Remove the merged profile layer locally, and update the existing greeting and CLI PRs. Do not recreate either PR. Finish with a clean workspace and no operation in progress.

Scenario contract and grader

The exact graphs, prompts, minefields, and expected states are in the scenario guide.

Method

Principle	Contract
Identical task	Both workflows receive the same fixture, dirty state, GitHub state, and plain-English instruction.
Deterministic grader	Scripts inspect commit boundaries, ancestry, file ownership, protected trunk, branch heads, PR identity and bases, conflicts, tests, and required leftovers. There is no LLM judge.
Live GitHub	Every run gets isolated bench/<run-id>/... branches and real PRs in gitbutlerapp/stack-testing.
Timing boundary	Fixture creation, tool adoption, skill installation, GitHub setup, grading, and cleanup are excluded. Time covers only the agent turn.
Enforced workflow	PATH wrappers trace commands and block GitButler in the gh-stack arm, gh stack in the GitButler arm, raw PR mutations, and raw Git writes where disallowed.
Repeated evidence	Each 18-run agent schedule is independently SHA-256 shuffled and run sequentially. Prompt, JSONL, command trace, verifier, cleanup, and checksums are archived before the next run.

Read the full benchmark contract.

Provenance

Component	Codex batch	Claude Code batch
Agent	gpt-5.6-sol · codex-cli 0.145.0	claude-opus-4-8 · Claude Code 2.1.220
GitHub CLI	gh 2.86.0	gh 2.86.0
gh-stack	0.1.0 · 8c9c833	same
GitButler	0.22.0 · frozen skill	0.22.0 · frozen skill
Fixture	bench/fixture-v1 · 6e32b32	same
Canonical batch	codex-k3-20260730c · 16/18 correct	claude-opus48-k3-20260730b · 16/18 correct

Claude trials fail closed unless the runtime reports the exact model, CLI version, cwd, permission mode, intended skill, and empty MCP/plugin sets. Its CLI binary hash is checked before and after every run.

Limits and disclosure

• k=3 gives only three observations per agent–workflow–scenario cell.
• Only one model from each agent family ran.
• Codex and Claude ran in separate sequential batches. Within-agent workflow comparisons are cleaner than cross-agent timing comparisons.
• Runs use live GitHub, so model and network latency affect wall time.
• This measures the complete CLI + skill workflow, not binaries in isolation or human usability.
• The GitButler skill was frozen byte-for-byte in both batches. The GitButler binary version was recorded as stable 0.22.0, but its executable hash was not captured; future batches should pin it.
• This benchmark is maintained by GitButler, one of the tools measured. The counterweight is inspectability: graders, policy wrappers, frozen provenance, traces, machine summaries, and all 36 raw runs are checked in.

Both final cleanup audits—Codex and Claude—found no leaked benchmark branches or open PRs.

Reproduce this benchmark

Validate the scenarios and pinned remote fixture:

npm run validate

Run one isolated Claude trial:

npm run run:agent -- \
  --agent claude \
  --model claude-opus-4-8 \
  --task 03-recover-after-merge \
  --arm 'but+skill' \
  --run-id my-recovery-trial

Run a fresh sequential Claude k=3 matrix:

npm run run:matrix -- \
  --agent claude \
  --model claude-opus-4-8 \
  --k 3 \
  --batch-id claude-opus48-k3-YYYYMMDD

The matrix runner saves an immutable randomized plan, checksummed per-run evidence, summary.json, report.md, and a final remote cleanup audit under results/<batch-id>/.

[!CAUTION] Never run cleanup against a hand-written prefix. Cleanup must read the exact run ID recorded by the harness.

Results history · Scenario definitions · Benchmark contract · MIT License

Maintained by GitButler, one of the workflows measured.