feat(terminal-bench): selectable agent harness plus full control and tracking

[abhinav-pola]

Why

Terminal Bench hardcoded the pi coding agent, so a result measured pi + model with no way to compare a different agent on the same tasks. This PR adds a selectable agent, converges Terminal Bench onto the Harbor sandbox it should always have used, and extends the agent selection to the other Harbor benchmarks.

Three commits, each reviewable on its own:

1. feat(terminal-bench) — selectable agent harness plus full control and tracking
2. refactor(terminal-bench) — converge onto the harbor sandbox
3. feat(harbor) — run Claude Code via ori on swe-atlas and deep-swe

1. Selectable agent on Terminal Bench

agent defaults to pi, whose direct pi --print --mode json invocation is unchanged.

# pi (default)
bun src/cli/index.ts --benchmark terminal_bench --model anthropic/claude-opus-5 \
  --solver-config '{"taskSubset":["build-cython-ext"]}'

# Claude Code via Ori Harness
bun src/cli/index.ts --benchmark terminal_bench --model anthropic/claude-opus-5 \
  --solver-config '{"agent":"claude","taskSubset":["build-cython-ext"]}'

Ori Harness is not a proxy — it sets ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN to OpenRouter and execs the real CLI, resolving auth from the OPENROUTER_API_KEY the sandbox already receives. It installs as a standalone Linux binary.

Control

Knob	pi	claude
reasoning level	--thinking (gains max)	--effort
systemPrompt override	--system-prompt	--system-prompt
appendSystemPrompt	--append-system-prompt	--append-system-prompt
allowedTools / disallowedTools	--tools / --exclude-tools	--allowedTools / --disallowedTools
isolateAgentConfig	--no-extensions/-skills/-prompt-templates/-context-files	--exclude-dynamic-system-prompt-sections

thinking and effort are separate enums because claude's scale has no off/minimal. Prompts and tool lists pass through env vars rather than argv to avoid shell quoting issues.

Tracking

generation_ids now resolves against the OpenRouter generations API for either agent, so spend is reconcilable from stored results.

Field	pi	claude
generation_ids	responseId	message.id
cost	usage.cost.total	total_cost_usd
reasoning tokens	usage.reasoning	thinking_tokens
generation_time_ms	wall clock (was 0)	duration_ms
agentTurns / agentToolCalls	event counts	num_turns / tool_use blocks
response_items	stream events (was null)	stream events

ori-parquet.test.ts runs the solver through the real parquet writer, reads the file back, and asserts no unexpectedly null columns.

Bug fix: silent pi API failures

A pi request failing at the API — for example a model id valid at Anthropic but not on OpenRouter — exits 0 with empty content, stopReason: "error" and all-zero usage. The old parser returned undefined for zero tokens and the solver substituted zeroed usage, making a hard 400 indistinguishable from a cheap successful run scoring 0. That is now surfaced in the verifier output with agentIsError in metadata. Past pi runs showing reward 0 with zero cost are worth re-checking.

2. Harbor convergence, and a dropped-settings bug

Terminal Bench is a Harbor benchmark — tasks come from harbor-framework/terminal-bench-2-1 and its task.toml is the format deep-swe parses — but it carried a parallel reimplementation of the harbor sandbox.

That divergence silently dropped per-task settings. TaskTomlSchema requires cpus, memory_mb, gpus and defaults allow_internet, but the dataset never read them and the local Modal layer passed only a timeout, workdir and command. Every task ran at Modal's defaults: 22 of 90 request more than 2048 MB and 6 request more than 1 CPU. An under-provisioned task OOMs or thrashes and scores 0, which reads as model failure.

terminal-bench/sandbox.ts and terminal-bench/modal-sandbox.ts are deleted; session.ts builds harbor's CreateSessionInput with the declared resources and the task files as uploads, and runTerminalBenchVerifier replaces the sandbox-owned runTests(). Terminal Bench also gains attach().

Reward parsing is unchanged in practice: all 90 tasks write echo 1 or echo 0, so harbor's parseReward and the old strict === "1" agree on this dataset.

3. Claude Code on swe-atlas and deep-swe

The agent-CLI machinery moves to agent-cli/ (schema.ts, harness.ts, runner.ts); terminal-bench/ori-solver drops to a thin wrapper. terminal-bench/schema re-exports what it previously declared, so the published entry point is unchanged.

swe_atlas (all three tracks) and deep_swe gain agent, defaulting to native (the existing runAgentLoop). Selecting claude installs the agent into the task image and runs it in place of the loop; each benchmark keeps its own verifier, so swe-atlas verifies in the agent sandbox and deep-swe still extracts a patch and verifies in a second one.

bun src/cli/index.ts --benchmark deep_swe --model anthropic/claude-opus-5 \
  --solver-config '{"agent":"claude","taskSubset":["abs-stepped-slices"]}'

wandr is deliberately excluded: it injects OpenRouter server tools into its own requests, which an external CLI replaces with its own, so it would no longer measure what it is for.

Capability gaps on the CLI path

• stepLimit has no equivalent. Neither pi 0.84.1 nor Claude Code 2.1.228 exposes a turn limit, so wall-clock timeout is the only bound.
• deep-swe checkpoint resume is skipped. A claude -p process cannot resume from step N, so the CLI path always starts a fresh sandbox instead of attaching.

Behavior changes to call out

Per CONTRIBUTING, for solver and scorer changes:

• Honoring declared resources will move scores on the 28 tasks that request more than the default cpus or memory. Presumably upward, since they were under-provisioned, but the new numbers are not comparable to previous runs.
• terminalBenchScorer now emits a verifier_log trajectory, filling scorer_trajectory for both agents. Additive; score values unchanged.
• pi parquet columns change value: generation_time_ms and reasoning_tokens from hardcoded 0 to real numbers, response_items from null to the full event stream (larger files).
• CLI-path results are a different system than runAgentLoop and are not comparable to existing harbor baselines. Native remains the default everywhere it existed.
• No other change here alters scoring.

isolateAgentConfig defaults to false, preserving current pi behavior. With the default, pi reads AGENTS.md/CLAUDE.md from the task working directory, so task-shipped files can influence the agent; the knob now exists to rule that out, but flipping the default would move scores and is left as a separate decision.

Verification

All five gate commands exit 0: format:check, check, typecheck, bun test (1248 pass, 0 fail), build.

Beyond fixtures, flags and parsers were checked against the real CLIs (pi 0.84.1, Claude Code 2.1.228): --thinking max with all isolation flags returns stopReason: stop; --effort high --disallowedTools returns is_error: false; parsers run against captured success and failure streams from both agents; a captured generation id resolves via /api/v1/generation to matching cost and reasoning tokens; generated run scripts pass bash -n in minimal and all-knobs-on forms. A dataset test walks all 90 real task files asserting every declared cpus and memory survives into metadata.

Not yet verified: no Modal run has happened. The in-sandbox image build (nvm + agent package + ori install) and real autonomy behavior are unproven, and the score impact of honoring resources is unquantified. Validate one task per agent per benchmark before any sweep, and confirm generation_ids is non-null in the resulting parquet.

Known and not addressed

oriInstallUrl and agentPackage are interpolated into Dockerfile RUN steps without validation, as flagged by Devin Review. piPackage already had this shape on main. Config reaches these values only via --solver-config from a caller who already has a shell, so this is hardening rather than a live boundary; deliberately left out of this PR.

🤖 Generated with Claude Code

[devin-ai-integration]

Devin Review found 2 new potential issues.

View 3 additional findings in Devin Review.

[devin-ai-integration]

🔴 Coding-agent runs for DeepSWE tasks cannot reach the model provider, so every such run fails

The DeepSWE agent container is started with outside network access turned off (allowInternet: meta.allowInternet at src/benchmarks/deep-swe/solver.ts:135) even when the newly added in-container coding agent is selected, so that agent can never contact the model service and the run always fails.
Impact: Selecting the Claude agent for DeepSWE produces failed/zero-scoring runs for every task.

Why deep-swe tasks are always network blocked on the CLI path

DeepSWE declares allow_internet with default false (src/benchmarks/deep-swe/schema.ts:36) and readDeepSweMeta also defaults it to false (src/benchmarks/deep-swe/dataset.ts:152-155, DEFAULT_ALLOW_INTERNET = false at src/benchmarks/deep-swe/dataset.ts:34). The Modal layer maps this to blockNetwork: !input.allowInternet (src/benchmarks/harbor/modal-sandbox.ts:81).

On the native path this is harmless because model calls are made from the host. On the new CLI path (src/benchmarks/deep-swe/solver.ts:190-197) the ori claude process runs inside the sandbox and must reach OpenRouter over the network, so with blockNetwork: true the agent invocation cannot succeed. Note the image build steps do have network (build phase), which masks the problem until the agent actually runs.

swe-atlas is unaffected because its schema defaults allow_internet to true (src/benchmarks/swe-atlas/schema.ts:28).

Prompt for agents

When the deep-swe solver runs an in-sandbox agent CLI (agent=claude), the agent process itself needs outbound network access to reach OpenRouter, but deep-swe tasks declare allow_internet=false by default (see src/benchmarks/deep-swe/schema.ts and src/benchmarks/deep-swe/dataset.ts), and makeModalSandboxLayer translates that into blockNetwork: true (src/benchmarks/harbor/modal-sandbox.ts). As a result the CLI path will always fail on deep-swe. Consider whether the agent sandbox should force allowInternet: true when cliHarness !== undefined (documenting the deviation from the task's declared network policy), or whether an explicit config knob should be required, and make sure the verifier sandbox keeps the task-declared policy so scoring stays offline.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟨 Unvalidated config values are interpolated into Dockerfile RUN commands and piped to a shell

agentPackage and oriInstallUrl come straight from user-supplied benchmark config (agentPackage: z.string().optional() and oriInstallUrl: z.string().default(...) in src/benchmarks/benchmark-config.ts:146-147) and are interpolated unescaped into RUN lines, including curl -fsSL ${opts.oriInstallUrl} | ... bash and npm install -g ${opts.agentPackage}. A value containing shell metacharacters becomes arbitrary commands executed during the image build, and an attacker-influenced install URL causes an unverified remote script to be piped into bash inside the build container.

---

Was this helpful? React with 👍 or 👎 to provide feedback.