feat: add agent_dx benchmark for OpenRouter agent-experience evals

[devin-ai-integration]

TL;DR

Ports the agent_dx benchmark — OpenRouter's agent-experience (AX) eval measuring how well coding agents build with OpenRouter across models, harnesses, and installed resources — from openrouter-web/packages/bench-harness into this repository, following the new subtree ownership model where benchmark implementations live upstream here.

What changed?

• New benchmark at src/benchmarks/agent-dx/: dataset (12 benchmark tasks, 1 regression task, 2 discoverability tasks as task.toml + instruction.md + verifier fixtures), sandbox solver for two harnesses (OpenCode, Claude Code) with five AX profiles (baseline, docs, skills, mcp, agents), an independent out-of-band verifier per task with a structured JSON verdict channel for failure classification (agent / platform / fixture), an LLM judge for work quality and OpenRouter primitive alignment, trace parsing for resource-usage diagnostics (skills/MCP/docs/web-fetch activation), discoverability evidence collection (was OpenRouter chosen when not named), composite AX scoring (0.50 correctness + 0.25 quality + 0.15 efficiency + 0.10 reliability, with renormalization when billed cost is unavailable), a run-comparison CLI, and conformance tests that pin the per-task test scripts and duplicated verifier helpers to canonical implementations.
• Registration: agent_dx added to benchmark-meta.ts, registry.ts, benchmark-config.ts (AgentDxOptionsSchema / AgentDxConfigSchema), and the CLI config builder.
• Benchmark interface gains optional makeDatasetLayerForConfig so dataset size can depend on the run config (agent-dx dataset size varies by suite and task subset); datasetSizeById and the CLI progress total use it when available.
• Terminal-bench sandbox CreateSessionInput gains optional sandboxBufferSec (default unchanged at 300) so agent-dx's post-verifier stages (evidence collection, judge) fit inside the sandbox lifetime.
• Package exports for the dashboard-consumer surface: benchmarks/agent-dx/{schema,ax-score,result-row-metrics,trace}.

Why?

Agent-DX previously lived in openrouter-web/packages/bench-harness, which is now a vendored subtree of this repository and must not be edited in place. This PR lands the benchmark upstream; openrouter-web PR #30790 will be slimmed to the consumer pieces (Mission Control dashboard, Temporal wiring) on top of the subtree.

Concrete learning from the completed 24-cell, 3-epoch matrix run with this benchmark: harness identity outweighs model identity, and the agents resource setup compensates exactly where the harness is weak (Claude Code baseline→agents gained up to +11.5 AX while OpenCode gained nothing at higher cost).

How to test

bun test src/benchmarks/agent-dx        # 168 tests: dataset, solver, scoring, traces, conformance
bun run cli -- --benchmark agent_dx --model <slug> --epochs 1   # live run; requires MODAL_* and OPENROUTER_API_KEY

A live run provisions a Modal sandbox per trial, runs the harness against the task instruction, then runs the task verifier out-of-band against live generation records. Expect one parquet row per trial with reward, subchecks, judge quality, verdict category, and resource-usage diagnostics.

Benchmark impact

New benchmark; no existing benchmark's scores, datasets, solvers, or scorers change. The two shared-surface changes (makeDatasetLayerForConfig, sandboxBufferSec) are optional and default to prior behavior. Task dataset is authored in-house (no external dataset or licensing concerns); verifiers call the live OpenRouter API with a run-scoped key.

Reviewer focus

• The optional makeDatasetLayerForConfig addition to the Benchmark interface and its use in datasetSizeById / CLI progress totals
• Failure-classification flow: structured verdict file written by verifiers, read back in solver.ts, legacy log-scan fallback in result-row-metrics.ts
• Sandbox key handling in solver.ts / harness.ts (provided vs absent modes; absent mode rejects benchmark/regression suites)

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/295defde62e8417297c6fe7c67caddfe

[devin-ai-integration]

Original prompt from kenny.rogers

i want to create this for openrouter, i want to use the same set of tasks across models and harnesses, being able to do new runs when new ones are released, the goal is to see how well agents can successfully build things with openrouter across models, agent harnesses, skills/no skills, the mcp etc. to help devrel figure out what resources are needed and where we can improve. https://tempo.xyz/developers/blog/introducing-stable-bench-v1

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[perry-the-pr-reviewer]

Perry's Review

Verdict: 💬 Comments / questions

Risk: 🟢 Low

Review details

Summary

Re-review of the agent_dx benchmark port on the new head SHA (79df2a1). The incremental commit since the last review escapes newlines in verifier fail() messages (message.replaceAll(/\r?\n/g, "\\n")) so agent output can no longer forge verifier diagnostic lines — a solid defense-in-depth fix, now pinned by the conformance test. CI is green (validate + CodeQL).

Full re-read of all 100 changed files: the benchmark is well-structured with comprehensive test coverage (168 tests across 12 test files), proper security handling (API key redaction, shell-injection prevention via regex validation on skillsSource/docsSource/opencodePackage/ADX_PRESET_SLUG, prompt-injection defense in the judge prompt, fetch body cancellation in callJudge and verifier fetchGeneration), and backward-compatible shared-surface changes (makeDatasetLayerForConfig optional on the Benchmark interface, sandboxBufferSec optional on CreateSessionInput, datasetSizeById optional second arg — all default to prior behavior).

The skillsVariant nit from the previous review is still unaddressed. One additional UX nit on endpointId handling.

Findings

1. skillsVariant still unused (carryover, benchmark-config.ts:172) — the field is accepted by AgentDxOptionsSchema but never read by the solver, harness, or any other code. Per the project's Minimal Interface Design principle, either wire it up or remove it.

2. CLI passes endpointId to agent_dx despite the benchmark rejecting it (cli/index.ts:400) — buildBenchmarkConfig includes endpointId in the config (it's inherited from ModelBenchmarkBaseSchema), but makeAgentDxLayer rejects it at layer-build time with a runtime error. Consider stripping or rejecting endpointId in the CLI case "agent_dx" branch so the error surfaces at parse time, not at run time.

Security analysis

• Sandbox key handling: provided mode injects OPENROUTER_API_KEY; absent mode injects ADX_HARNESS_KEY (non-standard name, forcing the agent to discover the key mechanism). The verifier always receives OPENROUTER_API_KEY regardless of mode. absent mode is gated to suite: "discoverability" with task-subset validation. Correct.
• Key redaction: redactKeyMaterial strips sk-or-* patterns and exact key matches from all stored output (agent stdout, stderr, verifier output, workspace dump). Correct.
• Shell injection: skillsSource, docsSource, opencodePackage, and ADX_PRESET_SLUG all have regex validation that rejects shell metacharacters. Tests verify injection attempts are rejected. Correct.
• Prompt injection in judge: the judge prompt explicitly instructs the model to ignore embedded instructions in workspace content. Correct.
• Verifier newline escaping: fail() now replaces \n/\r\n with \\n before writing to verdict.json and stderr, preventing agent output from forging VERIFY FAIL: lines. The test.sh scripts also prefix app output with [app] . Good defense-in-depth.

Risk: 🟢 Low

Risk assessment:

Dimension	Severity	Risk	Reasoning
Implementation risk	🟩	Low	168 tests cover scoring, solver, harness, traces, discoverability, quality, and conformance; the code follows existing patterns (e.g., makeDatasetLayerForConfig mirrors mmmu-pro-vision).
Premise risk	🟩	Low	The benchmark design is sound — composite AX scoring with weight renormalization, structured failure classification, and out-of-band verification are all internally consistent.
Estimated impact	🟩	Low	New benchmark; no existing benchmark's scores, datasets, or scorers change. Shared-surface changes are optional with prior-behavior defaults.

Risk Factor	Severity	Risk	Reasoning
Reversibility	🟩	Low	New code only; removal restores prior state.
Detectability	🟩	Low	CI validate + CodeQL pass; 168 tests provide regression coverage.
Blast radius	🟩	Low	Contained to the new agent-dx benchmark; shared-surface changes default to prior behavior.
Data integrity	🟩	Low	No persisted state mutated; benchmark writes parquet results only.
Financial exposure	🟩	Low	Benchmark consumes API credits for live runs but does not affect billing or accounting.
Security and privacy exposure	🟩	Low	API key redacted from all stored output; sandbox key modes gated; shell injection prevented.
Propagation	🟩	Low	Package exports are additive; no existing consumer breaks.
Availability	🟩	None	Benchmark-harness tooling; no production serving path affected.
Recovery cost	🟩	Low	Revert the PR; no data to reconcile.
Time to correct	🟩	Low	Two minor nits, each a one-line fix.

[perry-the-pr-reviewer]

The skillsVariant field is still accepted by the schema but has no consumer anywhere in this PR (solver, harness, and dataset all ignore it). This was flagged in the previous review and remains unaddressed. Per the project's Minimal Interface Design principle ("write only the minimal required fields; don't speculate on future needs"), either wire it up (e.g., pass it as a branch/ref suffix when cloning skills) or remove it until it's needed.

▶ Prompt for agents: Remove skillsVariant from AgentDxOptionsSchema if it has no consumer, or add a // TODO: with the intended usage if it's planned for a follow-up PR.

[perry-the-pr-reviewer]

endpointId is passed through to the agent_dx config here, but makeAgentDxLayer in benchmark.ts:36 rejects it at layer-build time with a runtime error. The error surfaces late — after the CLI has already started the run. Consider omitting endpointId from the buildSchemaValidatedConfig call (or validating it here) so a user who passes --endpoint-id with --benchmark agent_dx gets a CLI-time error instead of a runtime failure.

▶ Prompt for agents: In the case "agent_dx" branch, either drop endpointId from the buildSchemaValidatedConfig call or add an explicit if (endpointId !== undefined) throw new Error(...) guard before it.