feat(benchmarks): chess — full games vs Stockfish, first-party original

[LukasParke]

Stacked on #3. Ports openrouter-web#32320 (all four review rounds).

What

OpenRouter's first first-party-original game benchmark: the model plays full SAN games against depth-2 Stockfish and never sees a board after move 1 — position retention is measured over a whole adversarial game, not per-puzzle.

• Five tasks (CHESS_TASKS): standard opening (white/black), fixed-opening, strict-SAN, and an endgame-conversion task that requires actual checkmate (a winning eval scores 0 without the mate).
• Deterministic scoring: game points (1/0.5/0), ACPL vs the engine's preferred move (per-ply depth-10 evals from the model's perspective), blunder count (≥200cp), illegal-move rate. Engine-tournament adjudication: −900cp for 3 consecutive model turns = adjudicated loss; ±400cp at the ply cap decides win/loss/draw.
• Per-turn cost ledger (TurnCost): every model call records its generation id + token usage + cost, so cost joins to billing per iteration — never only a self-reported aggregate (the WSJ cost-reporting requirement).
• Robustness (the four review rounds):
  • all engine interactions fail as typed SolverError → one degraded game, never a torn-down fan-out; fail-closed at benchmark start if Stockfish is missing (STOCKFISH_PATH), before any model spend
  • verify-probes bounded per turn (a model that only ever asks to verify forfeits)
  • one engine analysis per model ply (cpLoss and the ply record share the search)
  • strict mode requires the reply to round-trip to canonical SAN (chess.js's lenient parser no longer lets e2e4 through unpenalized)
  • engine bestmove validated on a probe board before applying; UCI timeouts poison the engine (stale info/bestmove lines can never answer a later search); aspiration-window lowerbound/upperbound lines skipped
  • interrupt-proof cleanup: engines quit via an ensuring() outside the spawning effect; stdin errors can't become uncaught exceptions
  • typed GAME_RESULTS union (mistyped outcome = compile/schema error); full untruncated transcripts; maxTokens/reasoningEffort/timeoutMs passthrough; fixed-temperature config (supplying one is a schema error — solver pins T=0)
• Concurrency: engine pair per game, Threads=1, deterministic evals; validated in the monorepo at 10 concurrent games with zero leaked processes.
• New dependency: chess.js@1.4.0 (board state/legality; the engine does all evaluation).

Testing

19 chess unit tests (extraction, strict SAN, adjudication, scorer thresholds, schema round-trips incl. TurnCost); full suite 1226 pass.

[perry-the-pr-reviewer]

"## Perry's Review\n\nVerdict: 💬 Comments / questions\n\n> Note: The maintainer app is not installed on OpenRouterTeam, so I can't post an APPROVE. The review is clean — please approve manually once CI is green.\n\nRisk: 🟢 Low\n\n

\n

PR #4 — chess benchmark, full games vs Stockfish

\n\nWell-designed chess benchmark with fully deterministic scoring (no judge model). The game state machine, UCI engine driver, and per-ply evaluation are robust and correctly handle edge cases (engine failures, timeouts, malformed bestmoves). The test coverage on the deterministic parts (move extraction, result classification, scorer, schema round-trip) is thorough.\n\nKey observations:\n\n- The UCI engine driver is well-engineered: poison-pill spawn failure, timeout-based dead marking, SIGKILL fallback in quit(), and per-game engine isolation for concurrency safety.\n- The chess solver's game loop correctly handles the full lifecycle: opening move, model turns with retry/forfeit, engine responses, adjudication (hopeless-loss and maxPlies cap), and terminal position detection.\n- The TurnCost ledger is a good design for per-iteration billing attribution — each model call gets its own cost row.\n- degradeSolverErrors: true correctly maps engine/sandbox failures to Skipped rather than aborting the run.\n- The endgame-conversion task (K+Q vs K, require-mate) is a nice discriminating test of position retention.\n\nEstimated impact: Low — the schema.ts file is a Zod validation schema for chess game records, not a database schema or migration. No auth/payment/migration surface touched.\n\nOne inline suggestion below about the generationId field.\n

\n"

[perry-the-pr-reviewer]

The chess solver reads output.generationId to build the per-iteration turnCosts ledger, but openrouter-model.ts's decodeResult never includes generationId on the returned ModelOutput — it calls recordGenerationId(result.id) (fiber-scoped collector) but omits result.id from the succeed({...}) return. So turnCosts[].generationId will always be undefined in production when using the OpenRouter provider. The test data hardcodes generationId: "gen-abc" which masks the gap.

To make the billing-join feature functional, add ...(result.id !== undefined && { generationId: result.id }) to the succeed({...}) in decodeResult (openrouter-model.ts ~line 470). The responses-model.ts provider has the same gap.

▶ Prompt for agents: Wire result.id into the ModelOutput returned by decodeResult in openrouter-model.ts so the chess solver's per-iteration cost ledger can join to billing records. Verify the responses-model.ts provider too if it's used for chess.