feat(harness): durable per-(sample, epoch) result store + fiber-scoped request context - #1
feat(harness): durable per-(sample, epoch) result store + fiber-scoped request context#1LukasParke wants to merge 1 commit into
Conversation
…d request context Ports the monorepo evals-product stack's harness plumbing onto the standalone layout (openrouter-web#31077 review rounds included): - SampleResultStore: durable per-(sample, epoch) records with a pinned wire schema; completed entries seed the accumulator on retry so a re-run chunk reproduces the same aggregate. Degraded (error-synthesized) scores are written only at end of run and ignored by the resume skip-list, so retries re-attempt them for free. Failed writes fail the run after bounded retries — a successful run guarantees durable records. - request-context: fiber-scoped (sampleId, epoch) identity, stamped into each request's x_bench extension by the OpenRouter model layer, so a budget-gateway can key request coalescing per epoch (multi-epoch runs must not replay one epoch's answer). OpenRouter ignores the field on direct calls. - wandr primary score/reward mean exclude Skipped (infrastructure- degraded) samples, matching aggregateScores' accuracy denominator. - runBenchmarkById accepts an optional sampleResultStore (noop default).
![perry-the-pr-reviewer[bot]](https://avatars.githubusercontent.com/in/3222961?s=60&v=4){kind=small}
There was a problem hiding this comment.
"## Perry's Review\n\nVerdict: 💬 Comments / questions\n\n> Note: The maintainer app is not installed on OpenRouterTeam, so I can't post an APPROVE. The review is clean — please approve manually once CI is green.\n\nRisk: 🟢 Low\n\n
PR #1 — durable per-(sample, epoch) result store + fiber-scoped request context
\n\nClean port of the harness plumbing onto the newsrc/ layout. The durable result store, fiber-scoped request context, and degraded-outcome handling are well-designed and thoroughly tested (13 resume tests covering skip/resume, degraded persistence, retry, and failure modes).\n\nKey observations:\n\n- The SampleResultStore abstraction is clean: the encode/decode round-trip is fully tested, the rest-satisfies pattern on SampleScore/ModelUsage/Score provides compile-time exhaustiveness, and the pinned format_version prevents silent schema drift.\n- The x_bench fiber context correctly stamps per-(sample, epoch) identity into model requests so the gateway's coalescing hash separates epochs. The unsafeMake FiberRef is the standard Effect pattern.\n- The degraded-outcome lifecycle is well-reasoned: degraded records are deferred to post-stream, marked in the record, ignored by the retry skip-list, and counted by the finalization fold. The Skipped score correctly excludes infrastructure failures from the accuracy denominator.\n- The WANDR scorer change (excluding Skipped from the reward mean/weight) is the right semantic alignment with the new degraded handling.\n\nEstimated impact: Low — new benchmark harness infrastructure, no auth/payment/migration surface. The SampleResultRecordSchema is a Zod validation schema for benchmark result records, not a database migration.\n\nOne inline question below.\n| }; | ||
|
|
||
| /** Persist the run's degraded outcomes, deferred to after every sample has settled. */ | ||
| function persistDegradedOutcomes( |
There was a problem hiding this comment.
The persistDegradedOutcomes function calls writeEntry for each degraded outcome, and a persistent write failure fails the entire run. But degraded outcomes are already in the fold accumulator (they passed through accumulateOutcome during the stream), so the run result is already computed at this point. The only consequence of not persisting a degraded record is that a future activity retry re-runs that sample — which is the desired behavior for degraded records (the skip-list ignores them anyway). Is failing the run here intentional, or should a degraded-write failure degrade to a warning (log + continue) rather than a hard run failure?
▶ Prompt for agents: If this is intentional, consider adding a one-line comment noting that degraded persistence is fail-closed by design (consistent with the non-degraded contract) so future readers don't assume it's an oversight.
1 participant
Ports the harness-side plumbing of the openrouter-web evals-product stack (PR #31077, all review rounds) onto the standalone layout. First PR of a 4-PR stack: harness-plumbing → custom-eval → manifest-schema → chess-benchmark.
What
SampleResultStore(src/harness/sample-result-store.ts): durable per-(sample, epoch) records with a pinned wire schema (SampleResultRecordSchema, format-versioned).runBenchmarkskips already-recorded pairs and seeds the accumulator from them, so a retried chunk reproduces the same aggregate as an uninterrupted run.degraded— durable for finalization folds, but ignored by the resume skip-list so a retry re-attempts them for free.request-context(src/harness/request-context.ts): fiber-scoped (sampleId, epoch) identity, stamped into each outgoing request'sx_benchextension by the OpenRouter model layer. A budget gateway keys request coalescing on it so multi-epoch runs get fresh answers per epoch (same-epoch retries still coalesce). OpenRouter ignores the field on direct calls. Same FiberRef pattern asgenerationIdCollector.Skipped(infrastructure-degraded) samples, matchingaggregateScores' accuracy denominator.runBenchmarkByIdaccepts an optionalsampleResultStore(noop default); new package exports./sample-result-store,./request-context.Why here
openrouter-web vendored this repo as a read-only subtree (openrouter-web#32329), so harness changes land upstream first. The monorepo side (bench-gateway, Temporal wiring, GCS store implementation) stays in the openrouter-web stack and consumes these exports after a
subtree pull.Testing
bun test— 1226 pass (12 new resume tests, store schema round-trip tests);tscclean; oxlint clean.