feat(benchmarks): custom_eval — declarative user evals as run-config data

[LukasParke]

Stacked on #1. Ports openrouter-web#31078 (all review rounds).

What

• EvalSpec v1 (src/benchmarks/custom-eval/spec.ts): a fully declarative eval — dataset (inline cases or an HF repo id, shape-validated to owner/name so path traversal can't reach the HF API), prompt template, system prompt, and a deterministic scorer (exact / contains / regex / numeric, with a schema-time refine rejecting pathological regex constructs).
• custom_eval benchmark: resolves entirely from the run config — user evals are rows, not registry code. Prompt templates substitute every {input} occurrence (replaceAll, function form so literal $ in inputs survives).
• HF revision pinning (src/datasets/huggingface.ts): optional pinned commit SHA verified against the dataset's current default-branch revision before streaming; mismatch fails closed with the observed SHA (the Dataset Viewer /rows endpoint only serves the default branch, so pinning is enforced by verification). Memoized — one dataset-info request per run.
• Config-bound dataset sizing: custom_eval's config-free makeDatasetLayer FAILS loudly instead of answering with a placeholder (an earlier 1-case stand-in silently truncated orchestrated runs to their first item — the worst finding of the review). New Benchmark.makeDatasetLayerForConfig + datasetSizeById(benchmarkId, config?) size the real dataset; sizing a config-bound benchmark without a config is a loud Left.

Testing

14 custom-eval tests (spec validation, renderPrompt, scorers, dataset layers), revision-pinning tests on the HF layer, registry coverage; full suite green.

[perry-the-pr-reviewer]

"## Perry's Review\n\nVerdict: 💬 Comments / questions\n\n> Note: The maintainer app is not installed on OpenRouterTeam, so I can't post an APPROVE. The review is clean — please approve manually once CI is green.\n\nRisk: 🟢 Low\n\n

\n

PR #2 — custom_eval: declarative user evals as run-config data

\n\nClean implementation of the declarative custom-eval system. The EvalSpec schema is well-structured with proper Zod validation, the scorer dispatch is comprehensive (exact, contains, regex, choice, numeric), and the HF dataset revision pinning is a thoughtful comparability guarantee.\n\nKey observations:\n\n- The renderPrompt function's use of replaceAll with a function form correctly preserves literal $ sequences in user input — a subtle but important correctness point.\n- The extractLastNumber function handles commas, signs, and decimals correctly.\n- The regex scorer validates patterns at parse time (not scoring time), preventing runtime crashes from malformed patterns.\n- The makeDatasetLayerForConfig / makeDatasetLayer split is the right design: config-bound datasets fail closed when probed without a config, preventing the silent single-case truncation the comment describes.\n- The HF revision verification (fail-closed on mismatch) is well-tested with both matching and divergent SHAs.\n\nOne inline question below.\n

\n"

[perry-the-pr-reviewer]

The template application guards on typeof message.content === "string" — if message.content is an array of content parts (multimodal input), the template is silently skipped for that message. Is this intentional for the rung-1 surface, or should non-string content raise an error so the user knows their prompt template isn't being applied?

▶ Prompt for agents: If intentional, consider adding a brief comment noting that multimodal content is not template-rendered in rung-1, so future readers don't treat the silent skip as a bug. If not intentional, throw on non-string content when a promptTemplate is configured.