feat(analyst): add prime as a first-class analyst-benchmark runner

[drewstone]

What

--analyst prime joins dspy-rlm|direct in the analyst benchmark: an OpenAI-compatible-bridge runner (createPrimeBenchmarkRunner) that drives the prime-agent RLM harness through cli-bridge on the same CodeTraceBench rows, scored by the same Hungarian-assignment grader. New --bridge-url flag; model form prime/zai/glm-5.2.

The protocol is a faithful port of the externally-proven rig (30+ live cases at time of writing): inline-JSON viewTrace projection with a chunked viewSpans fallback for oversized traces, the short-strings block contract (measured stream-splice corruption on long strings — rationale field removed; the scorer never read it), one bounded repair turn that carries the malformed reply but never the trajectory, typed failure observations, per-observation primeAnalystProtocolSha256. Two deliberate deviations, both documented in-code: no machine-local mutex (the benchmark's own run lock governs), and span enumeration from the prepared store (which already carries the appended verification artifacts) rather than raw OTLP.

docs/prime-analyst.md documents the exact same-rows reproduce commands (--analyst prime vs --analyst dspy-rlm), bridge prerequisites, and protocol rationale; linked from docs/trace-analysis.md.

Proof

• New runner tests 13/13 (fake transport; happy/repair/fail/oversized/usage-uncaptured/HTTP/deadline paths); command e2e through the dependency seam with the real runner (28/28).
• Analyst suite 548 passed | 1 skipped; full suite 4770 passed | 3 skipped; typecheck + lint + build clean; implementation-digest check green; merge-tree clean into main.
• dspy-rlm/direct paths byte-unaffected: zero pre-existing assertions changed; receipts byte-identical via conditional spreads.
• Zero live-LLM surface in tests (verified by an independent adversarial pass).

Verifier notes (accepted, fail-safe direction)

Two minor receipt drifts vs the external rig, both with raw bridge usage preserved in metadata: partial token reports collapse to uncaptured (never a silent zero), and the bridge's estimated flag lives in metadata.bridgeUsage rather than the receipt. A prime response-cache for --resume of in-flight cases is a named follow-up.

[tangletools]

✅ Auto-approved drewstone PR — 26c1cb08

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-08T03:42:59Z}

[tangletools]

🟠 Value Audit — better-approach-exists

Verdict	better-approach-exists
Concerns	2 (1 medium-concern, 1 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	134.4s (2 bridge agents)
Total	134.4s

💰 Value — better-approach-exists

Adds a third scored analyst arm (RLM agent via OpenAI-compatible bridge) cleanly into the existing benchmark plumbing, but reimplements the package's own OpenAI chat-completions client + fenced-JSON extraction instead of reusing it.

• What it does: Introduces --analyst prime (codetracebench-only): a new createPrimeBenchmarkRunner that sends the CodeTraceBench task + inline-JSON trajectory to a cli-bridge /v1/chat/completions, parses fenced {answer,blocks} JSON, optionally does one bounded repair turn, and expands accepted blocks into scored findings via the shared expandCodeTraceFailureBlocks — scored by the same Hungarian grader a
• Goals it achieves: Make the prime-vs-dspy-rlm comparison reproducible from this repo alone, on identical selected rows, with byte-unaffected receipts for the existing two arms — so the RLM coding agent can be scored as a first-class arm of the analyst benchmark rather than only on an external rig.
• Assessment: Mostly in-grain: it slots behind the existing AnalystBenchmarkRunner interface, reuses the shared block-expansion/scoring/comparison path, and the type-widening + fail-loud requireModelOwnerSettings narrowing keeps dspy-rlm/direct receipts byte-identical via conditional spreads. The prime runner's contract design (short-strings rule, raw-usage-in-metadata, honest-null-vs-failure, per-observati
• Better / existing approach: The repo already has a canonical, heavily-tested OpenAI /chat/completions client (src/llm-client.ts:573-868 callLlm) and a provider-neutral ChatClient that ships a literal cli-bridge transport (src/analyst/chat-client.ts:34,79-83,131-140). The prime runner instead reimplements the POST+status+JSON-parse+content/usage-extraction (benchmark-runner-prime.ts:142-186 bridgeCall, `:287-322
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound-with-nits

A coherent, well-fitted third analyst arm that reuses the existing runner/scoring/expansion machinery and adds minimal new surface (a cli-bridge transport + output contract); ahead of its first live caller, which is fine.

• Integration: Fully reachable: --analyst prime is parsed (benchmark-command.ts:569-605), dispatched by cli.ts:160, and the default createAnalystRunner factory (benchmark-command.ts:226-243) constructs the prime runner when config.analyst==='prime'. createPrimeBenchmarkRunner is exported through the public barrel (index.ts:154-168 via benchmark-real-model.ts:36-44), and the new file is registered in th
• Fit with existing patterns: Follows the established runner pattern exactly: same AnalystBenchmarkRunner<AnalystRunInputs> interface as createPublicBenchmarkRlmRunner/createPublicBenchmarkDirectRunner, same expandCodeTraceFailureBlocks expansion (benchmark-runner-prime.ts:251), same Hungarian-assignment scoring. The new PublicAnalystBenchmarkModelSettings type (benchmark-public-types.ts:90-94) cleanly models 'prime
• Real-world viability: Robust on the non-happy paths: node:http transport with explicit AbortController deadline (benchmark-runner-prime.ts:150-170) deliberately avoids undici's 5-min header-kill; oversized-trace chunked viewSpans fallback fails loud on any dropped span (:399-419); usage-merge treats uncaptured as poisoning rather than silent zero (:646-657); malformed reply becomes a typed error observation identical i
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

🎯 Usefulness Audit

🟡 Default bridge transport is http-only; a TLS bridge fails with a misleading TypeError [robustness] ``

nodeHttpPrimeBridgeTransport throws bridge URL must be http: for any non-http protocol (benchmark-runner-prime.ts:291). This is defensible for the documented localhost bridge (docs/prime-analyst.md only references http://localhost:4181) and the rationale is avoiding undici's header timeout, but a reviewer should confirm a remote/TLS bridge is genuinely out of scope. If one is ever wanted, HTTPS support would need its own deadline-safe transport rather than relaxing this check. Not gating — the

💰 Value Audit

🟠 Reimplements the package's own OpenAI client instead of extending it [duplication] ``

src/analyst/benchmark-runner-prime.ts:142-186 (bridgeCall), :287-322 (nodeHttpPrimeBridgeTransport), and :506-542 (extractReplyContent/extractRawUsage/extractJsonObject) collectively reimplement the OpenAI chat-completions POST, status/JSON handling, choices[0].message.content extraction, usage parsing, and fenced-JSON recovery that already live in src/llm-client.ts (callLlm at :573-868, extractJsonPayload at :502). The provider-neutral ChatClient even has a dedicated cli-bridge

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260808T034705Z}