perf: amortize replay buffer compaction in ToolEventBroadcaster and ReusableReadableStream

[devin-ai-integration]

Summary

Port of typescript-sdk#798 onto the repo where this code actually ships. That PR was written against @openrouter/sdk's hand-written src/lib/{reusable-stream,tool-event-broadcaster,model-result}.ts, but openrouter-web's inference path reaches these classes through @openrouter/agent (packages/router/plugins/server-tools lazily imports @openrouter/agent/call-model), so the SDK copies are dead relative to production traffic and the fix has no effect there.

Two changes, both opt-in-safe.

1. Replay compaction behind a new streamReplay option. Both replay classes retained every event for the lifetime of a callModel() run, which is expensive for long generator-tool streams in a constrained runtime. Replay is also a compatibility contract — delayed and sequential stream getters must still start from event zero — so the tradeoff is now explicit at the call site. 'full' is the default and preserves current behavior exactly; 'active-consumers' releases entries once every attached consumer has advanced past them.

Consumer positions become absolute against a logical buffer head, so the common case is O(1) and the physical copy is amortized rather than quadratic:

physicalIndex = bufferHead + consumer.position - trimOffset

// after every successful read, and after return()/throw() deregisters a consumer
const nextHead = bufferHead + minConsumerPosition - trimOffset;
if (nextHead <= bufferHead) return;
trimOffset = minConsumerPosition;
buffer.fill(undefined, bufferHead, nextHead);        // release referents now
if (nextHead >= 1024 && nextHead * 2 >= buffer.length) {
  buffer = buffer.slice(nextHead);                    // amortized physical copy
  bufferHead = 0;
} else {
  bufferHead = nextHead;                              // O(1) common case
}

Clearing slots is what frees payload memory; the slice() only bounds backing-array length. Reading a cleared slot is unreachable under the position invariant, so it throws rather than ending the stream — a silently truncated response is the worst available failure mode here.

2. Terminal response events end the stream. A provider can send an authoritative response.completed / response.failed / response.incomplete without closing the HTTP body. The pump now buffers and broadcasts that event, marks itself complete, cancels the source reader at most once, and releases the lock exactly once; cancellation failure is cleanup-only and never overrides the provider's terminal result. Tool execution can therefore continue instead of waiting for an EOF that never arrives.

Divergence from the SDK PR

model-result.ts has diverged heavily between the two repos (~7.2k lines here), and this copy already captured terminal events in its initial pipe. Rather than paste the SDK's version over it, only the missing pieces were added, expressed against agent-only machinery that the upstream PR never had to consider:

• initialResponse / initialResponseError cache the terminal result so aggregate getters stay correct once history is trimmed, and getInitialResponse() emits the pending model call before returning a cached response — otherwise the PostModelCall hook would silently stop firing on that path.
• extractCompletionFromBuffer / tryExtractCompletionFromBuffer teardown paths (including getUsage()) now consult that cache first, because in 'active-consumers' mode a backward buffer scan can no longer find a terminal event that has already been released.
• streamReplay is client-only, so it is stripped in both places: clientOnlyFields in async-params.ts for the async path, and the already-resolved destructure in model-result.ts for reconstructed follow-up request bodies.

API example

import { callModel } from '@openrouter/agent';

const result = callModel(client, {
  model: 'openai/gpt-4o',
  input: 'Summarize this document.',
  // was: full replay history retained for the lifetime of the call (still the default)
  // now: opt into releasing events every attached consumer has passed
  streamReplay: 'active-consumers',
});

Landing this is also a prerequisite for deleting the SDK-side copies, which are being removed separately.

How I verified

pnpm lint, pnpm typecheck, and pnpm test in packages/agent: 98 files, 1168 tests passing, no type errors. New coverage includes default full replay for sequential and post-completion consumers in both classes, active-consumer watermark behavior and repeated amortized compaction, active-mode aggregate response/tool/error caching, PostModelCall firing exactly once on the cached-terminal path, usage still resolving after an active-mode consumer has drained the terminal event, absence of streamReplay from captured initial and follow-up request JSON, open bodies ending at completed/failed/incomplete events, and exactly-once cancellation plus reader-lock release.

The 50k-event drain measurements in the SDK PR were taken against the SDK files and have not been re-run here, so treat them as indicative rather than as evidence for this diff.

Link to Devin session: https://openrouter.devinenterprise.com/sessions/140e41de7e104f599a273d35fadbdfe4
Requested by: @mattapperson

[devin-ai-integration]

Original prompt from matt.apperson

SYSTEM:
=== BEGIN THREAD HISTORY (in #eng-devex-team) ===
sam (U06DJ8YS066) [ts=1786558308.342879]: any thoughts on this patch? l think src/lib/ is where we put our non-generated stuff right?
<https://openrouter.slack.com/archives/C0A3LMAP2D6/p1786497807373249?thread_ts=1786370592.446849&amp;amp;cid=C0A3LMAP2D6|https://openrouter.slack.com/archives/C0A3LMAP2D6/p1786497807373249?thread_ts=1786370592.446849&amp;amp;cid=C0A3LMAP2D6>

[Slack unfurl — this is an automatic link preview, not a user message]
Quote of conversation (https://openrouter.slack.com/archives/C0A3LMAP2D6/p1786497807373249?thread_ts=1786370592.446849&amp;amp;cid=C0A3LMAP2D6):
> From sam
> solid one in our ts sdk to start -- should help the server tools reqs
> > Replace per-watermark front splice() in both agent replay buffers with immediate slot clearing plus amortized physical compaction. At 50,000 events, representative drain medians fall from 66–902 ms to 2–8 ms, removing matching scheduler/fetch starvation
> <https://github.com/OpenRouterTeam/typescript-sdk/pull/798|github.com/OpenRouterTeam/typescript-sdk/pull/798> -- need to see if this is custom code or generated still

> ATTACHMENT:"https://openrouter.devinenterprise.com/attachments/ea5bc446-6807-49e3-9b9f-c76de8771f70/image.png"
> Posted on August 12, 2026 at 01:23 AM

Matt Apperson (U096UCX4GDS) [ts=1786563815.345379]: This is generated so something we should raise to <#C0991STJE1G>

sam (U06DJ8YS066) [ts=1786565482.618679]: mm gotcha thanks! will post it over there

sam (U06DJ8YS066) [ts=1786565660.175509]: hm wait so these are all generated? i thought callModel / modelResult and that stuff was custom

ATTACHMENT:"https://openrouter.devinenterprise.com/attachments/47977d5d-cbe6-41e8-838a-a4a9dd9f8753/image.png"

Matt Apperson (U096UCX4GDS) [ts=1786566714.123509]: Ah we have some cruft here @Christine Chen .

Call model and model result are in the typescript-agent repo now.

Christine Chen (U0B8Q... (1388 chars truncated...)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring