Stream Codex HTTP responses incrementally

[mulfyx]

Stream Codex HTTP responses incrementally

Summary

• decode Codex Responses SSE incrementally on the HTTP transport
• forward translated Anthropic events to Claude Code before upstream completion
• emit Anthropic ping events during startup and silent reasoning gaps
• preserve the retry window across control-only events such as
  response.created and response.in_progress
• retry retryable HTTP/SSE failures only before semantic model output is exposed
• stop upstream reads and retry backoff when the downstream client disconnects
• preserve HTTP fallback, continuation, and opt-in traffic-capture behavior

Root cause

We found the original streaming bug while diagnosing long-running Claude Code
background agents through the proxy. The HTTP transport waited for
response.bytes() to collect the complete Codex response before parsing its SSE
events. Codex was streaming correctly, but Claude Code received the translated
body only after the model finished, so an active agent appeared stuck and then
delivered its output in one burst.

The initial incremental implementation exposed a second boundary bug. A Codex
attempt can emit control events and later fail inside an HTTP 200 SSE stream.
Once response.created had produced an Anthropic message_start or heartbeat,
the proxy treated the downstream response as committed. A later retryable
response.failed was therefore translated into an Anthropic api_error
instead of using the existing bounded retry policy.

We reproduced this with a real upstream overload response:

response.created
response.in_progress
response.failed: Our servers are currently overloaded. Please try again later.

The buffered HTTP implementation did not have this regression because it could
inspect the complete upstream response before opening the downstream body.

Implementation

The HTTP transport now owns a restartable incremental SSE reader.

• Control-only events can continue producing downstream heartbeats without
  closing the retry window.
• Stateful non-semantic events are held until the attempt produces semantic
  output or completes successfully.
• Overload, rate-limit, transient 5xx, body timeout, connection failure,
  malformed/truncated SSE, and pre-terminal EOF can restart the upstream
  attempt while no semantic output has been exposed.
• Pending attempt-local state is discarded before retrying.
• The existing retry cap, backoff, Retry-After, auth refresh, and request body
  are reused.
• The downstream Anthropic stream remains one logical message: retries do not
  emit duplicate message_start or message_stop events.
• Once thinking, text, or tool-call output has been forwarded, full-request
  retry is disabled to avoid duplicate model output or tool execution.
• Dropping the downstream response cancels both an active upstream read and a
  pending retry delay.

Comparison with Codex CLI

This was implemented independently, but the current Codex CLI source confirms
the same broad separation between SSE parsing and turn-level recovery:

• the Responses SSE parser classifies response.failed and extracts a
  server-requested rate-limit delay:
  https://github.com/openai/codex/blob/5d325ba2234ead1a6c03313de5d6ecf8df015f1a/codex-rs/codex-api/src/sse/responses.rs#L386-L415
• an SSE error, idle timeout, or EOF before response.completed becomes a
  stream error:
  https://github.com/openai/codex/blob/5d325ba2234ead1a6c03313de5d6ecf8df015f1a/codex-rs/codex-api/src/sse/responses.rs#L491-L526
• the outer sampling loop applies bounded retry/backoff to retryable stream
  errors:
  https://github.com/openai/codex/blob/5d325ba2234ead1a6c03313de5d6ecf8df015f1a/codex-rs/core/src/session/turn.rs#L1141-L1208
• Codex has an integration test proving that an SSE stream closed before
  response.completed is retried as a new request:
  https://github.com/openai/codex/blob/5d325ba2234ead1a6c03313de5d6ecf8df015f1a/codex-rs/core/tests/suite/stream_no_completed.rs#L23-L101

There is one relevant current difference. Codex main still excludes
ServerOverloaded from its generic retry set:

https://github.com/openai/codex/blob/5d325ba2234ead1a6c03313de5d6ecf8df015f1a/codex-rs/protocol/src/error.rs#L176-L204

However, openai/codex#31058 proposes dedicated model-capacity recovery with a
separate bounded retry budget, long backoff, and cancellation-aware waits:

• PR: fix(core): retry model capacity errors openai/codex#31058
• retry policy and cancellation:
  https://github.com/openai/codex/blob/49b5b721c12dc1ae674abe47a347baf7f28e82d1/codex-rs/core/src/responses_retry.rs#L16-L102
• separate capacity counter in the turn loop:
  https://github.com/openai/codex/blob/49b5b721c12dc1ae674abe47a347baf7f28e82d1/codex-rs/core/src/session/turn.rs#L1160-L1235
• same-turn recovery test:
  https://github.com/openai/codex/blob/49b5b721c12dc1ae674abe47a347baf7f28e82d1/codex-rs/core/tests/suite/client.rs#L3528-L3612

CCP cannot copy Codex's boundary exactly because CCP has already opened an
Anthropic-compatible downstream stream. Its additional semantic-output guard is
what makes restarting safe: control frames do not close the retry window, but
real model output does.

Independent prior art

After diagnosing the original buffering problem, we checked other proxy
implementations and found the same incremental-forwarding and heartbeat pattern:

• codex-proxy forwards translated chunks immediately and emits heartbeats only
  during inactive gaps:
  • https://github.com/icebear0828/codex-proxy/blob/899d540410078a335c32dccd29b8f59e7d2d3292/src/routes/shared/response-processor.ts#L90-L108
  • https://github.com/icebear0828/codex-proxy/blob/899d540410078a335c32dccd29b8f59e7d2d3292/src/routes/shared/response-processor.ts#L126-L152
• CLIProxyAPI selects between upstream chunks and a keepalive ticker, flushing
  after every chunk or heartbeat:
  https://github.com/router-for-me/CLIProxyAPI/blob/06a4e46f693b4abbd0031dc7e042df3c51771feb/sdk/api/handlers/stream_forwarder.go#L64-L119
• auth2api parses partial SSE input and writes transformed chunks without
  collecting the complete body first:
  https://github.com/AmazingAng/auth2api/blob/a34c011f9fda1013ff3f9299160694c2ab62e4db/src/upstream/streaming.ts#L125-L229
• a separate Claude Code proxy report describes the same user-visible buffering
  symptom:
  [BUG] claude code 接入 cliproxyapi 使用时，模型的输出没有呈现流式，而是一下子蹦出来回答结果 router-for-me/CLIProxyAPI#1620

These projects support the incremental-streaming mechanics; they were not used
as evidence that semantic-aware retry was already implemented elsewhere. No
source code was copied from them.

Verification

• cargo test --lib: 466 passed
• cargo test --test smoke_cutover: 25 passed
• rustfmt --edition 2024 --check passed for the modified Rust files
• git diff --check passed
• regression coverage verifies:
  • first downstream output arrives before upstream completion
  • overload after control events retries successfully
  • rate-limit and transient 503 failures retry successfully
  • pre-semantic EOF retries successfully
  • retry exhaustion remains bounded
  • overload after semantic output is not retried
  • downstream cancellation prevents another upstream attempt
  • one logical Anthropic message_start is emitted across retries
• a temporary real CCP process backed by a mock Codex endpoint completed the
  overload-to-success retry through curl
• the current implementation was rebuilt into the local Alpine image and is
  running healthy on 127.0.0.1:18765

Temporary diagnostic logging used to isolate the bug is intentionally not part
of the PR.

[mulfyx]

Moving this back to draft while the implementation is being finished. The branch still needs to be rebased onto the latest main, and the HTTP streaming retry boundary needs additional work so retryable upstream failures are not exposed after an early progress or heartbeat frame. The PR will be updated once the fix and regression coverage are ready.