fix(runner): never re-execute an already-run tool call in SessionToolRunner

[PranavMishra28]

What was broken

SessionToolRunner (the implementation behind client.beta.sessions.events.tool_runner()) can execute a tool call twice for the same tool_use_id when a stream reconnect happens after a result-post attempt has permanently failed (or exhausted SEND_RETRIES). For a side-effecting tool — bash, a file write, anything registered on agent_toolset — this is an at-least-once instead of at-most-once execution hazard.

Root cause

_reconcile() runs on every stream reconnect (including after any ordinary transient disconnect — network blip, load balancer timeout, a redeploy) and re-enqueues every agent.tool_use / agent.custom_tool_use whose id is not yet in self._answered. self._answered is only populated on a successful post, so a call whose post failed is indistinguishable, to _reconcile, from a call never dispatched at all — it goes back through _send_work, and _dispatch_loop hands it to _execute, which unconditionally re-runs the tool itself, not just the post.

The fix

Adds self._executed: dict[str, tuple[DispatchedToolResultParams, bool]], populated with a call's computed result as soon as its tool has run (regardless of whether the subsequent send succeeds), and cleared once the send is confirmed. _dispatch_loop checks it before calling _execute: an id already in _executed is routed to a new _resend (retries posting the cached result only) instead of _execute (which would re-run the tool). This preserves the runner's existing self-healing intent — a failed post keeps getting retried on the next reconcile, per the pre-existing code comment — while removing the unsafe re-execution. _execute's and _resend's shared tail (yielding the DispatchedToolCall to the consumer) is factored into _post_result.

Also updated DispatchedToolCall.posted's docstring to describe the new retry-until-success behavior (the consumer may now see a second yield for the same tool_use_id with posted=True after an initial posted=False).

How it's tested

Reproduced deterministically with the existing FakeAsyncEvents fake-event harness in tests/lib/tools/test_session_runner.py — no live API needed:

• test_reconnect_does_not_re_execute_tool_after_failed_send (new): a permanent 4xx on the first send attempt, followed by a stream reconnect (transient disconnect + reconnect), previously ran the tool twice for the same tool_use_id; confirmed failing on main before the fix (counter["calls"] == 2). With the fix: the tool runs exactly once, the first yield reports posted=False, and the reconcile-triggered retry succeeds and yields a second DispatchedToolCall with posted=True and the same, not recomputed, result content.

Full verification run for this PR:

• uv run pytest tests/lib/tools/ — 188 passed, 1 skipped, 1 xfailed (no regressions; includes the pre-existing test_skips_already_answered_events, test_reconcile_list_error_does_not_dispatch_partial, test_yields_with_posted_false_on_permanent_4xx, and the concurrency/idle-clock tests, all still green)
• uv run ruff check . / uv run ruff format --check . on the changed files — clean
• uv run mypy . — clean (1052 source files)
• uv run pyright — 0 errors, 0 warnings

(tests/api_resources/ requires the local Prism mock server via ./scripts/test and 2 unrelated failures in tests/lib/test_credentials.py reproduce identically on clean main before this change — neither is touched by this diff.)

What this deliberately does not change

• Does not change the outer retry/backoff policy for the post itself (SEND_RETRIES, is_fatal_status_error in lib/_retry.py) — only which code path a reconcile-triggered redispatch takes.
• Does not address Managed Agents: intermittent 400 "does not match any agent.tool_use event" confirming a tool_use on a non-primary session_thread_id #1744 (an unrelated, separate report about session_thread_id routing for multiagent sub-thread confirmations) — I looked into that issue first but could not confirm the suspected fix without either live API access or a maintainer confirming server-side behavior for an undocumented field; flagging it here in case it's useful context, but this PR does not touch that code path.

Fixes #1749

Developed with Claude Code; reviewed and tested by Pranav before marking ready for review.

[PranavMishra28]

Small fix for a real at-most-once bug in the Managed Agents SessionToolRunner — a stream reconnect after a failed result-post could re-run the tool (so a bash/file-write call can fire twice), and this keeps the reconnect retrying only the post, with a deterministic regression test; would appreciate a look whenever someone has a cycle 🙏

[PranavMishra28]

@dtmeadows-ant flagging this your way since you last touched the session runner. small fix — it re-executes a tool on stream reconnect when the result post failed (so bash/file-write tools can run twice). CI's paused awaiting approval since it's my first PR here, so an approve-and-run would get the checks green. no rush on the review itself, thanks!

[kalyanamdewri]

Fault-injection result on 57f46e2bef99675a2405a51d495de44bfefe5ecb

For one logical call ID toolu_slot3_fixed_001, the tool executed 1 time(s) and result posting was attempted 4 time(s) across 1 reconnect(s).

Cancellation during tool result submission after execution: one logical submission failure exhausted the default three physical send attempts, then the SSE stream disconnected left 0 background asyncio live tasks (asyncio.all_tasks() excluding one harness task; 2 threads in-loop and 1 after loop close) and 0 tracked custom open transports (one was created and closed; reconnects are stream GETs minus one, not TCP connections) after 2000 ms. Base produced 2 executions, with the successful fourth POST carrying the recomputed execution:2 result; this PR produced 1 execution, with all four POST attempts carrying the cached execution:1 result.

This SUPPORTS at-most-once execution within one runner lifetime. I did not test process restart, so this is not evidence of durable exactly-once behavior.

Seed none-deterministic-fixture-v1; minimized trace 40 per PR run (41 per base run) events; artifact /tmp/fault_injection_1750/fault_injection_1750_artifact.tar.gz (sha256:2edb9de6a575f760562cfe336370bc5b31feffa5d214865dbcb05007544dc8ef).

[PranavMishra28]

appreciate the fault-injection @kalyanamdewri — at-most-once within a single runner lifetime is exactly the guarantee this is going for (the cached result gets re-POSTed on reconnect, the tool itself never re-runs), so that lines up with what you saw. and agreed, durable exactly-once across a process restart is a separate problem and out of scope here. thanks for taking the time to test it!