feat(compass-e2e): script canned-model tool_calls turns for the dogfood harness (SEA-1787 gap-2) - #208
Open
seal-agent wants to merge 2 commits into
Open
feat(compass-e2e): script canned-model tool_calls turns for the dogfood harness (SEA-1787 gap-2)#208seal-agent wants to merge 2 commits into
seal-agent wants to merge 2 commits into
Conversation
…od harness (SEA-1787 gap-2)
The dogfood e2e canned-model backend served a single fixed text reply on
every request, so it could only exercise a one-turn leg. The H4 legs-3/4
scenario needs the agent to make several model round-trips (emit a spawn
tool-call, observe the result, then settle on a closing text turn), which the
single-reply backend cannot express.
Replace the single `reply string` with an ordered `script []CannedTurn` the
server serves one turn per request (request N settles on script[N]):
- `CannedText(reply)` — a pure-text turn (delta.content + finish_reason "stop"),
the pre-script behavior expressed as one turn.
- `CannedToolCall(toolName, argsJSON)` — a single tool-call turn: one
delta.tool_calls entry (deterministic call_<n> id, type "function", the
arguments already serialized as the OpenAI JSON string) + finish_reason
"tool_calls", which the SDK's mapStopReason maps to stopReason toolUse the
agent loop gates tool execution on.
A request past the end of the script is a test bug: the handler answers HTTP
500 naming exhaustion (before writing the 200 stream header) rather than
hanging or serving a default turn. An empty script is a construction error.
`WithCannedModel(reply)` stays a single-text-turn convenience so the existing
H2 leg-2 caller is untouched; `WithCannedScript(...)` is added for the
multi-turn H4 callers. Both set the same underlying script field.
Grounded firsthand against the openai-completions provider in the SDK fork:
delta.tool_calls carries {index, id, type, function{name, arguments}} with
arguments a JSON string parsed in one shot, and finish_reason "tool_calls"
drives the tool-execution gate — so one chunk with the complete arguments plus
a terminal finish chunk plus [DONE] is a well-formed tool-call turn.
Tests (hermetic, no container, ctx-bounded, no sleeps/polls): the tool-call
wire shape, the per-request multi-turn advance, loud 500 exhaustion, and the
empty-script construction guard; red-checked (a wrong finish_reason reddens
the tool-call tests).
Refs SEA-1787
Co-authored-by: Matt Wilkinson <matt@sealedsecurity.com>
…rn omitempty wire shape Review of PR #208 flagged two narrow test-adequacy gaps (both low, non-gating): - The tool-call turn test never asserted the turn carries no assistant content, so a regression that stamped delta.content on a tool-call turn (the SDK would then append spurious text alongside the call) would not redden. Assert turn.content == "". - The omitempty claim on chatDelta.ToolCalls ("a text turn serializes with no tool_calls key, not an empty/null array") was only functionally covered by len(toolCalls)==0, which cannot distinguish an absent key from a present-but- empty one. Capture each raw SSE frame and assert a text turn's frames contain no "tool_calls" substring, so dropping the omitempty tag reddens. Both assertions red-checked: dropping omitempty makes a text turn emit "tool_calls":null and the multi-turn test fails; stamping content on a tool-call turn fails the tool-call test. Refs SEA-1787 Co-authored-by: Matt Wilkinson <matt@sealedsecurity.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack containing 3 PRs:
mainThe dogfood e2e canned-model backend served a single fixed text reply on
every request, so it could only exercise a one-turn leg. The H4 legs-3/4
scenario needs the agent to make several model round-trips (emit a spawn
tool-call, observe the result, then settle on a closing text turn), which the
single-reply backend cannot express.
Replace the single
reply stringwith an orderedscript []CannedTurntheserver serves one turn per request (request N settles on script[N]):
CannedText(reply)— a pure-text turn (delta.content + finish_reason "stop"),the pre-script behavior expressed as one turn.
CannedToolCall(toolName, argsJSON)— a single tool-call turn: onedelta.tool_calls entry (deterministic call_ id, type "function", the
arguments already serialized as the OpenAI JSON string) + finish_reason
"tool_calls", which the SDK's mapStopReason maps to stopReason toolUse the
agent loop gates tool execution on.
A request past the end of the script is a test bug: the handler answers HTTP
500 naming exhaustion (before writing the 200 stream header) rather than
hanging or serving a default turn. An empty script is a construction error.
WithCannedModel(reply)stays a single-text-turn convenience so the existingH2 leg-2 caller is untouched;
WithCannedScript(...)is added for themulti-turn H4 callers. Both set the same underlying script field.
Grounded firsthand against the openai-completions provider in the SDK fork:
delta.tool_calls carries {index, id, type, function{name, arguments}} with
arguments a JSON string parsed in one shot, and finish_reason "tool_calls"
drives the tool-execution gate — so one chunk with the complete arguments plus
a terminal finish chunk plus [DONE] is a well-formed tool-call turn.
Tests (hermetic, no container, ctx-bounded, no sleeps/polls): the tool-call
wire shape, the per-request multi-turn advance, loud 500 exhaustion, and the
empty-script construction guard; red-checked (a wrong finish_reason reddens
the tool-call tests).
Refs SEA-1787
Co-authored-by: Matt Wilkinson matt@sealedsecurity.com