feat(trajectories): canonical trace IR with ACP and ATIF converters - #984
feat(trajectories): canonical trace IR with ACP and ATIF converters#984Galius5136 wants to merge 25 commits into
Conversation
`trajectory/acp_trajectory.jsonl` is produced by the ACP-session capture path and read by the viewer, the judges, the skill evaluator, task verifiers and the review skill — but it had no schema, no version and no written specification. Every consumer hand-rolls its own parser. This writes the format down and makes it machine-checkable, with zero runtime behaviour change: - `src/benchflow/trajectories/schemas/acp-capture-event-v1.schema.json` — Draft 2020-12 schema for one line of the artifact, scoped deliberately to the ACP-session capture events (`_events_to_trajectory` plus the `ACPSession` legacy fallback). `kind` stays an open string because `_canonical_tool_kind` passes agent-supplied values through unchanged; `status` is a closed enum because the serialized value is always `ToolCallStatus(...).value`. `additionalProperties: false` on every event variant, so a field the emitter does not produce today fails the suite. - `docs/trace-interop.md` — the format as emitted, per-event required and optional fields, producers and consumers, the current state of ATIF and OpenTelemetry, and a table of the information losses observable in the existing ACP -> ATIF / ADP conversions. Claims are marked FACT or PROPOSAL; nothing marked PROPOSAL is agreed or implemented. - `tests/trajectories/test_acp_capture_event_schema.py` — conformance suite over two corpora: records generated by driving real `ACPSession` objects through the production capture path and writer, and the ACP event lists already used as exporter inputs by the ATIF and ADP tests. Ten cases pin what the schema must reject. The event-type vocabulary is read out of `_events_to_trajectory` by AST rather than hardcoded, so adding a branch to that function fails the suite until the schema documents it. - `jsonschema` declared as a dev dependency. It was already resolved as a litellm transitive; declaring it keeps the test suite off another package's dependency tree. `uv.lock` gains the two corresponding lines. Scope is narrower than the artifact and the schema says so: oracle-mode rollouts replace the trajectory with an oracle-only list, and session-factory Sessions bypass the ACP-session emitter entirely via `_snapshot_session_trajectory`. Neither is modelled here. No artifact-level contract is defined or implied. No runtime module imports the schema. Nothing under `src/benchflow` changes behaviour. Verified: 25 new tests; tests/trajectories 239 passed; ACP regression (capture, streaming, acp, native usage) 147 passed; ruff check/format clean; `uv lock --check` and `uv sync --locked --extra dev` both pass.
Slice A completeFormalizes the ACP-session capture-event subset written to What landed
Design choices worth flagging for review
Human verification (run by @Galius5136, not by CI)
What this surfaced Preparing H2 exposed a real boundary that the first draft of the documentation So the same artifact path can hold different record families depending on how Not included, deliberately No canonical IR, no OTel work, no ATIF reader, and no capture-layer enrichment. |
Picks up d30527b (fix(acp): retain live subprocess stderr, benchflow-ai#980). No conflicts: that commit touches acp/container_transport.py, sandbox/process/* and their tests, none of which Slice A or the trajectories export path touches.
test_export_atif.py pins the shape ATIF export produces. This pins which
information survives the conversion and which does not, so a future change to
the converter either keeps the property or fails a test that says in one line
what changed.
Adds tests/trajectories/test_atif_preservation.py: 10 preservation invariants,
12 loss characterizations, 2 producer-boundary tests. No runtime module and no
public format is touched, and no dependency is added.
Two choices make the suite falsifiable rather than decorative. Losses are
asserted with sentinel values absent from the whole serialized document, not
with structural checks on the one field a test happened to look at. And both
sides of the oracle-source divergence are read from source via AST -- the
converter's event-type branches and the validator's accepted `source` set -- so
changing either fails a test instead of silently invalidating the premise.
The load-bearing producer-boundary test drives a real ACPSession with a
tool_call update carrying rawInput, rawOutput, locations and _meta and shows
that those fields are absent before the ACP -> ATIF converter runs. Real
rollout evidence independently shows non-empty tool arguments in provider
capture for calls whose ATIF arguments are {}. Together, these observations
place the loss upstream of the ATIF converter rather than inside it.
docs/trace-interop.md gains a per-field table (5.1), one previously unrecorded
loss -- the agent_thought join is irreversible, so one thought containing a
blank line is indistinguishable from two events -- and a section recording what
two real gemini rollouts showed (5.2), including that the proxy capture holds
the tool arguments the ATIF document records as {}.
Slice A2 — ACP → ATIF preservation/loss characterizationTurns the ATIF rows of the §5 loss table into executable assertions. 24 tests, one new file, no runtime change, no public format change, no new dependency — same profile as Slice A. Deliberately not Slice B. Slice B in the roadmap above is the canonical-IR prototype, and it is gated on open question 1. This work is a sibling of A: it holds whichever way that question is answered, and it gives a baseline of what the existing converter preserves before anything is redesigned.
Both sides of the One previously unrecorded lossDocumented as #10: What real rollouts showed§5.2 records two
The observation worth calling out: in those rollouts the proxy capture ( This is not evidence that the ACP Also observed and now documented: an ATIF document opens with two identical Not exercised by a real rolloutStill resting on code reading plus the synthetic tests, and labelled as such in the document: the non-text content-block loss (#5), because neither agent emitted a file-edit or terminal block; and the Still a draft — Slices B–F remain a proposal pending the open questions above. |
Four trace-shaped representations already exist here — the ACP-session capture
events, ATIF, ADP and the Verifiers/ORS record — and all three exporters walk
the same ACP event list from the same call site. The cost of that is not the
edge count; it is that each edge answers the same questions privately, and the
answers already diverge: for one `tool_call` event ATIF emits `"arguments": {}`
while ADP emits `"kwargs": {}`, ATIF keeps the tool status in a non-standard
`extra` while ADP drops it, both join thought boundaries irreversibly, and
neither represents `agent_timeout` at all.
This adds a canonical hub so each format becomes one edge against a written
contract, and so the information loss is a typed value rather than a comment in
a module docstring.
The direction is not approved. The four open questions in docs/trace-interop.md
§6 have had no maintainer answer, so this takes a provisional position on the
first one and implements it in isolation, where it can be reviewed as code and
reverted by deleting two files.
- `src/benchflow/trajectories/ir.py` — the IR types, the loss model
(`LossReport` / `LossRecord` over an unsupported/dropped/normalized/
synthesized taxonomy, the same one §5.1 already uses), and `validate_trace`,
which returns one string per invariant violation rather than raising.
The rule the module is built on is that the IR is a pragmatic superset of
what BenchFlow can observe, not a model of what an agent trace could
contain. Three consequences are load-bearing: optional values are tri-state
(a value / `None` "this source never carried it" / an empty value "carried
and empty"); a `None` that is not covered by a loss record makes the trace
invalid, so absence is declared and never silent; and normalization is
non-destructive — `source_type` keeps the source's own type string next to
the normalized `kind`, and `name_semantics` records that an ACP `kind` is a
category rather than a function name.
The IR deliberately cannot fabricate an agent version, a synthetic tool-call
id, a timestamp, or an OTel span id. Those are target-side obligations and
belong in converters, which record them as SYNTHESIZED.
- `tests/trajectories/test_trace_ir.py` — 25 tests. Each invariant is exercised
with a violating trace and a clean one, so a rule that stopped firing fails
here instead of passing silently. The IR's tool-status vocabulary is checked
against the ACP `ToolCallStatus` enum and its event kinds against the
vocabulary `_events_to_trajectory` actually emits, read from source by AST —
the mechanism the Slice A conformance suite already uses, so adding a branch
to the capture path fails this suite until the IR accounts for it.
Two tests pin the isolation claim rather than the design: no module under
`src/benchflow` imports the IR, and the IR imports no benchflow module. The
first is the executable form of "zero runtime behaviour change"; wiring the
IR into a run path has to update it deliberately.
- `docs/trace-interop.md` §8 — replaces the "ideas not yet agreed" placeholder
with the design: why a hub, the four alternatives considered and why each was
not taken, the field classes (supported today / optional / needs enrichment /
must not be invented), the planned ACP→IR and IR→ATIF mappings, an OTel
sketch marked unverified because no version of the GenAI conventions is
vendored here, the invariants, and an explicit list of what a review can
still reject. §6 gains a note that question 1 now has a provisional, unagreed
position and that questions 2-5 are untouched.
The worked example in §8.4 is generated from the models and compared against
the document by a test, so it cannot drift.
Not included, deliberately: `ACP → IR`, `ATIF ↔ IR`, any OTel work, any wiring
into a run path, any on-disk artifact. The representation is meant to be
reviewed before anything depends on it.
No existing format, exporter, artifact or code path changes. No new dependency
— pydantic is already a runtime dependency and `uv.lock` is untouched.
Verified: 25 new tests; tests/trajectories 288 passed; the trajectory, capture,
streaming and ACP regression lane 453 passed; `ruff check .` and
`ruff format --check` clean; `ty check src/` clean.
Slice B — provisional canonical Trace IRThe four open questions in the PR description are still unanswered. Rather than Feedback on the architecture is still very welcome, and it is still cheap to Why a hub rather than direct convertersFour trace-shaped representations already exist here — the ACP-session capture
Three independent decisions about the same event, taken three times, recorded The alternatives I considered and did not take — promoting ATIF or the ACP What is in the slice
The rule the module is built on: the IR is a pragmatic superset of what
The IR deliberately cannot fabricate an agent version, a synthetic Two tests pin the isolation claim rather than the design: nothing under The event-kind and tool-status vocabularies are checked against the real Deliberately not included
What a review can still changeEverything, and cheaply. §8.6 lists it explicitly — the hub itself, the What a review cannot change by rejecting the IR: the losses in §5 are properties Verification25 new tests; |
…nwired Slice B proposed a hub and a contract: every `None` in the IR is covered by a `LossRecord`, and a conversion's cost is a typed value rather than a comment in a docstring. This is the first real edge, and its job is as much to stress that contract as to convert. `acp_events_to_ir` reads the event list of `trajectory/acp_trajectory.jsonl` — the ACP-session capture vocabulary Slice A pinned, plus the oracle and unknown record families that share the file — and returns one `CanonicalTrace` carrying its own report. It reads no other artifact: `result.json`, `timing.json` and the proxy capture are separate, so a value that lives only there is a declared loss rather than a silent enrichment. What the conversion keeps that the existing exporters lose: - `agent_timeout` becomes an event with its fields in `extensions`, and sets the trace outcome. Every exporter drops it today (§5 loss benchflow-ai#4). - Non-text content blocks are carried as `opaque` with the block verbatim, instead of being skipped by `content_blocks_to_text` (loss benchflow-ai#5). - Thought boundaries survive: one capture record is one reasoning segment, and nothing is joined, so the ambiguity `ThoughtBuffer` creates never arises (loss benchflow-ai#10). A thought whose own text contains a blank line stays one segment, because splitting it would invent a boundary the source does not have. - `""` and absent stay distinguishable everywhere; text-empty events are kept rather than dropped. - The `oracle` record keeps its own kind and role instead of becoming an agent step prefixed `[oracle: …]` that a consumer can only undo by string matching. - An unrecognized `type` becomes `unknown` with the record carried verbatim, instead of being skipped silently. What it refuses to invent: arguments, timestamps, tool-call ids, agent version. It also does not prepend the `prompts` argument as leading user events, which both existing exporters do — §5.2 showed the cost, an ATIF document that opens with two identical `user` steps so user turns over-count by one. Those steps are not ACP events; a target that wants them adds them at its own edge as SYNTHESIZED. Loss addressing distinguishes three shapes. `events[i].…` is a field of one IR event, and is what `validate_trace` matches, so the per-call `arguments` records use it. `source[i]` is an input entry that produced no IR event, which cannot be addressed as `events[i]` because that index belongs to a different event once an entry is skipped. Systemic losses — timestamps, per-event usage, agent version, stop reason — are declared once each under an unindexed `events[].…` path. That last choice is what makes the contract affordable, and it is measured rather than asserted: the report is `n_tool_calls + 5` records and does not grow with trace length. On the two real gemini rollouts of §5.2 it is 7 records (H1, 2 tool calls) and 6 (H2, 1 tool call and a real wall-clock timeout). - `tests/trajectories/test_ir_from_acp.py` — 35 tests. Preservation is checked against events produced by driving a real `ACPSession` through the production capture path, reusing the Slice A2 fixture rather than hand-written dicts that would only prove the converter agrees with itself. The report is asserted as a complete set, so an undeclared loss and a spurious one both fail. One test removes a declared loss from a converted trace and shows the trace becomes invalid — the Slice B contract, demonstrated end to end. Two tests pin the volume property, including that doubling the non-tool events does not change the report at all. The block classifier is pinned against `content_blocks_to_text` so the IR and every existing consumer cannot disagree about what counts as text output. - `tests/trajectories/test_trace_ir.py` — the isolation test is restated at the boundary that now matters. Slice B asserted that nothing under `src/benchflow` imports the IR; the converter necessarily does, so `ir.py` and `ir_from_acp.py` are declared a closed family and the test asserts nothing outside it imports either. The guarantee is unchanged in substance and the test is strictly stronger: a new converter has to join the family explicitly, and a stale name in that list now fails a test of its own. - `docs/trace-interop.md` — §8.3 gains the implemented mapping table with a class per row and the measured loss counts; §8.7 records what is now implemented and restates the isolation property; §8.8 records what writing the first converter showed about the declared-absence rule, including the part of it most likely to be revised in review. No runtime module imports either module, no capture path or exporter changes, no on-disk format changes, and no new dependency. Verified: 35 new tests; tests/trajectories 324 passed; the trajectory, capture, streaming and ACP regression lane 489 passed; `ruff check .`, `ruff format --check` and `ty check src/` clean. Two real rollouts converted and cross-checked against their source files by hand.
Found by reading a converted real rollout by hand, not by the test suite, which
was green: §8.4 published its worked example with `exclude_none=True`, so
`arguments` was absent from the document while the loss report kept a record
addressing `events[1].tool_call.arguments`. The declaration that legalizes the
absence pointed at a key no reader of that document could find.
The suite could not see it because both sides of the comparison used the same
non-canonical encoding, so it was self-consistent.
`None` in this IR is a positive statement — the source did not carry this field
— and every one of them is paired with a `LossRecord` that addresses the field
by path. Dropping the key makes the address dangle and collapses "we looked and
it was not there" into "this version has no such field". Both encodings
re-validate to an equal pydantic model, so the model layer cannot enforce this;
the rule is stated in the `ir.py` docstring and enforced by tests.
- `ir.py` — declares the canonical encoding: nulls retained,
`exclude_none=True` is not a valid encoding of a Trace IR document. No
serializer is added: there is no on-disk artifact yet, and a writer would
anticipate an interface this proposal has not earned. Adds the corollary that
a record names the outermost absent node, so a conversion with no usage
declares `usage`, not `usage.input_tokens`.
- `CanonicalTrace.outcome` is no longer optional. Applying the new guard
immediately found a second instance of the same class: `outcome.stop_reason`
is a loss every ACP conversion declares, and it could not resolve in any trace
that did not time out, because the section itself was null. It is now always
present with `None` fields, like `agent` — which is exactly why
`agent.agent_version` resolved and `outcome.stop_reason` did not.
- `docs/trace-interop.md` — §8.2 gains the encoding rule as a fourth design
choice, §8.4 is regenerated in the canonical encoding so `arguments: null` and
its loss record are visible in the same document, §8.5 records the new
test-pinned property, and §8.8 records what the human end-to-end pass showed,
including the limit below.
- Tests — `test_every_concrete_loss_path_resolves_in_the_canonical_encoding`
(IR) and `test_the_canonical_document_shows_null_arguments_beside_their_loss_record`
plus `test_every_concrete_loss_path_of_a_converted_trace_resolves` (converter).
Each asserts in the same test that the discarded encoding *fails* to resolve
those paths, so none of them can pass for both encodings at once.
Known limit, recorded rather than fixed: the invariant forces a converter to
declare an absence, it cannot stop one from writing `arguments: {}` instead of
`null`. Such a trace is valid. Closing that would mean the IR taking a position
on what an empty map means for each source, which the tri-state rule
deliberately leaves to the converter.
Verified: tests/trajectories 328 passed; the trajectory, capture, streaming and
ACP regression lane 493 passed; `ruff check .`, `ruff format --check` and
`ty check src/` clean. Human end-to-end H1-H4 run against real captured
rollouts, all PASS.
Slice C —
|
Preparation for the first outbound converter, and a gap the inbound one had already papered over. `LossRecord.field` was documented as a path in IR terms, but not every record is about an IR node. Slice C already needed an escape and invented a `source[i]` string convention in a module docstring, recognized by `startswith` in two guards. The ATIF edge makes the same gap unavoidable in the other direction: a prompt-derived step, a `message: ""` the format requires, a `total_steps` the document computes about itself — none of these has an IR antecedent, and giving them an invented IR path would produce an address that does not resolve. Adds `PathSpace` — `hub` · `source` · `target` — and `LossRecord.space`, defaulting to `hub` so every record written before this field existed keeps its meaning. `field` is the path *inside* its space and carries no prefix repeating it: the space is a property of the record, never inferred from the string. Two records may legitimately hold the identical path in different spaces, which is exactly the `acp -> ir` case where a skipped input entry and the IR event that inherited its index are both `events[3]`. Three spaces cover every direction and a new format adds none, because every edge has the IR on exactly one side and therefore exactly one non-hub space: an inbound edge can talk about its source, an outbound one about its target. Only `hub` records compose across edges — the IR is the output of an inbound conversion and the input of an outbound one, so `events[1].tool_call.arguments` denotes the same field in both reports and the records join on it. `source` and `target` records are terminal by construction. No unified report is introduced; composition happens at read time over `(direction, field, space)`. The same asymmetry settles which side owns a report, now stated explicitly in the module docstring: a trace is built exactly once so an inbound conversion may attach its report to it, while a trace may be converted to many targets so an outbound conversion returns its report and leaves `trace.losses` untouched. - `validate_trace` invariant 7 now requires a *hub* record. A `target` record holding the same path addresses another document and declares nothing about this trace; previously the string alone would have satisfied it. - Both canonical-path guards filter on `space is HUB` instead of testing string prefixes. `_is_per_event` classifies by space and takes a record rather than a string. - `LossReport` gains `by_space`, and `for_field` takes the space — the space is part of the address, so it is not defaulted away silently. - `ir_from_acp` migrates its one `source[i]` record to `space=SOURCE, field=events[i]`: the path is now the one the input actually uses, since the space no longer has to be spelled into the string. Five new tests pin the contract, including that no space is inferred from the string in either direction: a `source`/`target` record with a hub-shaped path does not satisfy the declared-absence invariant, and a hub record with a source-shaped path does. `docs/trace-interop.md` §8.2 gains the spaces and the report-ownership rule as a fifth design choice; §8.4 is regenerated so the example carries the new field. Additive: no existing path changes meaning, no runtime module imports either IR module, and no format, exporter or artifact changes. Verified: tests/trajectories 334 passed; the wider trajectory/capture/ACP lane 466 passed; `ruff check .`, `ruff format --check` and `ty check src/` clean.
The first outbound edge, and the first converter that has to fabricate. Where
`ACP -> IR` stress-tested whether every absence could be declared, this one
stress-tests the other half of the taxonomy: ATIF requires values the IR does
not carry, so `SYNTHESIZED` stops being a decorative enum member.
`export_atif.py` is untouched and remains the only writer of
`trainer/atif.json`. Nothing imports the new module.
The claim it is built to support, and the reason the slice is worth anything:
ir_to_atif(acp_events_to_ir(events), prompts=P)
==
trajectory_to_atif_record(events=events, prompts=P)
Parity with the existing direct exporter, on the same inputs, for the document.
A hub that lost anything the direct path preserved would fail that equality.
Asserted over events driven through the production capture path plus nine
further shapes, and confirmed against the two real gemini rollouts of §5.2: for
both, the document produced through the hub is identical to the
`trainer/atif.json` those rollouts actually wrote, back when they ran.
Parity is about the document, not the report — the direct exporter produces no
report, which is the difference this whole proposal is about.
**One deliberate deviation, enumerated by a test rather than left in a diff.**
`acp_events_to_atif_steps` renders an oracle record as a `source: "agent"` step
prefixed `[oracle: …]`, recoverable only by string matching; §5.1 records this
as a live divergence, since the in-repo validator already accepts
`source: "oracle"` while no emitter produces one. The IR carries the role, so
this edge emits `source: "oracle"` with the command as the message. A test runs
both paths over a trajectory holding every capture event type plus an oracle
record and asserts that step is the only one that differs.
Seven fabrications, each recorded where it is invented: `agent.version` and
`agent.name`, a tool-call id, a `function_name`, `arguments`, `steps[].message`,
and `final_metrics.total_steps` — plus the prompt-derived user steps. The first
five are hub-space records at the IR field whose absence forced them; the last
three have no IR antecedent and are target-space.
`arguments` keeps its `{}`. It is what ATIF requires and what the direct
exporter writes, and departing from it would trade a real compatibility property
for a cosmetic one. What changes is that it is no longer silent: the fabrication
is declared at the same hub path the ACP edge declared `UNSUPPORTED`, so reading
the two reports together says the source never had arguments and the target
demanded them anyway. A tool call carrying real arguments — including a
genuinely captured `{}` — passes through and is declared nothing.
Losses are declared only for what this edge actually loses given the trace in
hand. An ACP-derived trace has no per-event timestamps or usage, and the inbound
report already declared those as `UNSUPPORTED`; repeating them here as `DROPPED`
would double-count one fact and misdescribe an edge that loses nothing it was
given.
- `src/benchflow/trajectories/ir_to_atif.py` — returns `(document, report)`
rather than attaching, because a trace may be converted to many targets and
none of those conversions describes how the trace was built. It imports one
benchflow module, the IR: `ATIF_SCHEMA_VERSION` is redefined and pinned equal
to `export_atif`'s by a test, and the `ThoughtBuffer` join is reimplemented and
pinned the same way, so the hub does not depend on export plumbing.
- `tests/trajectories/test_ir_to_atif.py` — 35 tests, organized around parity.
Also pins that an outbound conversion leaves the input trace byte-identical,
that two conversions of one trace are independent, that every hub record of
the outbound report resolves in the canonical encoding while target records do
not, and that a hand-built trace converts without going through ACP.
- `test_only_the_ir_family_imports_the_ir` gains `ir_to_atif`. That test failing
first was the mechanism working: a new converter joins the family explicitly.
- `docs/trace-interop.md` — §8.3 gains the implemented mapping with a class per
row and the measured counts, §8.7 the status, §8.9 what the first outbound
converter settled.
Measured on the real rollouts: H1 produces 12 outbound records (6 synthesized,
4 dropped, 2 normalized; 9 hub, 3 target), H2 produces 14, the extra drops being
the timeout event, its extensions and the trace outcome.
Known gap, recorded not fixed: `TraceUsage` has no cost field, so
`final_metrics.total_cost_usd` cannot be produced through the hub. Neither real
rollout carries one, so parity is unaffected there; closing it means adding a
field to the IR.
Verified: 35 new tests; tests/trajectories 369 passed; the trajectory, capture,
streaming and ACP regression lane 534 passed; `ruff check .`,
`ruff format --check` and `ty check src/` clean.
…ound loss Two holes in Slice D's central claim, found by auditing it rather than by a test. Both are about the same thing: the outbound edge was describing a trace shaped like the one ACP happens to produce, not the trace it actually received. ## Cost `trajectory_to_atif_record` accepts `total_cost_usd` and writes it to `final_metrics`. The IR had no cost field, so a document produced through the hub could not carry it — the parity claim held only for traces without one. `TraceUsage` gains `cost_usd` and `price_source`, mapped to `final_metrics.total_cost_usd` (preserved) and to a declared `DROPPED` (ATIF has no slot). Cost sits with the token counters because that is where every BenchFlow object already puts it — `agent_result`, `TaskTelemetry` and ATIF's own `final_metrics` — and because it is derived from them. `price_source` is to `cost_usd` what `source` is to the counters: BenchFlow computes no prices of its own, it imports a number from the model gateway's log, so a cost without the table that produced it is not comparable. The writing path is real: `providers/litellm_logging.py:618-623` sums the callback log's per-entry `cost` into `Trajectory.metadata["cost_usd"]`, it surfaces as `AgentResult.cost_usd`, and `rollout/_results.py:448` hands it to the ATIF writer. Stated precisely, because the audit turned up a nuance worth recording: **no rollout artifact available here has ever carried a non-null cost** — including four whose `usage_source` is `provider_response`, which went through the proxy but whose gateway log carried no per-entry cost. So the field's production is established by reading the code, not by observation, and parity with a cost is asserted on synthetic input given to both paths. Per-call cost stays out of scope: it exists in the proxy capture, and modelling it is a larger question this proposal does not open. ## Conditional outbound losses The rule was right and the coverage was not. Probing with a trace that really carried the values showed six kinds of information the IR can hold, ATIF cannot represent, and the report did not mention: trace `trace_id` / `started_at` / `finished_at` / `provenance` / `extensions`, `agent.provider`, per-event `outcome`, tool-call timestamps, the `raw` block behind a rendered text block, and a `role` that disagrees with the source ATIF derives from the event kind. Two were also addressing bugs rather than gaps: a tool-call timestamp was covered by a condition that emitted a record at `events[].started_at`, blaming the event for a value belonging to its tool call, and `finished_at` was never named at all. Timestamps are now four separate claims, each addressed where the value lives. `role` is declared per event and only when it disagrees with the implied source — for anything ACP produces the two always agree, so an ACP trace declares nothing for it, which is the rule working. ## Pinning the rule Two tests convert the same shape twice: once through the ACP edge, where the fields are absent and nothing is declared, and once hand-built with every field present, where the complete `DROPPED` set is asserted as an equality. A third test derives the field list from the IR models themselves and requires each one to have a disposition here — mapped, normalized, declared, container, or representation. Adding a field to the IR that ATIF cannot represent now fails the suite until its fate is decided, which is what makes the first two tests stay honest as the IR grows. ## Verification H1 and H2 parity is unchanged: for both, the document produced through the hub is still identical to the `trainer/atif.json` the rollout really wrote. Their outbound reports grow from 12 to 14 and from 14 to 16 records, the two additions being the trace-level `provenance` and the `raw` behind each text block — both values those traces really carry and ATIF really drops. 45 tests in the ATIF suite; tests/trajectories 379 passed; the wider lane 544 passed; `ruff check .`, `ruff format --check` and `ty check src/` clean. One intermittent failure was seen once in `tests/test_acp.py::TestIdleTimeoutDiagnostics` — a wall-clock assertion that passes alone and on re-run, in a module that imports nothing from the IR family.
Slice D —
|
| edge | space | field | class |
|---|---|---|---|
acp -> ir |
hub | events[2].tool_call.arguments |
unsupported |
ir -> atif |
hub | events[2].tool_call.arguments |
synthesized |
One field, two edges, one path: the source never carried arguments and the
target demanded them anyway. source and target records are terminal by
construction, and there is no unified report — composition happens at read time.
The same asymmetry settles ownership: a trace is built once, so an inbound
conversion may attach its report to it; a trace may be converted to many
targets, so ir_to_atif returns (document, LossReport) and leaves the
input trace byte-identical.
One deliberate deviation
acp_events_to_atif_steps renders an oracle record as a source: "agent" step
prefixed [oracle: …], recoverable only by string matching — §5.1 records this
as a live divergence, since the in-repo validator already accepts
source: "oracle" while no emitter produces one. The IR carries the role, so
this edge emits source: "oracle" with the command as the message. This is
the only intended difference, and a test runs both paths over a trajectory
holding every capture event type plus an oracle record and asserts that step is
the only one that differs.
The loss report describes the trace it receives
Not the trace ACP happens to produce. An ACP-derived trace has no per-event
timestamps or usage, and the inbound report already declared those absences —
re-declaring them here would count one fact twice. The same conversion over a
trace that does carry them declares every one, addressed where the value lives
(a tool-call timestamp under the tool call, not under its event).
Both halves are asserted, the second as a complete set. A companion test derives
the field list from the IR models themselves, so a field added to the IR that
ATIF cannot represent fails the suite until its fate is decided.
Cost
TraceUsage gained cost_usd and price_source before this was published,
because the direct exporter accepts total_cost_usd and the IR could not carry
it — the parity claim would have been true only for traces without one. Cost
sits with the token counters because that is where agent_result,
TaskTelemetry and ATIF's own final_metrics already put it. price_source has
no ATIF slot and is declared dropped: BenchFlow computes no prices of its own,
so a cost without the table that produced it is not comparable. Per-call cost is
out of scope.
Verification
Two kinds, deliberately not merged.
Machine checks
45 tests in the ATIF suite; tests/trajectories 379 passed; the wider
trajectory / capture / streaming / ACP lane 544 passed; ruff check .,
ruff format --check and ty check src/ clean.
One intermittent failure was seen once, in
tests/test_acp.py::TestIdleTimeoutDiagnostics — a wall-clock assertion that
passed alone and on re-run, in a module that imports nothing from these modules.
Recorded rather than attributed.
Human end-to-end — run by @Galius5136, not by CI
D1, D2, D3 use real rollouts already captured (the gemini runs of §5.2,
docker sandbox, ACP transport, standard artifact writers).
- D1 — real tool use. PASS. Three-way comparison identical; 14 outbound
records; theunsupported → synthesizedcomposition above verified on both
tool calls. - D2 — real wall-clock timeout. PASS. Three-way comparison identical; the
timeout is preserved as far as the IR and then declareddroppedat the ATIF
edge, which is the honest result — 16 outbound records. - D3 — hand inspection. PASS. An independent
diffof the produced document
against the rollout's ownatif.jsonreports IDENTICAL.arguments: {}
reproduced,agent.version: "unknown"reproduced, and the two leadinguser
steps reproduced — including the duplicate §5.2 documents. Parity means
reproducing the defect too, not silently fixing it.
D4 and D5 are constructed controls, and are labelled as such:
- D4 — the oracle deviation. No real rollout ran in oracle mode; the input is
built by hand. - D5 — negative control. Removing a tool title from the IR flips the parity
check fromTruetoFalse, so the comparison demonstrably discriminates.
Not established
- Real cost end-to-end is not verified. Every rollout artifact available
carriescost_usd: null— including four whoseusage_sourceis
provider_response, whose gateway log simply carried no per-entry cost. The
writing path is established by reading the code; the mapping is covered by
tests. D6 is a machine/integration proof and is not a human E2E. - Non-text content blocks remain covered by tests only — neither rollout
emitted a file-edit or terminal block. - Oracle mode likewise: constructed input, never a real oracle rollout.
- The richly-populated trace used to pin the complete loss set is
hand-built; no source in the repository produces one yet. - No new rollout was run for this slice.
Feedback on the architecture is still welcome and still cheap to act on: nothing
imports these modules, so redirecting this costs a revert rather than a
migration. The open questions in the PR description remain unanswered, and §8.6
still lists what a review can reject.
Galius5136
changed the title
…nwired
Closes the ATIF pair: `ir_to_atif` writes the document, `atif_to_ir` reads one
back. Nothing imports either, `export_atif.py` is untouched, and it is still the
only writer of `trainer/atif.json`.
The module is built on one rule — read what the document says, never what it
probably meant. An ATIF document is the output of a lossy conversion and several
of its values were fabricated by the converter that wrote it: `agent.version` is
the literal "unknown" whenever BenchFlow had none, `arguments` is `{}` for every
ACP-derived call, `message` is `""` on a step carrying only a tool call. Reading
those back as absent would be guessing which values its counterpart invented,
and would make a round trip look better than it is. So every value is taken
verbatim.
The consequence is the point rather than a wart: a fabricated value returns
indistinguishable from an observed one. `arguments: {}` reads back as an
observed empty argument map, which the IR's tri-state contract cannot separate
from a real one. The information was not lost, it was replaced by a false
statement of the same shape — and only the two loss reports carry the truth.
What no ATIF document holds — trace id, timestamps at any level, run outcome,
provider, per-event provenance — is declared UNSUPPORTED, not DROPPED: the value
is absent from the source, not discarded here, and that is the distinction which
decides where a fix would land. Every DROPPED record in this direction addresses
the source document instead, in the source path space.
Two shapes are deliberately not undone. A step with both `message` and
`reasoning_content` stays one event, because the blank-line join that produced
it is not injective (§5 loss benchflow-ai#10) and splitting it would invent a boundary. A
step with several tool calls becomes one event per call, since the IR models one
per event; no writer here emits that, another producer can.
54 tests, over documents from both writers — the direct exporter and the hub —
plus malformed input a document read off disk can actually have.
…wired Slice D proved the hub reproduces the direct exporter's document. With both ATIF edges in place the loop closes, and this answers the harder question: how much of a trace is still there after a trip through the interchange format. It is a measurement, not an assertion. `compare_traces` reads the two traces and never the loss reports, so a converter that lost something without declaring it is caught rather than confirmed. No percentage, because one number would merge two unrelated things. The report crosses an observed axis — preserved / transformed / lost / fabricated, from the comparison alone — with a declared one: whether ATIF has a slot at all. A loss with a slot is a gap in our own edge and fixable; a loss without one is a cost of the format. The declared half is a table with one entry per IR field, and a test derives the field list from the models so a new field cannot reach the round trip undecided. Comparison is by canonical path rather than by event position: after a conversion that fuses and drops events there is no recoverable correspondence between event i and event j, and inventing an alignment would be guessing. Measured on the two real rollouts of docs §5.2, machine measurement only: - Nothing representable is lost, on either. Every value the loop drops is dropped because ATIF has nowhere to put it, so there is no gap in our edges to close and the remaining loss belongs to the format. A test pins this, and a failure there names a converter bug. - The trace comes back with MORE values than it left with — 46 in, 56 out for H1 — while having lost information. Four fields are fabricated every run, and two of them matter: `agent.agent_version` and `tool_call.arguments` leave declared SYNTHESIZED and return unmarked, so the reconstructed trace asserts they were observed. The information was not lost so much as overwritten with a plausible value of the same shape, and only the pair of reports still carries the truth. - A timeout costs six fields. H2 differs from H1 only by ending in one, and that single fact takes the event, its reason, the run status and three extension fields with it — §5 loss benchflow-ai#4, measured rather than asserted. Docs: §8.3 gains the ATIF -> IR mapping table, §8.10 is the measurement, §8.7 is updated. The doc for the converter that landed in the previous commit is here rather than there, so the two tables could be written against each other. 31 tests. Still nothing imported by a run path; `export_atif.py` untouched and still the only writer of trainer/atif.json.
… input Four defects found by adversarially probing the edge with documents it does not write itself. None of them touches a conformant document — the H1/H2 numbers are unchanged — but each one broke the rule the module is built on, which is exactly the rule that makes the round-trip measurement worth anything. - A JSON `null` in a string slot became the literal `"None"`. `message: null` produced a four-character string the document never contained, and `reasoning_content: null` produced a segment list holding it. Invariant 5 was satisfied throughout, which is why the suite could not see it. Null and absent now both read as no value. - A step with no usable `source` was attributed to the agent. Missing, non-string and out-of-vocabulary sources all became `agent_message` — not a harmless default, since that is the kind a consumer counts as a model turn. They become UNKNOWN, as `ir_from_acp` already does for an unrecognized record. - A non-string `source` was discarded. It cannot be the IR's `source_type`, but it was the only thing the document said about the step's origin, so it is kept in extensions with a record. - A `tool_calls` or `observation` this converter cannot read vanished with no record in any path space. Unreadable here is not the same as not present; both are now kept verbatim and declared in the source space. Also closes a hole in the measurement itself. `lost` counts fields the table calls representable that did not survive, and the table is written by hand — so a field wrongly marked unrepresentable would move a real, fixable loss into the format's column and leave `lost` at zero for free. Two tests make the table falsifiable: on a trace populating every IR field `lost` is 2 (`ir_to_atif` writes no per-step metrics, so per-event usage is lost through a slot ATIF has), and no field the table calls unrepresentable comes back with any value intact. `lost = 0` on the captured rollouts is now a result rather than a definition. `arguments` joins `content[].raw` as an opaque path: both hold a mapping whose keys come from a tool rather than from the IR, and without it the field's path depended on its own contents. Docs: the timeout costs five fields and the event carrying them, not six — the sixth entry in H2's column is the source content block, which H1 loses too. 20 tests added, all of them failing on the code before this commit.
§8.10 described its own numbers as a machine measurement and not a human E2E. That was accurate when written and is no longer: the loop's output has since been checked against the raw artifacts by hand on both rollouts — parity, the conversion read in both directions, every fabricated value confirmed present in the ATIF document and absent from the capture, every unrepresentable one confirmed the other way round, and the negative control. The declared limits are unchanged: oracle rollouts and non-text content blocks remain test-only, no artifact here carries a cost, the representability table is checked against our converters rather than against a vendored ATIF schema, and two rollouts from one agent stay a demonstration rather than a survey.
Slice E —
|
Adds the inbound OpenTelemetry edge, plus the AnyValue decoder it is built on. Inbound only: no `IR -> OTel` emitter, no wiring into a run path, no on-disk artifact, and no existing format, exporter or artifact changes because it exists. `uv.lock` is untouched. §4.2 is FACT — this repository has no OTel code and no OTel dependency — so the mapping is written against the two artifacts `uv.lock` already pins rather than against recollection: `opentelemetry-proto` 1.41.1 for the wire shape, `opentelemetry-semantic-conventions` 0.62b1 for the `gen_ai.*` vocabulary. Both versions are recorded in code, both wheel hashes were checked against the lock, and neither package is imported — the edge reads JSON dictionaries. Reading them settled four things a mapping written from memory gets wrong, and the collector deleted in e84795d got three of them wrong: `intValue` is a JSON string in the canonical encoding; an `AnyValue` writes `""`, `0` and `false` explicitly, so its `stringValue or intValue or ...` read observed values as absent; `gen_ai.usage.total_tokens` does not exist at 0.62b1 and the cache counters are spelled `cache_read.input_tokens`; enums serialize as member names by default and as integers under a flag, and both parse. The rule the edge is built on: a span is evidence of an operation, not a statement about an agent. Exactly one span shape becomes a typed IR kind — the `execute_tool` operation the pinned vocabulary defines as that — and every other span becomes UNKNOWN with its whole content carried in `extensions.otel`. So of the IR's seven EventKind members this edge reaches two. For four of the eight `gen_ai.operation.name` values there is no IR candidate at all; for `chat`, `generate_content` and `text_completion` there is one, and the blocker is checkable rather than a matter of taste — filling it means reading `gen_ai.input.messages`, whose structure the pinned package defines by reference to a JSON schema it does not ship. Order is preserved and never sorted by time: siblings overlap, a batch may omit a parent, and two spans can share a start instant. The `parentSpanId` edge set is preserved per span instead, together with the envelope coordinates, so the `resourceSpans`/`scopeSpans` partition survives flattening — two spans batched separately under an equal `scope` stay distinguishable. Identifiers are carried exactly as written and never re-encoded: a 32-character hex trace id fed to the pinned JSON parser is accepted as base64 and silently yields 24 bytes, so hex and base64 are not reliably distinguishable and normalizing would make identity depend on a heuristic. Two losses are structural and both are declared: nanoseconds do not fit a `datetime`, and OTLP models `name`, the timestamps, `parentSpanId`, `kind` and the `dropped*Count`s as scalars with no presence, so absent and default are the same document. A third is a distinction worth keeping — decoding `AnyValue` wrappers into a map preserves the values and loses the wire form, which is `normalized`, not `preserved`. Trace-level `started_at`, `finished_at` and `usage` are left empty on purpose. The class says why: NORMALIZED when the spans carry the information per span and aggregating it would assume the payload holds the whole run, UNSUPPORTED when there was nothing to aggregate. An OTLP export request is a batch. 129 tests. The contract guard derives the IR field list from the models and requires every field to be filled or declared **per instance** — per event, per content block — with two exemptions that are properties of the IR rather than of a converter. It found two real undeclared absences while the edge was being written, and a structural review of the finished slice found four more, one of them in the guard itself: it had been field-level, so a field the fixture happens to fill passed on every payload. 33 targeted mutations, all caught. The suite's fixture is not hand-written: it is `MessageToJson` output from the pinned library, so its encoding is the library's rather than this suite's, and `e2e-f/regenerate_fixture.py` rebuilds it from a throwaway venv and diffs. HUMAN E2E VERIFIED 2026-08-18, F1-F6 all PASS, recorded in §8.11 together with the limits it does not extend past: the fixture's encoding is authoritative and its content is constructed; no OTLP payload from a real agent has ever been read, because nothing in BenchFlow emits spans; the mapping is checked against two pinned packages, not the specification; the GenAI conventions are still `_incubating`; there is no `IR -> OTel`; and there is therefore no OTel round trip and none is measured. `0 dropped` on the fixture is a property of that payload, not of the edge — two fully conformant inputs do produce DROPPED records.
…ision Adds §8.12. No code changes: the edge it describes is the one already committed in 796e436, and `IR -> OTel` is not implemented here or anywhere. For this implementation OpenTelemetry is an ingest boundary. `OTLP/JSON -> IR` exists; `IR -> OTel` is deferred. What the section is careful about is *whose* decision that is: it is ours, it is current, and it is reversible. No maintainer has said the emitter is out of scope, and none has said it is required — open question 5 is untouched and stays on the list in §8.11. The reasoning is five things checkable in the tree rather than a reading of anyone's intent: no outbound OTel consumer exists, no exporter or backend is integrated, the OTel support that did exist was a receiver, the consumer named by the task's own issue body reads ACP, and there is no contract to build an emitter against. That last one is the load-bearing reason — without a consumer, the span tree for a rollout, the id encoding and the commitment to an `_incubating` vocabulary would all be decided by this implementation alone, and the hub is the one place a wrong answer propagates to every format. Three things the section refuses to claim, because each would be an overreach in a different direction: that `IR -> OTel` will never be needed; that the `<->` in the task title has been resolved (it has not, and this decision is not evidence about what it meant); and that the task has been narrowed — if the answer comes back "both", the work is an additional edge, not a redesign. The reversal condition is a consumer or a contract, and one constraint holds until then: the IR must not grow a span-tree field, an id-minting policy or an OTel-shaped slot in anticipation. The hub is a superset of what BenchFlow can observe, and a field added for a converter that does not exist has no evidence behind it. The emitter, when written, takes the IR as it finds it and declares what it cannot express — the contract every outbound edge already has.
Slice F —
|
…halves Two inconsistencies §8.7 was left with after Slice F. "Not implemented, deliberately: `IR -> OTel`" was true but said nothing about whose deliberation, which is the whole distinction §8.12 exists to draw. The status now reads as one state — the OTel direction is implemented inbound and deferred outbound, by a decision of ours, reversibly — instead of as an implemented edge sitting next to an unexplained absence. The isolation paragraph still claimed "each converter imports one benchflow module — the IR", which stopped being true when `_otlp_anyvalue` was split out of the OTel edge. It now says what each converter actually imports, and keeps the property that was the point: every converter reads or writes its format as data rather than through the module that already handles it. Documentation only. No code, no test, and no `IR -> OTel`.
…nwired The fifth edge of the hub, and the first one whose target refuses partial output. Nothing imports it, nothing writes its result to disk, `_capture.py` is untouched and still the only producer of the format, and `uv.lock` is intact. The target is the **ACP-session capture event format** — the three record shapes Slice A's schema pins — and deliberately not the `acp_trajectory.jsonl` artifact. That file also holds oracle records, `hosted_env` records and session-factory `steps`, and §2.1 already recorded that no artifact-level contract exists in this repository. An edge cannot target a format nobody has defined; this one targets the part that is defined and says so. Every record shape has `additionalProperties: false` and six required fields on a tool call, so this is the first outbound edge that can be *unable* to write. It fails closed: `ir_to_acp_capture_events` either represents every event or raises `AcpCaptureNotRepresentable` — a `ValueError` carrying the blocking `LossRecord`s, following `PrimeSftTrajectoryJsonlError` and the `ValueError` `ir_to_atif` already raises. A partial event list is indistinguishable from a complete one once it leaves the function: the target has no envelope, no count and no marker for "some events are missing". `acp_capture_blockers` answers the same question without an exception, through the same pass, so asking and converting cannot disagree. Nothing is invented. `status` is never synthesized — ACP's vocabulary is closed, every member asserts a lifecycle state, and a fabricated one would reach the viewer and be shown to a person as an observation. `ORACLE` and `UNKNOWN` have no record shape in the contract, so they are refused rather than dropped, and the schema is not widened to admit them. A content block with no source block is refused: ACP stores wire blocks verbatim, two shapes are known to be consumed, and choosing one would be inventing structure. `title` is the single exception, written as `""` only because the contract documents that as its representation of an absent title, and declared SYNTHESIZED because a reader cannot tell it from an observed empty one. `tool_call_id` carries identical documentation and is deliberately not treated the same way: an empty title is a missing label, an empty id would turn an absent identity into an observable one. **`name_semantics` gates the `kind` slot**, and a review is what put it there. An ACP `kind` is a *category* — `ToolKind` calls itself "Category tag for tool calls" — while ATIF's `function_name` and OTel's `gen_ai.tool.name` name particular tools. An earlier version wrote the name through regardless, so an ATIF document with a usable status exported `kind="read_file"`: a tool name in a category slot, with only a DROPPED record to show for it. That is not a normalization a reader can undo, it is a reinterpretation. Only `name_semantics == "acp_kind"` is writable now, and the string is never inspected — a `function_name` of `read` collides with a real ToolKind member and is refused anyway. Representability is decided by the data and never by the lineage. `Provenance` is not read, and a test asserts the module never references it: an OTel-derived trace carrying every required value would export, and an ACP-derived trace missing one does not. `ACP → IR → ACP` reproduces its input exactly — structural equality and key-order preservation — on the two captured rollouts of §5.2 and on the suite's fixture with its drop-one subsets. Three corpora, not a universality claim. 77 tests, every emitted record validated against the published schema. Human-verified end to end, G1-G10, against `e2e-g/PROCEDURE.md`; §8.3 records the limits that sign-off does not extend to — no artifact-level support, no exportable oracle or unknown events, no general ATIF/OTel tool results, no `OTel → Viewer`, and no wiring.
Slice G —
|
The round-trip harness measures what changed (D) and reads only the traces. Each converter separately declares what it could not carry, or had to invent (L). Nothing joined the two, so a divergence no edge had declared looked exactly like one that was. ir_conformance.py is that join, kept out of both halves so the measurement stays independent of the declarations it is supposed to be checked against. Two things the naive D subset-of L misses, and what the gate does instead: - The sides address different path spaces. ir_to_atif writes schema_version and step_id into the ATIF document and can only declare them in TARGET space: a hub path would address a node the input trace does not contain, which ir.py's resolution invariant correctly rejects. ir_from_atif then reads them back into the hub, so the round trip observes values the input never had, without either edge having lied. TARGET_TO_HUB relates the two, and a test checks every entry against real values rather than asserting it. It is a bridge and not an exemption: remove the SYNTHESIZED record and the gate fails anyway. - Events are fused away, so some divergence is structural rather than a mishandled field. structure_explained requires every value that went missing to be one a vanished event was holding - per value, not per path - so an edit to an event that survived the trip is still caught. ir_to_atif now declares schema_version and steps[].step_id as SYNTHESIZED instead of leaving them undeclared. SYNTHESIZED and not NORMALIZED, because neither has a source value being reshaped: the dialect string is not an observation about the run, and step positions are not the IR's event index renumbered. The rules: an undeclared fabrication is always a violation, with no allowlist for structural metadata; a loss at a path the target can hold is a violation; a transformed path must account for both the values that left and the values that arrived, since TRANSFORMED conflates the two. Mutation testing found a real gap - the suite would have accepted a NORMALIZED record where the rule asks for a SYNTHESIZED one - and a test now covers it. Still unwired. ir_conformance joins the IR family in the isolation test, nothing on a run path imports it, and no artifact changes. ir.py, ir_round_trip.py, the viewer and every runtime module are untouched. tests/trajectories: 699 passed. Mutation harness: 11/11 applied and caught. Human verification H1-H8 recorded in docs/trace-interop.md section 8.13.
ir_to_view.py converts a trace into the step list a viewer page renders. It produces steps and deliberately not a whole payload: of a payload's five fields, only the steps are a function of the trace. rollout_name is a directory name, verifier is four sidecar files, and meta comes from result.json and timing.json - the IR has no slot for task_name, skill_mode, reward, partial_trajectory or trajectory_source at all. Returning a payload here would mean synthesizing run metadata to fill a shape rather than converting a trace, so the trace-level fields the IR does carry are declared UNSUPPORTED: outside this edge's codomain, not lost by it. Assembly belongs to the wiring slice, which has the directory. The wire shape is read from the viewer package proposed in benchflow-ai#1034, at the commit recorded in VIEW_SCHEMA_ORIGIN. Nothing is imported, vendored or fetched from that branch - it is unmerged, and the family rule is the one ir_to_atif already follows for ATIF: read the target format as data, pin what matters by test, never reach into the module that handles it. That pin protects this edge from drifting; it cannot notice benchflow-ai#1034 changing. tool.name_semantics is a seventh key on an object benchflow-ai#1034 defines with six, and it is here because the alternative is inference. benchflow-ai#1034 derives a tool's display category with tool_hue(kind, title), which scans both strings for needles: an ATIF function_name of 'execute' becomes the execute category, and an OTel gen_ai.tool.name of 'read_file' becomes 'read' because the word appears in it. This edge emits a hue only when the semantics say acp_kind and the value is already a member of the display vocabulary - membership, never inference - and otherwise the neutral 'other', which the renderer maps to the secondary tokens and which therefore asserts nothing. On the corpora that means 'execute' is spelled identically in the ACP and ATIF rows and gets a category in one only. Unlike the ACP edge this one never raises. A viewer is a display: an event it cannot type must still reach the page. Every event becomes exactly one step, UNKNOWN and ORACLE included, and what the shape cannot hold goes to the loss report. ORACLE has no member in the step vocabulary, so it lands on 'unknown' with its identity in the type slot, and both untyped kinds carry a serialization of the canonical IR event - not a raw source record, since the IR does not hold one, and the loss record says so. Sentinels are declared per slot and only where the target has no null to write. An absent title and an observed empty one both render as "", so the report is the only place that difference survives; a block the IR holds with no text contributes no string rather than having its raw form rendered as JSON. Still unwired. ir_to_view joins the IR family in the isolation test, nothing on a run path imports it, no page is rendered and no artifact changes. ir.py, ir_round_trip.py and the existing viewer.py are untouched. tests/trajectories: 752 passed (699 + 53). Mutation harness: 13/13 applied and caught. Human verification V1-V10 recorded in docs/trace-interop.md section 8.14, along with what it does not cover.
Base update only, taken before building the viewer seam against the current renderer rather than against the one this branch forked from (43 commits back, including the viewer restyle benchflow-ai#1019/benchflow-ai#1020, the --confirm bar benchflow-ai#1021, redaction transparency benchflow-ai#1022 and the header-badge fix benchflow-ai#1023). No conflicts: the two sides overlap only on pyproject.toml and uv.lock, and there the merge is main's file plus the two jsonschema dev-dependency lines this branch adds. Nothing else was touched in this commit.
Pure extraction, no behaviour change: `_page`, `_prompt_block`, `_message_block`, `_thought_block` and `_result_block` come out of `_render_acp_events` so a second producer of blocks can emit the same page without copying its markup. The block builders take text the caller has already escaped and truncated. That is what keeps the extraction pure: the two prompt sites do those two things in opposite orders (`prompts.json` slices then escapes, an inline `user_message` escapes then slices), and a helper that picked one order would have changed the other site's output. The asymmetry is pinned by a test and left for its own commit. `tests/trajectories/test_viewer_primitives.py` freezes each card's markup and the assembled body of a whole ACP page, plus three properties of the current renderer that are facts rather than bugs: an `agent_timeout` and an unrecognized record reach no card, and a tool card carries kind/title/status but not the tool's output. Verified byte-identical on six pages rendered from the two captured rollouts of Slice A2 and from raw ACP and Codex session files — same SHA-256 before and after, ruff format included.
`ir_to_view_html` closes the chain `ir_to_view` opened: a viewer step list becomes one HTML page, built from the card builders `viewer.py` already renders its ACP page with. The dependency runs one way — this module imports the viewer, the viewer imports no part of the IR family — so what crosses the boundary is a plain step document and the renderer stays a renderer. It is written against the viewer on main. benchflow-ai#1034 stays a design reference for the step vocabulary and nothing else: no code from that branch is imported, vendored or fetched, and this edge works whether or not it lands. It does not rebuild steps into ACP capture events. The IR holds records ACP has no type for, a status it cannot spell, and tool names whose semantics are the point; forging capture events would launder all three back into the assumption the hub exists to remove. Measured against `_render_acp_events` on the two captured rollouts: H2's four events render three cards there and the word "timeout" appears nowhere, an unrecognized record reaches no card, and a tool card carries kind/title/status but not the tool's output. Those are that renderer's four branches, not bugs; this edge has six step kinds to place, places them, and declares the difference. `test_viewer_primitives.py` pins the legacy behaviour as a fact so the day a branch is added, this edge is revisited with it. Classification is a table over the eight display hues, never a substring: `viewer._tool_accent_class` is not imported and an AST test asserts the name never appears here. The same `execute` gets `acc-bash` as an ACP kind and stays neutral as an ATIF `function_name` in the same run; `read_file` from OTel stays neutral too. `think` resolves to the neutral accent because the stylesheet has none for it, and that one is declared DROPPED rather than hidden. `name_semantics` rides in the metrics line and in `data-name-semantics`; a step with no typed slot is labelled `Canonical IR representation` so the page never passes a reconstructed event off as a source record; cuts carry an explicit marker and a NORMALIZED record. `render_trace` returns both reports without merging them, and this edge never addresses the hub — it has the IR on neither side. Unwired: nothing under src/benchflow calls it. `python -m benchflow.trajectories.ir_to_view_html <path> [out.html]` is how a person looks at a page. Docs in §8.15, family list and status in §8.7.
…ace IR One branch in `render_rollout`, guarded by BENCHFLOW_VIEWER_TRACE_IR and delegating to `_trace_ir_page` — the whole wiring surface. The switch off is the default and returns None, so the branch order below it is the branch order that was there before; a captured page is byte-identical to the one `_render_acp_trajectory` produced. The import of `ir_to_view_html` is lazy and inside that function: importing the viewer never imports the IR, deleting the family cannot stop this file from importing, and reverting this commit unwires everything. A conversion that raises falls back to the ACP page with a line on stderr — a viewer that stops showing a run because a converter broke is worse than one that says so. With the switch on, an ACP rollout goes source -> CanonicalTrace -> ir_to_view_steps -> ir_to_view_html, and never through `_render_acp_events`: a test patches that function to raise and the canonical page still renders, then proves the patch bites by rendering the same rollout with the switch off. An ATIF-only rollout, which the ACP path answers with the no-trajectories sentinel, gets a page; a stream-json rollout and a directory with no readable trajectory are untouched either way. `rollout_to_trace` and `render_rollout_page` move the directory reading out of the adapter's __main__ so the branch and the module entry point answer the same question the same way. The isolation guard narrows rather than ends: `WIRING_SITES` names viewer.py and nothing else, one test asserts every listed site really does import the family, and another asserts the viewer holds no module-level import of it. Docs in §8.15 and §8.7.
…othing else leaks A human browser check of the ATIF page found a silent loss. `export_atif` writes an `agent_thought` as `reasoning_content` on the agent step it precedes, so a faithful reading of a real H1 export gives a TOOL_CALL event carrying `reasoning` — and this edge read `reasoning` only under AGENT_REASONING. The value reached no step key and no loss record, which is the one thing the loss model exists to make impossible. The suite was green throughout: on the ACP corpus a thought is its own event, so the hole never opened. `steps[].reasoning` is the fix, and it is a second additive key beside `tool.name_semantics`. Two other placements were refused: a second `thought` step would invent an event boundary and an ordering the source never declared, and `steps[].text` would keep the string while losing that it is reasoning. Strings are never joined to fit a slot. A reasoning event keeps exactly the shape it already had. Two remaining values with no honest slot now get a per-event record instead of silence: a reasoning event that also carries user-visible text (the one text slot is already holding the reasoning — declared unless the two strings are equal, in which case the value is on the page), and a terminal signal on an event that is not the timeout. The renderer shows the value inside the card of the step that carried it, above that step's own content, in the stylesheet's existing `.thinking` style. Not a second card: inventing an event boundary one layer down is the same fabrication. A test pins the with-reasoning cards against the viewer primitives they copy. What keeps this from recurring is the guard, not the three lines: every string-bearing field of TraceEvent, on every EventKind, must reach the emitted step or be named by a record — with the field list derived from the model, so a new string field fails until given a disposition. The guard's predicate is a function with its own negative control, because a predicate only ever asserted in the positive direction is one nobody has tested. The ATIF document's two identical `user` steps are deliberately untouched: they are `export_atif`'s, documented in §5.2, and the page's job is to show the document it was given. Docs in §8.14 and §8.15.
V1-V12 and V4b, all PASS, 2026-08-19, in a browser against the four corpora. The table says what a person confirmed, and the paragraph after it says what the sign-off does not extend to: four corpora on one machine in one browser, the OTLP payload's content still a construction, oracle events and non-text content blocks still test-only. The verification found the reasoning loss fixed in the previous commit, which is the third time in this work that a green suite was not evidence a document was right. Recorded as such. Also states, in one place, what this slice depends on and what it does not: benchflow-ai#984 does not depend on benchflow-ai#1034 (design reference for the step vocabulary, no code imported, every commit targets viewer.py as it stands on upstream/main); the wiring is opt-in and the switch off is byte-identical; IR -> OTel stays deferred on the terms of section 8.12; and the ATIF document's two identical user steps are the production exporter's, documented in section 5.2, shown faithfully rather than repaired here. A BrokenPipeError seen from the stdlib server during V10 is recorded as an incidental observation about serve(), not as a finding about this edge.
Slices H and I: the viewer path, built against the viewer that exists, verified in a browserHead The last edge renders a step list through the card builders Built against
|
| current page | through the IR | |
|---|---|---|
| the run that ends in a real wall-clock timeout | 3 cards from 4 events; the word "timeout" appears nowhere | 4 cards, one a typed timeout |
| an unrecognized record | reaches no card | a card labelled Canonical IR representation, body = the canonical event |
| tool output | not on the page at all | shown when the block carried text |
| an ATIF-only rollout directory | <p>No trajectory files found</p> |
rendered |
None of that is a bug being fixed — those are the renderer's four branches. This
edge has six step kinds to place, places them, and declares the difference. Three
tests pin the current behaviour as a fact so that if a branch is ever added, this
edge's reason for existing is revisited with it.
Classification is a table, never a substring
_tool_accent_class scans the tool kind and then the human title for needles.
It is not imported here, and an AST test asserts the name never appears in the
module. The hue arrives already decided — a category only when the source said
the string is one — and is mapped through a table over the eight display hues.
The same run makes the point: execute arrives as an ACP kind and gets the
execute accent; the identical string in that run's own trainer/atif.json
arrives as a function_name and stays neutral. OTel's read_file and
write_file stay neutral for the same reason.
One incidental observation about the current renderer, from a real capture
rather than a constructed example: a think tool call whose title is
Update topic to: "Executing System Commands" is painted with the bash/execute
accent, because the classifier falls back from the kind to the title and finds
the word Commands. Not touched here — flagging it because it is the same
mechanism, in your code, on a real rollout.
Human browser verification
V1–V12 and V4b, all PASS, in a browser, 2026-08-19. Legacy-vs-canonical on
the same directories, the timeout, the unrecognized record, the ATIF colour
result, the OTel neutrality, the untyped spans, tool output, the diagnostic
label, the byte-identical switch-off page, bench eval view on both paths, and
one request in the Network panel — the document itself, nothing external.
The verification found a real defect, and the suite had not
CanonicalEvent.reasoning was lost with no loss record whenever it
accompanied an event kind other than AGENT_REASONING. That is exactly what a
real ATIF export produces: acp_events_to_atif_steps writes an agent_thought
as reasoning_content on the agent step it precedes, so reading one back gives
a TOOL_CALL event carrying reasoning. On the ACP corpus a thought is its own
event, so the hole never opened and the suite stayed green throughout.
Fixed before the sign-off: an additive steps[].reasoning, rendered inside the
card of the step that carried it rather than as an invented second event, plus a
guard that is the real fix — every string-bearing field of TraceEvent, on
every EventKind, must reach the emitted step or be named by a record, with
the field list derived from the model so a new string field fails until given a
disposition. The guard's predicate has its own negative control, and five
mutations (emission, the normalization record, the loss record, the rendering,
the guard) are all caught.
Third time in this work that a green suite was not evidence a document was
right. It keeps being the artifact-reading step that finds these.
Not claimed
Four corpora, one machine, one browser. The OTLP payload's encoding is
authoritative — it comes from opentelemetry-proto's own MessageToJson — but
its content is a construction: nothing in BenchFlow emits spans. Oracle
events and non-text content blocks stay test-only, as they have been since §8.3.
IR → OTel remains deferred on the terms of §8.12, a decision of ours that a
consumer or a contract would end.
The ATIF document's two identical user steps are not touched: one comes
from the prompts argument and one from the captured user_message, they are
acp_events_to_atif_steps's doing (§5.2 records the over-count), and the page's
job is to show the document it was given. Our own ir_to_atif does not write
them.
Docs: docs/trace-interop.md §8.14 and §8.15.
|
Hey @Galius5136. Viewer-side notes from #1034, since slice I is built against
No preference on merge order. |
2 participants
Trace interoperability: OTel ↔ ATIF ↔ ACP
Draft. This PR tracks the whole trace-interoperability task and lands it in
reviewable slices. Ten are included. OpenTelemetry is read but not written:
the inbound edge is implemented, and the emitter is deferred rather than
rejected — §8.12 of
docs/trace-interop.mdrecords why, and records it as ourown decision.
One opt-in wiring site, and nothing else reaches a run path. Everything
below is inert unless
BENCHFLOW_VIEWER_TRACE_IR=1is set:render_rolloutgains a single branch with a lazy import inside it, and with the switch off the
trajectory viewer emits the bytes it emitted before — verified on six pages,
same SHA-256 before and after.
export_atif.pyis untouched and still the onlywriter of
trainer/atif.json, no artifact changes, and a test names the onefile allowed to import the IR family so a second importer fails the suite.
Reverting the wiring commit unwires it completely.
Goal
OTel ↔ ATIF ↔ ACP trace interoperability.
What is included
ACP → IRIR → ATIFATIF → IR+ round-trip measurementOTLP/JSON → IRuv.lockpinsIR → ACP capture eventsIR → viewer trace stepsviewer trace steps → a rendered pageviewer.pyas it stands onmain, opt-in, verified in a browserEach slice has its own comment on this PR with the details and the verification.
Why an IR at all
From reconnaissance of the current tree:
trajectory/acp_trajectory.jsonlis read by the viewer, judges, the skill evaluator, task verifiers and the
review skill, with no version field and no written specification. Each consumer
parses it independently.
src/readstrainer/atif.jsonback; its shape is pinned by prose in a module docstring.
gen_ai.*handling. The one module that existed was removed as unwired.each with its own private answer to the same questions — what happens to a tool
call with no arguments, whether a thought boundary survives, whether a timeout
is representable. Those answers already diverge (
docs/trace-interop.md§5).A hub makes each format one edge to a written contract, and makes the
information loss a value rather than a comment.
What the round trip measures
Slice D shows the hub reproduces the direct exporter byte for byte, which says
the IR is sufficient for ATIF. Slice E asks the more useful question — how much
of a trace is still there after a trip through the format — and answers it as a
measurement over the two real rollouts in
docs/trace-interop.md§5.2:IR ↔ ATIFfield mapping, nothing representable is loston the two measured rollouts. Everything the loop drops is dropped because
ATIF has nowhere to put it, so the remaining loss belongs to the format rather
than to these converters. That is a result and not a definition: on a trace
populating every IR field the same measurement reports two representable
losses, which is a gap in our own edge rather than a cost of the format.
one rollout — while having lost information.
argumentsleaves declaredSYNTHESIZEDand returns unmarked, so the reconstructed trace asserts the toolwas observed to be called with none. The information was not so much lost as
overwritten with a plausible value of the same shape, and only the pair of loss
reports still carries the difference.
reason, the run's own outcome status, and the three fields the marker carried.
§5 loss Readme suggestion #4, measured instead of asserted.
That last point is the argument for the hub in the one form an assertion cannot
take: it quantifies what a format costs instead of claiming it.
Full tables are in
docs/trace-interop.md§8.10. These are machine measurementsover real captured rollouts; verification status for each slice is documented
separately in its PR comment.
Roadmap
ACP → ATIFpreservation and loss invariantsACP-session capture events → IRIR → ATIF, with parity against the direct exporterATIF → IR, and theACP → IR → ATIF → IR′measurementOTLP/JSON → IR, built against theopentelemetry-proto/opentelemetry-semantic-conventionsversionsuv.lockalready pins, with no OTel dependency addedCanonical Trace IR → ACP capture events— implemented andHUMAN VERIFIED (G1–G10). The target is the ACP capture event format
of §2.1, not the whole
acp_trajectory.jsonlartifact, which has nocontract and holds records from four producers.
ACP → IR → ACPreproduces its input — structural equality and key order — on the two real
captured rollouts of §5.2. The edge is fail-closed: when the target
requires semantics the trace does not carry it raises rather than emitting
a partial trajectory.
name_semanticsgates thekindslot, so an ATIFfunction_nameor an OTelgen_ai.tool.namecannot be laundered into anACP kind — a category tag — even when the value happens to spell one.
ORACLEandUNKNOWNhave no record shape in the contract and arerefused. No runtime wiring, and no file is written.
OTel → Viewerisnot supported: an OTel trace is still four independent kinds of refusal
away from exporting
IR → OTel— deferred, see §8.12: no outbound OTel consumer and nocontract to build an emitter against exist yet. A decision of ours, current
and reversible; open question 2 below is a separate matter and still open
implemented and HUMAN VERIFIED (H1–H8),
docs/trace-interop.md§8.13.An independent join between the divergence the round trip observes
and the loss each converter declares: the measurement still reads only
the two traces, the converters still produce their own reports, and the
gate is a third module that relates them. An undeclared fabrication is
always a violation — no allowlist, including for the exporter's own
structural metadata, which
IR → ATIFnow declares like anything else.The rule asks for
SYNTHESIZEDspecifically, not merely for somerecord at the path: a
NORMALIZEDdeclaration does not excuse aninvented value, since there is no source value being reshaped. Where a
value is honestly declared in
TARGETspace and observed at a hub path,a small target→hub bridge relates the two; it is empirically
checked against real values and is not an exemption — remove the
declaration and the gate fails anyway. Divergence caused by events being
fused away is explained per value, not per path, so an edit to an
event that survived the trip is still caught. No runtime wiring. No
OTel round trip, because
IR → OTelremains deferred (§8.12) and agate cannot measure a loop that does not exist
Canonical Trace IR → viewer trace steps— implemented andHUMAN VERIFIED,
docs/trace-interop.md§8.14. Steps only, deliberatelynot a payload: four of a viewer payload's five fields are functions of the
rollout directory rather than of the trace. A tool's display category is
emitted only when the source said the string is a category and the
value is already a member of the display vocabulary — membership, never
inference — so an ATIF
function_nameofexecuteand an OTelgen_ai.tool.nameofread_filestay neutral. Every event becomesexactly one step:
ORACLEandUNKNOWNland on a diagnostic stepcarrying a serialization of the canonical IR event, declared as such
because the IR holds no source record
viewer trace steps → the page the current viewer renders—implemented and HUMAN VERIFIED in a browser, §8.15. Built against
viewer.pyas it stands onmain, not against the reviewer-grade viewerproposed in feat(viewer): reviewer-grade trajectory pages, multi-run browsing, and hf:// dataset sources #1034: that branch is a design reference for the step
vocabulary and nothing else — no code is imported, vendored or fetched
from it — so this works whether or not it lands. A pure refactor first
(
_page,_prompt_block,_message_block,_thought_block,_result_blocklifted out of_render_acp_events, byte-identicaloutput), then an adapter that renders a step list through those cards,
then one opt-in branch. Steps are never rebuilt into ACP capture
events: the IR holds records ACP has no type for, and forging them would
launder back the assumption the hub removes. Measured against the current
page on the two captured rollouts: an
agent_timeoutreaches no cardthere and the word "timeout" appears nowhere, an unrecognized record
reaches no card, and a tool card carries kind/title/status but not the
tool's output — all three are that renderer's four branches rather than
bugs, and this edge places its six step kinds and declares the difference
Scope / open questions
The canonical IR and the OpenTelemetry direction are still not approved
decisions. Everything above is built so it can be reviewed as code and deleted
in one commit if the answer is no. The questions, in the order they block work:
N ↔ 1intermediaterepresentation the shape you want, or would you rather see direct converters?
Three exporters walking the same event list through shared helpers is what
suggested it, but that is an inference, not a mandate.
only, and it was removed deliberately. Reviving a receiver and emitting spans
are different projects with different risk. This is the one blocking the
next slice.
rawInput,rawOutputandlocationson tool calls;handle_updatereadsfive fields and drops the rest, which is why every exported format emits empty
tool arguments. Per-event timestamps go the same way. Closing those gaps
changes an on-disk format several consumers already parse, so it is a
compatibility decision rather than a converter one.
acp_trajectory.jsonlcontract. Should the file carry aschema_version? And are theoraclerecord and the session-factorypassthrough part of the trajectory contract, or separate concerns that happen
to share a filename?
Happy to split this into separate PRs per slice if that reviews better.