feat(agent): OpenUI bindings — library/fragment builders, toUIOutput, getUiStream() (DEV-773) - #92
Conversation
…r (DEV-773) The SDK half of the OpenUI backport from Noetic (DEV-765). Adds packages/agent/src/lib/openui/: defineComponent/createLibrary (Zod props with normative declaration order), the typed fragment() builder with uiRef/uiState/uiBuiltin, OpenUI Lang expression serialization, and the openui(library) helper producing the wire-shaped plugin preference (Zod -> JSON Schema). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Stream() (DEV-773) Tools can now author OpenUI render fragments: an optional toUIOutput sibling of toModelOutput on every executable tool shape. Fragments are broadcast as tool.ui_fragment stream events after successful execution (render-only, never sent to the model; a throwing toUIOutput degrades to no-fragment). getUiStream() on ModelResult surfaces UI events across all turns: tool-authored fragments plus the API's response.openui.* wire events (statement/fragment/document) from the openui plugin. Wire events not yet in the SDK's stream-event union arrive via its forward-compat Unknown catch-all, so translation reads the raw payload — the stream works both before and after the SDK regen (DEV-772). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New private package @openrouter/openui-playground: a local webapp for testing, benching, and evaluating OpenUI generative-UI support. - Progressive renderer over the demo component library (Stack/Card/ Heading/Text/Stat/Badge/Table/Input/Select/Button/Progress) — UI materializes statement-by-statement mid-stream - Two modes with identical event shapes: emulate (local library prompt + reference streaming parser over the text stream — works today) and native (openui() plugin + getUiStream() — flips on when DEV-771/772 land), so the paths can be A/B'd from the history table - Bench stats per run: TTFB, first-statement latency, total time, statement/diagnostic counts, token usage, cost; session history for comparing models and prompts - Reference incremental OpenUI Lang parser (the same logic DEV-770 ports into openrouter-web) with 11 conformance tests - Plain node:http + static client; no build step Verified end-to-end against live models: single-card and 12-statement dashboard prompts parse clean (0 diagnostics) and render progressively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serializeExpr emitted object keys raw, so a key with spaces, quotes,
punctuation, or a leading digit produced source the parser rejects. Keys
come from arbitrary tool-authored objects via toExpr, so they cannot be
assumed to be identifiers. The grammar already accepts a quoted key
(parseObject branches on '"'), so quoting the rest round-trips.
String(NaN)/String(Infinity) also emitted bare identifiers, which parse
back as refs to undefined names. JSON resolves the same hole as null; do
that rather than emit source that cannot round-trip.
fix(playground): describe enum props by their values, not "string"
describeSchema returned on json.type before checking json.enum, but an
enum serializes as {type: 'string', enum: [...]} — so every enum prop was
described to the model as a plain string and it never saw which values
are legal for Badge.tone, Stack.direction, Button.variant.
docs(changeset): add the required minor changeset for the OpenUI exports
~30 new exports plus ModelResult.getUiStream and the toUIOutput tool
option had no changeset, which the public-api-examples skill requires.
The example is compile-checked against the real signatures.
The gate's no_god_files rule is fan-out > 15, not file size: model-result.ts sat at exactly 15 outbound edges and this PR's ./openui/ui-stream.js import made it 16. Verified by removing that one import — the violation disappears. Re-exported UiStreamEvent and translateUiEvent from stream-transformers.js, which model-result.ts already depends on and which owns every other wire-event translation the loop performs, so no new edge is added. Complex functions were 9 -> 13, all four new here. Each is split along a seam it already had: - translateUiEvent: one function per wire event type (cc=18 -> under) - scanStatements: string-literal and bracket-depth state machines extracted - generate: the native path's per-variant event mapping extracted - renderCall: form controls and Table extracted to renderControl/renderTable Verified with sentrux 0.5.7, the version CI pins: God files 0 -> 0, complex functions back to 9 (the 9 remaining are all pre-existing on main and untouched), gate reports 'No degradation detected'. Behavior unchanged — full suite green, typecheck and lint clean.
The table of what each stream emits is the reference consumers use to pick one; getUiStream was absent.
…escript-agent-openui-module-libraryfragment-builders
cortex review —
|
Two findings from cortex's review pass. XSS: the diagnostics panel escaped `source` but interpolated `message` and `line` raw into innerHTML. Every field there is model-controlled — `ParseFailure.message` is built from the offending source line, and in native mode diagnostics arrive verbatim off the wire — so a model could inject markup by emitting a crafted statement. All three fields are now escaped. A11y: rendered Input/Select carried no accessible name, so a screen reader announced an unlabelled field. Both signatures already have a `name` prop that was going unused for labelling; it now sets aria-label (and the real `name` attribute), falling back to the placeholder for Input.
There was a problem hiding this comment.
cortex panel verdict: comment — details in the consolidated review comment.
- escape diagnostic messages and history model names before innerHTML (XSS)
- bind playground server to 127.0.0.1 (API-key-backed endpoint)
- require trailing separator in static-file public-root prefix check
- validate uiRef/uiState/uiBuiltin names as identifiers at construction
- aria-label form controls from their name prop; progressbar ARIA + text %
- aria-live status/diagnostics regions; error frames no longer overwritten
by the green 'done' status
- rename toUIOutput -> toUiOutput (match Ui casing convention pre-release)
- warn (tool name + call id) when toUiOutput throws instead of catch {}
- collect toUiOutput broadcasts and await as one batch off the follow-up
critical path
- sticky regexes + charCode skipWs in the playground parser (was O(n^2))
- memoize openui(library) wire shape per library (WeakMap)
- drop the playground's no-op build script / outDir
…yfragment-builders' of https://github.com/OpenRouterTeam/typescript-agent into lukeparke/dev-773-typescript-agent-openui-module-libraryfragment-builders-2 # Conflicts: # packages/openui-playground/public/app.js
There was a problem hiding this comment.
cortex panel verdict: approve — details in the consolidated review comment.
…dule-libraryfragment-builders
|
|
|
|
|
|
|
|
|
|
|
|
3 similar comments
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 similar comments
|
|
|
|
|
|
# Conflicts: # packages/agent/README.md # packages/agent/src/index.ts # packages/agent/src/lib/model-result.ts # packages/agent/src/lib/tool-types.ts # packages/agent/src/lib/tool.ts # pnpm-lock.yaml
| let base = evalExpr(expr.base, depth + 1); | ||
| for (const key of expr.path) { | ||
| base = base !== null ? base[key] : null; | ||
| } | ||
| return base; |
There was a problem hiding this comment.
🟡 Playground stops rendering and shows an error when generated UI reads a nested field that is missing
A nested field lookup keeps drilling into a value that turned out to be missing (base[key] at packages/openui-playground/public/app.js:95) instead of stopping, so a generated screen that reads a two-level field the data does not have aborts the whole run with an error.
Impact: A perfectly valid model response (e.g. the data.rows.title form the playground's own prompt teaches) can kill the live render and leave the page stuck on an error message.
Mechanism: undefined intermediate in the member-path loop
evalExpr for kind: 'member' guards only against null: base = base !== null ? base[key] : null. The first hop frequently produces undefined — e.g. data = Query(...) makes the ref branch return the raw call expression object (packages/openui-playground/public/app.js:100), so expr['rows'] is undefined; the second hop then evaluates undefined['title'] and throws a TypeError.
The throw propagates out of renderSurface() → handleEvent() → the for loop inside run()'s try, which catches it, calls setStatus(..., true) and stops consuming the SSE stream, so all remaining statements/stats for that run are dropped. The library prompt explicitly instructs the model to use member access with paths like data.rows.title (packages/openui-playground/src/lang/prompt.ts:60), so this is easy to hit.
| let base = evalExpr(expr.base, depth + 1); | |
| for (const key of expr.path) { | |
| base = base !== null ? base[key] : null; | |
| } | |
| return base; | |
| let base = evalExpr(expr.base, depth + 1); | |
| for (const key of expr.path) { | |
| base = base === null || base === undefined ? null : base[key]; | |
| } | |
| return base; |
Was this helpful? React with 👍 or 👎 to provide feedback.
| const rel = urlPath === '/' ? 'index.html' : urlPath.slice(1); | ||
| const file = normalize(join(PUBLIC_DIR, rel)); | ||
| if (!file.startsWith(PUBLIC_DIR)) { | ||
| sendJson(res, 404, { | ||
| error: 'not found', | ||
| }); | ||
| return; | ||
| } |
There was a problem hiding this comment.
🟨 Static file guard can be bypassed by a sibling directory name prefix
The static handler resolves the requested path and then only checks file.startsWith(PUBLIC_DIR) (packages/openui-playground/src/server.ts:95). Because PUBLIC_DIR has no trailing separator, a request such as /../public-notes/secret.txt resolves to <pkg>/public-notes/secret.txt, which still passes the prefix test, letting the dev server read files outside the intended public/ directory.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
The Agent-SDK half of the OpenUI backport from Noetic (DEV-773, part of the DEV-765 umbrella; spec: DEV-764 RFC).
Deliberately thin per the spec: the API owns parsing/prompting/validation; the SDK ships builders, the tool render surface, and stream access.
New
src/lib/openui/moduledefineComponent/createLibrary— component vocabulary from Zod prop schemas; prop declaration order is normative (positional args in OpenUI Lang map by order)fragment(library)+uiRef/uiState/uiBuiltin— typed constructors for tool-authored UI; literal props validated at construction timeopenui(library)— produces the wire-shaped{id:'openui', library, dialect}plugin preference (Zod → JSON Schema);pluginsalready type-flows throughCallModelInput, so nocallModelsignature changetranslateUiEvent+ UI stream event modeltool()render surfacetoUIOutputsibling oftoModelOutputon regular/generator/HITL toolstool.ui_fragmentstream event (render-only — never sent to the model; throwingtoUIOutputdegrades to no-fragment)ModelResult.getUiStream()response.openui.statement/fragment/documentwire eventsUnknowncatch-all; translation reads the raw payload, so the stream works before and after the SDK regen (DEV-772)Deferred (documented in the ticket)
uiSubmitted()/uiInteracted()/uiToAssistant()stop predicates — need interaction events that only exist once Phase-3 surface state (DEV-774) landsTest plan
openui.test.ts,openui-stream.test.ts): serialization, library ordering/validation, fragment builder, plugin wire shape, event translation (incl. Unknown encoding),getUiStreamfast path,broadcastUiFragmentsuccess/skip/throw paths🤖 Generated with Claude Code
API example