Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,11 +548,13 @@ traces analyze --last 1
traces analyze --last 1 --llm --budget 0.50
traces analyze --last 1 --analyzer halo --analyzer-prompt "find token waste"
traces analyze --all --last 20 --analyzer hodoscope
traces analyze --last 1 --analyzer prime
traces analyze --last 1 --analyzer my-installed-command
```

HALO returns a diagnosis report.
Hodoscope samples distinct behaviors and marks every sample `needs_review`.
Prime posts the full span projection to an OpenAI-compatible bridge (`TRACES_PRIME_BRIDGE_URL`, default `http://localhost:4181`) and returns validated findings with span evidence.
An arbitrary command returns a raw report unless its SDK adapter explicitly parses a stricter output type.

Read [Trace analysts](./docs/trace-analysts.md) for the output contract, a minimal custom analyst, and labeled benchmark setup.
Expand All @@ -564,7 +566,7 @@ Read [Trace analysts](./docs/trace-analysts.md) for the output contract, a minim
traces upload --since 24h --dry-run --redactor "my-pii-scrubber"
```

In the SDK these are the `ExternalAnalyzer` and `Redactor` interfaces (`haloAnalyzer`, `hodoscopeAnalyzer`, `commandAnalyzer`, `commandRedactor`, `applyRedactor`, `runExternalAnalyzers`).
In the SDK these are the `ExternalAnalyzer` and `Redactor` interfaces (`haloAnalyzer`, `hodoscopeAnalyzer`, `primeAnalyzer`, `commandAnalyzer`, `commandRedactor`, `applyRedactor`, `runExternalAnalyzers`).
See [`examples/external-engines.ts`](./examples/external-engines.ts).

> The built-in agentic analysts (`--llm`) run on the Tangle router by default: set `TANGLE_API_KEY`.
Expand Down
40 changes: 39 additions & 1 deletion docs/trace-analysts.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,12 @@ traces analyze --all --last 20 --analyzer hodoscope

# Run any installed command that accepts an OpenInference file path.
traces analyze --last 1 --analyzer my-trace-tool

# One-shot prime-RLM analysis through a local OpenAI-compatible bridge.
traces analyze --last 1 --analyzer prime --analyzer-prompt "find unsupported completion claims"
```

`--model` is forwarded to the built-in model-assisted analysts, HALO, and Hodoscope.
`--model` is forwarded to the built-in model-assisted analysts, HALO, Hodoscope, and prime.
HALO and Hodoscope use their own provider clients and credentials.
The Hodoscope adapter pins version `0.2.4` and uses Python 3.11 through `uvx`.

Expand All @@ -87,6 +90,41 @@ Raw JSON is still a `report`.
Only an adapter that validates the full finding shape may return `findings`.
Hodoscope always returns `discovery`, and each candidate has `status: 'needs_review'` plus its source trace and span.

### Prime engine

`--analyzer prime` runs a one-shot analyst over the emitted OTLP artifact through an OpenAI-compatible bridge, such as cli-bridge's prime backend.
Unlike the `--llm` analysts, which drill into the trace with paged tools, prime has no REPL and no trace tools: the full span projection is inlined into a single prompt as JSON.
Oversized projections are re-rendered with a per-attribute character cap; if the projection still exceeds the inline budget the engine fails loud instead of silently dropping spans.

Prerequisites:

- A running bridge that accepts `POST /v1/chat/completions` and routes the configured model to the prime backend.
- No key handling here: the bridge owns provider credentials.

Configuration (flags first, then environment, then defaults):

| Setting | Source | Default |
| --- | --- | --- |
| Bridge root URL | `TRACES_PRIME_BRIDGE_URL` | `http://localhost:4181` |
| Model | `--model`, then `TRACES_PRIME_MODEL` | `prime/zai/glm-5.2` |
| Per-call deadline | `TRACES_PRIME_TIMEOUT_MS` | `1200000` (20 min) |
| Question | `--analyzer-prompt` | a general diagnosis question |

```bash
traces analyze --last 1 --analyzer prime
TRACES_PRIME_BRIDGE_URL=http://localhost:4181 traces analyze --last 1 --analyzer prime --analyzer-prompt "find unsupported completion claims"
```

Output expectations:

- The reply contract is one fenced JSON block of short strings citing span ids verbatim; a structurally malformed reply gets exactly one bounded repair turn that carries the malformed reply and the contract, never the trajectory.
- A still-malformed reply after repair is a failed result (`ok: false`) with the raw reply preserved for inspection; one failed engine never discards the other results.
- Valid rows become full `findings` with `trace://` span evidence, validated against the artifact like every other findings-kind engine; rows citing unknown or ambiguous span ids are rejected with a recorded reason.
- Zero findings from a well-formed reply is an honest null, not a failure.
- Bridge-reported token usage and call counts are recorded in the result output; cost stays uncaptured because this adapter has no pricing table.

The scored prime-vs-dspy comparison — same trajectories, same scoring — lives in `@tangle-network/agent-eval`'s analyst benchmark (`runAnalystBenchmark`); this engine is the capture-side entry point, not the scoreboard.

## Write one analyst

An analyst receives a paged trace store and returns typed findings.
Expand Down
15 changes: 14 additions & 1 deletion examples/external-engines.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
commandRedactor,
haloAnalyzer,
hodoscopeAnalyzer,
primeAnalyzer,
writeOtlpFile,
} from '@tangle-network/traces'

Expand Down Expand Up @@ -41,7 +42,19 @@ console.log(
: `hodoscope unavailable: ${discovery.error}`,
)

// 3) External REDACTOR: scrub prose with your own PII model before upload. The
// 3) PRIME ENGINE: one-shot RLM through an OpenAI-compatible bridge (run
// cli-bridge's prime backend locally, or set TRACES_PRIME_BRIDGE_URL).
// The full span projection travels inline; findings come back with
// validated trace:// span evidence.
const prime = primeAnalyzer({ defaultPrompt: 'find unsupported completion claims' })
const primeResult = await prime.analyze(otlp)
console.log(
primeResult.ok
? `${primeResult.findings?.length ?? 0} prime finding(s)\n${primeResult.output}`
: `prime bridge unavailable: ${primeResult.error}`,
)

// 4) External REDACTOR: scrub prose with your own PII model before upload. The
// command reads a JSON array of strings on stdin and writes the scrubbed array
// on stdout (a 3-line wrapper adapts openai/privacy-filter's `opf`).
const redactor = commandRedactor({ command: 'my-pii-scrubber' })
Expand Down
Loading
Loading