docs: clarify rootless propagation trace semantics - #604
Conversation
Document that rootless contexts preserve Relay event parentage while starting a new OpenTelemetry trace. Add focused coverage for the exported root span identity and empty parent. Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
github-actions
Bot
added
size:S
WalkthroughThe change documents rootless Relay propagation and OpenTelemetry trace behavior. It adds a regression test that verifies rootless propagation creates a new local root trace. ChangesRootless OpenTelemetry propagation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/api/runtime/scope_stack.rs`:
- Around line 43-51: Update the documentation for PropagationContext’s missing
root_uuid behavior to state that importing via ScopeStack::from_propagation
preserves Relay parentage and starts a new OpenTelemetry trace at the first
local span after import, rather than implying the trace starts during import.
In `@crates/core/tests/unit/observability/otel_tests.rs`:
- Around line 381-404: Extend the test around the imported stack created by
create_scope_stack_from_propagation to assert capture_propagation_context()
returns root_uuid: None while retaining parent_uuid. Also exercise
fork_scope_stack() with this rootless imported stack and assert the fork
preserves root_uuid() as None and keeps the expected top().uuid.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 594c13d7-933b-41b8-a9ef-009869512cca
📒 Files selected for processing (4)
crates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/observability/otel_tests.rsdocs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
📜 Review details
🧰 Additional context used
📓 Path-based instructions (24)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/core/tests/unit/observability/otel_tests.rsdocs/about-nemo-relay/concepts/scopes.mdxcrates/core/src/api/runtime/scope_stack.rsdocs/configure-plugins/observability/opentelemetry.mdx
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/api/runtime/scope_stack.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/observability/otel_tests.rs
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/about-nemo-relay/concepts/scopes.mdxdocs/configure-plugins/observability/opentelemetry.mdx
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/runtime/scope_stack.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/api/runtime/scope_stack.rs
crates/core/src/api/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Preserve the documented pipeline order: conditional guardrails, request intercepts, request sanitization, execution intercepts, and response sanitization for tool and LLM execution; specialized sanitization, event creation, and dispatch for mark and scope events.
Files:
crates/core/src/api/runtime/scope_stack.rs
docs/{about-nemo-relay/concepts/subscribers.mdx,configure-plugins/observability/**/*.mdx}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Update observability documentation and examples alongside implementation changes, including configuration version 3 with one
opentelemetrysection containing typed endpoints and no standalone public OpenInference surface.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
🧠 Learnings (3)
📚 Learning: 2026-07-28T03:31:05.964Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 564
File: crates/core/src/api/runtime/subscriber_dispatcher.rs:297-314
Timestamp: 2026-07-28T03:31:05.964Z
Learning: In this codebase’s runtime API, do not implement incremental native LLM stream forwarding via the native ABI v3 asynchronous middleware protocol (it can only settle a single JSON value via a one-shot completion handle and cannot forward stream chunks incrementally). If a latency-sensitive plugin needs streaming behavior, review for use of synchronous native stream intercepts or worker plugins instead of trying to chunk-deliver or incrementally forward over the ABI v3 async path.
Applied to files:
crates/core/src/api/runtime/scope_stack.rs
📚 Learning: 2026-07-28T20:07:29.880Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 571
File: crates/core/src/api/runtime/state.rs:996-1020
Timestamp: 2026-07-28T20:07:29.880Z
Learning: In NeMo Relay (RELAY-509), sanitizer callback failures must be treated as intentional fail-open behavior. When an event/tool (request/response) or LLM (request/response) sanitizer callback fails, the sanitizer chain should retain and publish the last valid event/payload snapshot (rather than dropping/invalidating the data) and log the failure including callback context (e.g., which sanitizer/callback failed and relevant identifiers). Apply this consistently across all sanitizer chains mentioned in the RELAY-509 documentation/migration guide.
Applied to files:
crates/core/src/api/runtime/scope_stack.rs
📚 Learning: 2026-07-14T02:53:59.997Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 415
File: docs/configure-plugins/observability/opentelemetry.mdx:98-113
Timestamp: 2026-07-14T02:53:59.997Z
Learning: In NeMo-Relay’s OpenTelemetry/OpenInference observability projection docs under docs/configure-plugins/observability/, document the projected-attribute contract as follows: (1) emit scalar top-level `data`/`metadata` fields as typed dotted OTLP attributes (for example, `nemo_relay.start.metadata.tenant`); (2) keep nested objects/arrays as JSON strings at their top-level OTLP attribute (rather than expanding them into nested OTLP attributes); and (3) do not reference the legacy `*_json` payload attributes (e.g., `data_json`, `metadata_json`, `input_json`) because they were intentionally removed as a breaking change.
Applied to files:
docs/configure-plugins/observability/opentelemetry.mdx
🔇 Additional comments (4)
crates/core/src/api/runtime/scope_stack.rs (2)
447-449: LGTM!
470-474: 📐 Maintainability & Code QualityComplete the required Rust/core validation before handoff.
The supplied summary says that the focused Rust test reached code generation but could not complete linking because the disk ran out of space. Treat this result as incomplete. After freeing disk space, run
just test-rust,cargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,validate-change,just test-python,just test-go,just test-node, anduv run pre-commit run --all-files.As per coding guidelines, Rust changes require the Rust checks,
crates/corechanges requirevalidate-changeand the full language matrix, and handoff requires the all-files pre-commit pass.Source: Coding guidelines
docs/about-nemo-relay/concepts/scopes.mdx (1)
150-151: LGTM!Also applies to: 219-224
docs/configure-plugins/observability/opentelemetry.mdx (1)
32-36: 📐 Maintainability & Code QualityClarify W3C context precedence for rootless propagation.
The paragraph says that rootless Relay propagation starts a new OpenTelemetry trace and also recommends carrying
traceparentandtracestate. A W3Ctraceparentnormally identifies an existing remote parent. State whether integrations must ignore that parent for Relay-derived spans, use the fields only for sampling or vendor state, or let W3C extraction override the rootless rule.As per path instructions, documentation must be technically accurate against the current API and consistent across language bindings.
Source: Path instructions
Codecov Report✅ All modified and coverable lines are covered by tests. ❌ Your project check has failed because the head coverage (94.07%) is below the target coverage (95.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## release/0.7 #604 +/- ##
===============================================
- Coverage 92.83% 92.82% -0.01%
===============================================
Files 326 326
Lines 95101 95101
Branches 113 113
===============================================
- Hits 88278 88271 -7
- Misses 6823 6829 +6
- Partials 0 1 +1
... and 6 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
|
/merge |
5 participants
Overview
Clarify that importing a rootless Relay
PropagationContextpreserves Relay event parentage while its first local span intentionally starts a new OpenTelemetry trace.Details
PropagationContext,capture_propagation_context(), andfork_scope_stack().Validation:
cargo fmt --all -- --checkpassed.git diff --checkpassed.cargo test -p nemo-relay --lib rootless_propagation_starts_a_new_otel_tracepassed (1 test).Where should the reviewer start?
Start with
rootless_propagation_starts_a_new_otel_traceincrates/core/tests/unit/observability/otel_tests.rs, then review the public contract onPropagationContextincrates/core/src/api/runtime/scope_stack.rs.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
Documentation
traceparentandtracestatealongside Relay propagation.Tests