feat(adaptive): add opt-in caching for tool results - #597
feat(adaptive): add opt-in caching for tool results#597zhongxuanwang-nv wants to merge 6 commits into
Conversation
Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
WalkthroughThe response-cache system now supports opt-in tool-result caching. It adds tool policies, deterministic keys, runtime interception, validation, cache metadata, CLI diagnostics, and Node, Go, and Python configuration APIs. ChangesTool caching core
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant ToolCaller
participant ToolExecutionIntercept
participant CacheStore
participant ToolProvider
ToolCaller->>ToolExecutionIntercept: invoke configured tool
ToolExecutionIntercept->>CacheStore: read tool cache key
CacheStore-->>ToolExecutionIntercept: hit or miss
alt cache hit
ToolExecutionIntercept-->>ToolCaller: return cached result
else cache miss or bypass
ToolExecutionIntercept->>ToolProvider: execute tool
ToolProvider-->>ToolExecutionIntercept: return tool result
ToolExecutionIntercept->>CacheStore: store eligible result
ToolExecutionIntercept-->>ToolCaller: return live result
end
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
size:XL
Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
63c056d to
aa0ff7e
Compare
Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
aa0ff7e to
34cfa63
Compare
Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/adaptive/src/response_cache/tool.rs`:
- Around line 338-342: Move the inline #[cfg(test)] mod tests block in tool.rs
into a crate-level test file under tests/unit/response_cache, preserving the
policy-resolution and wildcard test bodies and imports. Replace the inline block
with a #[path] module include matching the existing sanctioned pattern, so the
tests retain access to resolve_policy, wildcard_match, wildcard_rank, and
wildcard_patterns_overlap without changing their visibility.
In `@crates/adaptive/tests/integration/response_cache_benchmark_tests.rs`:
- Around line 333-354: Move the shared helper implementations counting_tool and
tool_call from the integration test binary into response_cache_common.rs, then
remove the local duplicates and import the shared symbols in both integration
modules. Preserve their existing signatures and behavior, including run counting
and unwrapped tool execution.
In `@crates/cli/tests/coverage/shared/doctor_tests.rs`:
- Around line 1247-1289: Extend
collect_observability_reports_tool_cache_surface_for_cacheable_overrides to
cover both tool-cache branches: add a configuration with response-cache tools
disabled and assert the “configured but disabled” details, then add a separate
cache configuration with at least one cacheable class and assert its class count
independently from the override count. Preserve the existing enabled-tools
assertions for the cacheable override case.
In `@crates/node/tests/adaptive_runtime_tests.mjs`:
- Around line 29-42: Extend the test around adaptive.validateConfig and
adaptive.ComponentSpec to assert that the serialized configuration preserves
responseCache.tools in snake_case form, including its enabled state and
class/member data. Keep the existing empty-diagnostics assertion, but verify the
positive serialized payload so validator reachability is directly covered.
In `@crates/node/tests/adaptive_tests.mjs`:
- Around line 347-353: Align the response_cache.tools serialization asserted in
spec.config.response_cache.tools with the documented canonical shape used across
bindings. Include the canonical priority value and empty default ToolClass
fields rather than preserving their omission, while keeping the existing cache
settings unchanged.
In `@python/tests/test_adaptive_config.py`:
- Around line 265-280: Update test_invalid_tool_cache_section_is_rejected so
ResponseCacheConfig receives a non-empty namespace while retaining the duplicate
tool-class setup. Keep the existing assertion for
response_cache.tool_multiple_classes, ensuring the test isolates that diagnostic
without also triggering response_cache.missing_namespace.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 4fce90ea-a3ce-48f7-846e-c63746b28d93
📒 Files selected for processing (31)
crates/adaptive/src/config.rscrates/adaptive/src/lib.rscrates/adaptive/src/plugin_component.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/response_cache/tool.rscrates/adaptive/src/runtime/features.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/cli/src/diagnostics/mod.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/node/adaptive.d.tscrates/node/adaptive.jscrates/node/tests/adaptive_runtime_tests.mjscrates/node/tests/adaptive_tests.mjsgo/nemo_relay/adaptive.gogo/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gopython/nemo_relay/adaptive.pypython/nemo_relay/adaptive.pyipython/tests/test_adaptive_config.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (31)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/plugin_component.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rspython/nemo_relay/adaptive.pycrates/adaptive/src/response_cache/config.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/node/tests/adaptive_runtime_tests.mjscrates/adaptive/src/plugin_component.rscrates/node/tests/adaptive_tests.mjscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/node/adaptive.jscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rspython/nemo_relay/adaptive.pycrates/adaptive/src/response_cache/config.rscrates/node/adaptive.d.tscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/node/adaptive.jscrates/adaptive/tests/unit/response_cache/store_tests.rsgo/nemo_relay/adaptive/adaptive.gocrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rspython/nemo_relay/adaptive.pycrates/adaptive/src/response_cache/config.rscrates/node/adaptive.d.tscrates/adaptive/tests/unit/response_cache/tool_tests.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/node/adaptive.jscrates/adaptive/tests/unit/response_cache/store_tests.rsgo/nemo_relay/adaptive/adaptive.gocrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/src/response_cache/config.rscrates/node/adaptive.d.tscrates/adaptive/tests/unit/response_cache/tool_tests.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
crates/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep
crates/adaptivealigned with the canonical adaptive config schema, built-in section helpers, plugin lifecycle, and validation/report behavior.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/plugin_component.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/node/tests/adaptive_runtime_tests.mjscrates/adaptive/src/plugin_component.rscrates/node/tests/adaptive_tests.mjscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/node/adaptive.jscrates/adaptive/tests/unit/response_cache/store_tests.rsgo/nemo_relay/adaptive/adaptive.gocrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rspython/nemo_relay/adaptive.pypython/nemo_relay/adaptive.pyicrates/adaptive/src/response_cache/config.rscrates/node/adaptive.d.tscrates/adaptive/tests/unit/response_cache/tool_tests.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/plugin_component.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/node/adaptive.jscrates/adaptive/tests/unit/response_cache/store_tests.rsgo/nemo_relay/adaptive/adaptive.gocrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rspython/nemo_relay/adaptive.pycrates/adaptive/src/response_cache/config.rscrates/node/adaptive.d.tscrates/adaptive/tests/unit/response_cache/tool_tests.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/node/adaptive.jscrates/adaptive/tests/unit/response_cache/store_tests.rsgo/nemo_relay/adaptive/adaptive.gocrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rspython/nemo_relay/adaptive.pycrates/adaptive/src/response_cache/config.rscrates/node/adaptive.d.tscrates/adaptive/tests/unit/response_cache/tool_tests.rsgo/nemo_relay/adaptive.gocrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/adaptive/src/response_cache/store.rscrates/adaptive/src/plugin_component.rscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rscrates/adaptive/src/runtime/validation.rscrates/adaptive/src/runtime/features.rscrates/adaptive/tests/integration/response_cache_benchmark_tests.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rscrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/src/response_cache/config.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rscrates/adaptive/src/response_cache/tool.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/adaptive/tests/unit/response_cache/replay_tests.rscrates/cli/tests/coverage/shared/doctor_tests.rscrates/node/tests/adaptive_runtime_tests.mjscrates/node/tests/adaptive_tests.mjscrates/adaptive/tests/unit/config_tests.rscrates/adaptive/tests/unit/response_cache/store_tests.rscrates/adaptive/tests/unit/response_cache/key_tests.rspython/tests/test_adaptive_config.pycrates/adaptive/tests/integration/response_cache_benchmark_tests.rsgo/nemo_relay/adaptive_runtime_test.gocrates/adaptive/tests/unit/response_cache/intercept_tests.rscrates/adaptive/tests/unit/response_cache/tool_tests.rscrates/adaptive/tests/integration/response_cache_tests.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/adaptive/src/response_cache/store.rscrates/adaptive/src/plugin_component.rscrates/cli/src/diagnostics/mod.rscrates/adaptive/src/lib.rscrates/adaptive/src/response_cache/mark.rscrates/adaptive/src/runtime/validation.rspython/tests/test_adaptive_config.pycrates/adaptive/src/runtime/features.rscrates/adaptive/src/config.rscrates/adaptive/src/response_cache/key.rscrates/adaptive/src/response_cache/mod.rspython/nemo_relay/adaptive.pycrates/adaptive/src/response_cache/config.rscrates/adaptive/src/response_cache/tool.rs
crates/node/**/*.{js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use
camelCasefor Node.js public APIs.
Files:
crates/node/tests/adaptive_runtime_tests.mjscrates/node/tests/adaptive_tests.mjscrates/node/adaptive.jscrates/node/adaptive.d.ts
crates/{python,ffi,node}/**/*
⚙️ CodeRabbit configuration file
crates/{python,ffi,node}/**/*: Treat binding changes as public API changes. Check for parity with the other language bindings, FFI ownership/lifetime safety,
callback error propagation, stable type conversion, and consistent async/stream semantics.
Flag changes that update one binding without corresponding tests or documentation for the same surface elsewhere.
Files:
crates/node/tests/adaptive_runtime_tests.mjscrates/node/tests/adaptive_tests.mjscrates/node/adaptive.jscrates/node/adaptive.d.ts
crates/node/**/*.{js,ts,jsx,tsx,json}
📄 CodeRabbit inference engine (.agents/skills/test-node-binding/SKILL.md)
Format changed Node files with
npm run format --workspace=nemo-relay-node
Files:
crates/node/adaptive.jscrates/node/adaptive.d.ts
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
crates/node/adaptive.jsgo/nemo_relay/adaptive/adaptive.gopython/tests/test_adaptive_config.pygo/nemo_relay/adaptive_runtime_test.gopython/nemo_relay/adaptive.pycrates/node/adaptive.d.tsgo/nemo_relay/adaptive.go
go/nemo_relay/**/*.go
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
go/nemo_relay/**/*.go: Format changed Go packages withcd go/nemo_relay && go fmt ./...
Run Go tests withjust test-goto build and test the NeMo Relay Go binding
Usejust build-gowhen you want an explicit build-only pass or need the artifact for other work
Usejust ci=true test-gowhen you need the CI-style coverage and JUnit path
On macOS, setDYLD_LIBRARY_PATHto the../../target/releasedirectory before running the rawgo testcommand directlyUse
PascalCasefor public Go APIs.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gogo/nemo_relay/adaptive.go
**/*.go
📄 CodeRabbit inference engine (CONTRIBUTING.md)
When changing the experimental Go binding, format Go code with
gofmtand keepgo vet ./...passing.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gogo/nemo_relay/adaptive.go
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gopython/nemo_relay/adaptive.pypython/nemo_relay/adaptive.pyigo/nemo_relay/adaptive.go
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gopython/nemo_relay/adaptive.pypython/nemo_relay/adaptive.pyigo/nemo_relay/adaptive.go
go/nemo_relay/adaptive/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep Go adaptive helpers under
go/nemo_relay/adaptivesynchronized with the shared adaptive config model and lifecycle.
Files:
go/nemo_relay/adaptive/adaptive.go
go/nemo_relay/**
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep shared plugin helpers in
go/nemo_relayaligned with plugin registration, composition, and lifecycle behavior.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gogo/nemo_relay/adaptive.go
go/nemo_relay/**/*
⚙️ CodeRabbit configuration file
go/nemo_relay/**/*: Review Go binding changes for cgo memory ownership, race safety, callback cleanup, idiomatic exported APIs, and parity with Rust/FFI behavior.
Any API change should include focused Go tests and consider race-test behavior.
Files:
go/nemo_relay/adaptive/adaptive.gogo/nemo_relay/adaptive_runtime_test.gogo/nemo_relay/adaptive.go
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: When changing the Python wrapper package, tests, or docs tooling, lint with Ruff (E,F,W,I), format with Ruff formatter (120-character lines, double quotes), and passtytype checking.
Add the SPDX license header to all Python source files using the#comment form.
Files:
python/tests/test_adaptive_config.pypython/nemo_relay/adaptive.py
python/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)
python/tests/**/*.py: Pytest is used to run tests.
Do not add@pytest.mark.asyncioto any test; async tests are automatically detected and run by the async runner.
Do not add a-> Nonereturn type annotation to test functions.
When mocking a class, do not define a new class; useunittest.mock.MagicMockorunittest.mock.AsyncMock, with thespecconstructor argument when necessary.
Name mocked classes with themockprefix, notfake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in aconftest.pyfile.
When creating a fixture, use@pytest.fixture(name="<fixture_name>"[, scope="<scope>"])and define the fixture function asdef <fixture_name>_fixture() -> <return_type>:; only specifyscopewhen it is notfunction.
Preferpytest.mark.parametrizeover creating individual tests for different input types.
Files:
python/tests/test_adaptive_config.py
python/nemo_relay/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Python wrapper modules live under
python/nemo_relay/, and the native extension is built fromcrates/pythonwithmaturin.
Files:
python/nemo_relay/adaptive.py
python/nemo_relay/{adaptive.py,plugin.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-optimizer/SKILL.md)
Keep Python adaptive/plugin wrappers in
python/nemo_relay/adaptive.pyandpython/nemo_relay/plugin.pysynchronized with the shared adaptive/plugin boundary and lifecycle.
Files:
python/nemo_relay/adaptive.py
python/nemo_relay/**/*
⚙️ CodeRabbit configuration file
python/nemo_relay/**/*: Review Python wrapper changes for typed API consistency, contextvars-based scope isolation, async behavior, and parity with the native extension.
Stubs and runtime implementations should stay aligned.
Files:
python/nemo_relay/adaptive.pypython/nemo_relay/adaptive.pyi
crates/node/**/*.{ts,tsx,d.ts}
📄 CodeRabbit inference engine (.agents/skills/test-node-binding/SKILL.md)
Use
npm run check:docstrings --workspace=nemo-relay-nodeto validate public API docstring checks when surface docs changed
Files:
crates/node/adaptive.d.ts
🧠 Learnings (2)
📚 Learning: 2026-07-28T20:33:25.156Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 572
File: go/nemo_relay/adaptive_runtime_test.go:214-238
Timestamp: 2026-07-28T20:33:25.156Z
Learning: When adding/adjusting Go unit tests for `BuildCacheRequestFacts` (request-ID validation and related request parsing), set `CacheRequestFactsInput.Provider` to a valid provider in all tests that are intended to isolate request-ID behavior—because `BuildCacheRequestFacts` does not validate `Provider`. Then add separate test coverage for malformed `AnnotatedRequest` JSON so JSON parsing failures are not conflated with `Provider`-related inputs.
Applied to files:
go/nemo_relay/adaptive_runtime_test.go
📚 Learning: 2026-05-07T18:04:44.387Z
Learnt from: mnajafian-nv
Repo: NVIDIA/NeMo-Flow PR: 67
File: integrations/openclaw/src/modules.ts:1-2
Timestamp: 2026-05-07T18:04:44.387Z
Learning: In NVIDIA/NeMo-Flow, TypeScript source files should use `//` line comments for SPDX headers (e.g., `// SPDX-FileCopyrightText: ...` and `// SPDX-License-Identifier: ...`) rather than C-style block comments (`/* ... */`). The repo’s copyright checker enforces this mapping, so `//` SPDX headers in `.ts` files should not be flagged as a style violation.
Applied to files:
crates/node/adaptive.d.ts
🔇 Additional comments (40)
crates/adaptive/src/config.rs (1)
10-10: LGTM!Also applies to: 37-38, 194-195, 221-223, 237-237, 413-413
crates/adaptive/src/lib.rs (1)
37-37: LGTM!Also applies to: 61-61
crates/adaptive/src/response_cache/config.rs (1)
12-13: LGTM!Also applies to: 68-100, 102-118, 120-139
crates/adaptive/src/plugin_component.rs (1)
324-324: LGTM!Also applies to: 349-351, 378-449
crates/adaptive/src/runtime/validation.rs (1)
4-13: LGTM!Also applies to: 205-292, 294-347, 349-380
crates/adaptive/tests/unit/config_tests.rs (1)
10-11: LGTM!Also applies to: 37-50
crates/adaptive/src/response_cache/key.rs (1)
17-18: LGTM!Also applies to: 119-120, 131-131, 220-260, 450-469
crates/adaptive/src/response_cache/store.rs (1)
33-33: LGTM!crates/adaptive/src/response_cache/mark.rs (1)
106-128: LGTM!Also applies to: 137-142, 163-168, 177-177, 219-224
crates/adaptive/src/response_cache/mod.rs (1)
4-8: LGTM!Also applies to: 23-34
crates/adaptive/src/response_cache/tool.rs (3)
37-124: LGTM!
144-207: LGTM!
230-336: LGTM!crates/adaptive/src/runtime/features.rs (1)
45-47: LGTM!Also applies to: 489-489, 790-790, 800-800, 813-820, 835-845
crates/node/adaptive.d.ts (3)
55-70: LGTM!
83-134: LGTM!
335-336: LGTM!crates/node/adaptive.js (2)
153-154: LGTM!
188-241: LGTM!go/nemo_relay/adaptive.go (1)
70-70: LGTM!Also applies to: 97-129, 248-256
go/nemo_relay/adaptive/adaptive.go (1)
55-69: LGTM!Also applies to: 147-151
python/nemo_relay/adaptive.py (3)
254-362: LGTM!
365-411: LGTM!
429-429: LGTM!Also applies to: 556-558
python/nemo_relay/adaptive.pyi (2)
184-226: LGTM!
245-256: LGTM!Also applies to: 275-275
crates/cli/src/diagnostics/mod.rs (1)
667-695: LGTM!crates/adaptive/tests/integration/response_cache_benchmark_tests.rs (1)
356-419: LGTM!Also applies to: 421-515
crates/adaptive/tests/integration/response_cache_tests.rs (2)
1847-1876: LGTM!Also applies to: 1926-2252, 2254-2539
1807-1816: 📐 Maintainability & Code QualityDo not apply this lint finding.
field_reassign_with_defaultchecks assignments to top-level fields. All three sites assign the nested fieldconfig.backend.kind, so they do not match this lint pattern.> Likely an incorrect or invalid review comment.crates/adaptive/tests/unit/response_cache/intercept_tests.rs (1)
6-102: LGTM!Also applies to: 145-178, 180-242, 244-295
crates/adaptive/tests/unit/response_cache/key_tests.rs (1)
820-842: LGTM!Also applies to: 844-924, 926-964
crates/adaptive/tests/unit/response_cache/replay_tests.rs (1)
151-179: LGTM!crates/adaptive/tests/unit/response_cache/store_tests.rs (2)
218-233: LGTM!Also applies to: 246-255, 265-274
234-238: 🎯 Functional CorrectnessKeep the assertion.
entry_sizedoes not usecreated_unix_msorexpires_unix_ms, so the comparison is stable.> Likely an incorrect or invalid review comment.crates/adaptive/tests/unit/response_cache/tool_tests.rs (1)
1-50: LGTM!Also applies to: 52-125
crates/node/tests/adaptive_runtime_tests.mjs (1)
44-63: LGTM!crates/node/tests/adaptive_tests.mjs (1)
334-346: LGTM!python/tests/test_adaptive_config.py (1)
21-23: LGTM!Also applies to: 224-242, 244-263
go/nemo_relay/adaptive_runtime_test.go (1)
251-335: LGTM!Also applies to: 337-364
|
Big PR. We are going to need to align this with #575 for the 0.8 release. Please include me in reviews before merge |
|
@bbednarski9 Sounds great! I think 575 can go in before this, and I would request your review after this PR is completely ready! |
Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
willkill07
changed the title
2 participants
Overview
Adds opt-in tool-result caching to Adaptive response caching. A cache hit suppresses the real tool call, so caching remains disabled by default and only explicitly classified read-only, TTL-stable tools are eligible.
Details
mainat93c8be34; earlier logical-key and staged documentation changes are not part of this diff.just ci=true test-rust(3,717 passed), response-cache unit tests (61 passed), response-cache integration tests (50 passed), Python (613 passed), Node.js (342 passed), Go, workspace clippy with warnings denied, and all repository pre-commit hooks pass. The canonical non-CI Rust run also completed all unit/integration tests but encounters the existing upstreamscope_stack.rsdoctest reference to unavailablenemo_relay::Result.Where should the reviewer start?
Start with
crates/adaptive/src/response_cache/tool.rsfor policy resolution and fail-open execution, thencrates/adaptive/tests/integration/response_cache_tests.rsfor the behavioral contract.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
New Features
Bug Fixes