Skip to content

fix: report cancelled Python scopes as errors - #620

Merged
rapids-bot[bot] merged 3 commits into
NVIDIA:release/0.7from
mnajafian-nv:fix/python-cancelled-scope-status
Jul 31, 2026
Merged

fix: report cancelled Python scopes as errors#620
rapids-bot[bot] merged 3 commits into
NVIDIA:release/0.7from
mnajafian-nv:fix/python-cancelled-scope-status

Conversation

@mnajafian-nv

@mnajafian-nv mnajafian-nv commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Overview

Mark cancelled Python scopes as errors so scope end events do not leave otel.status_code=UNSET when asyncio.CancelledError unwinds through scope().

  • I confirm this contribution is my own work, or I have the right to submit it under this project's license.
  • I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

  • Treat asyncio.CancelledError as an explicit error path in python/nemo_relay/scope.py.
  • Set otel.status_code to ERROR and default the status description to cancelled when the cancellation has no message.
  • Preserve the existing re-raise behavior so task cancellation semantics do not change.
  • Add async regression coverage for both the fallback cancellation description and an explicit cancellation message in python/tests/test_scope.py.

Validation:

  • uv run pytest python/tests/test_scope.py -k 'cancelled_task_sets_error_status or scope_ctx_mgr'
  • just test-python
  • uv run pre-commit run --files python/nemo_relay/scope.py python/tests/test_scope.py
  • uv run pre-commit run --all-files (restored unrelated generated Node artifacts after local hook churn; the Python change remained clean)

Where should the reviewer start?

Start in python/nemo_relay/scope.py at the new asyncio.CancelledError branch in scope(), then review python/tests/test_scope.py for the parameterized cancellation regression.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • Relates to: none

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of cancelled operations within scopes.
    • Cancellation events now report an error status with a clear cancellation description before the cancellation is propagated.

Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The scope context manager now records asyncio.CancelledError as an error with a cancellation description before re-raising it. Tests cover default and custom cancellation messages and verify the emitted end event.

Changes

Scope cancellation handling

Layer / File(s) Summary
Record cancellation status and validate emitted events
python/nemo_relay/scope.py, python/tests/test_scope.py
The scope records cancellation as an ERROR status, preserves CancelledError propagation, and supports default or custom cancellation descriptions. Tests verify the emitted scope end event.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • NVIDIA/NeMo-Relay#615: Implements the same asyncio.CancelledError handling and cancellation-status test.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title follows Conventional Commits format and clearly describes the cancellation error-handling change.
Description check ✅ Passed The description includes the required overview, details, reviewer guidance, validation steps, and related-issues section.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size:S PR is small Bug issue describes bug; PR fixes bug lang:python PR changes/introduces Python code labels Jul 31, 2026
@mnajafian-nv mnajafian-nv self-assigned this Jul 31, 2026
@mnajafian-nv
mnajafian-nv marked this pull request as ready for review July 31, 2026 22:01
@mnajafian-nv
mnajafian-nv requested a review from a team as a code owner July 31, 2026 22:01

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/tests/test_scope.py`:
- Around line 106-143: Extend
test_scope_ctx_mgr_cancelled_task_sets_error_status to assert after
cancel_within_scope() raises that scope.get_handle().name is "root", verifying
the cancelled scope was removed from the current scope stack while preserving
the existing lifecycle-event assertions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: cb62cc18-e216-4543-8630-61c708c148e9

📥 Commits

Reviewing files that changed from the base of the PR and between 899000d and d002414.

📒 Files selected for processing (2)
  • python/nemo_relay/scope.py
  • python/tests/test_scope.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
  • GitHub Check: Preview docs
  • GitHub Check: Check / Run
🧰 Additional context used
📓 Path-based instructions (15)
**/*.{rs,py}

📄 CodeRabbit inference engine (AGENTS.md)

Follow binding naming conventions in Rust and Python: use snake_case.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
**/*.{rs,py,js,mjs,cjs,ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{rs,py,js,mjs,cjs,ts,tsx}: Use Json = serde_json::Value in Rust-facing runtime APIs where the existing code expects JSON payloads.
Use Result<T> with FlowError in core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
**/*.py

📄 CodeRabbit inference engine (CONTRIBUTING.md)

**/*.py: When changing the Python wrapper package, tests, or docs tooling, lint with Ruff (E, F, W, I), format with Ruff formatter (120-character lines, double quotes), and pass ty type checking.
Add the SPDX license header to all Python source files using the # comment form.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
**/*.{rs,py,go,js,ts,c,h}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Use language-appropriate naming conventions: Rust snake_case, C FFI exports prefixed nemo_relay_, Go PascalCase, Node.js camelCase, and Python snake_case.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
{crates/**/src/**/*.rs,python/**/*.py}

📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)

Do not add tests under src; Rust tests belong in crate tests/ trees, and Python SDK tests belong under python/tests.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
python/tests/**/*.py

📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)

python/tests/**/*.py: Pytest is used to run tests.
Do not add @pytest.mark.asyncio to any test; async tests are automatically detected and run by the async runner.
Do not add a -> None return type annotation to test functions.
When mocking a class, do not define a new class; use unittest.mock.MagicMock or unittest.mock.AsyncMock, with the spec constructor argument when necessary.
Name mocked classes with the mock prefix, not fake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in a conftest.py file.
When creating a fixture, use @pytest.fixture(name="<fixture_name>"[, scope="<scope>"]) and define the fixture function as def <fixture_name>_fixture() -> <return_type>:; only specify scope when it is not function.
Prefer pytest.mark.parametrize over creating individual tests for different input types.

Files:

  • python/tests/test_scope.py
**/*

📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)

**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, use maintain-dynamic-plugins and include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, prefer uv run pre-commit run --files <changed files...>.
Before review or handoff, run uv run pre-commit run --all-files.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
**/*.{rs,py,go,js,ts}

📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)

If a language surface changed, always run that language's test target even when Rust core did not change.

**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolve header_env values at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests and just test-rust when event fields change; run just test-python, just test-go, and just test-node when binding-native configuration or lifecycle changes.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}

📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)

Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
**/*.{md,mdx,py,go,js,ts}

📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)

Examples and documentation must use each exporter's documented flush/deregister order before shutdown.

Files:

  • python/tests/test_scope.py
  • python/nemo_relay/scope.py
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}

⚙️ CodeRabbit configuration file

{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.

Files:

  • python/tests/test_scope.py
python/nemo_relay/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

Python wrapper modules live under python/nemo_relay/, and the native extension is built from crates/python with maturin.

Files:

  • python/nemo_relay/scope.py
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}

📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)

Update the language-native bindings for every exposed surface in Python, Go, and Node.js.

Files:

  • python/nemo_relay/scope.py
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}

📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)

Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.

Files:

  • python/nemo_relay/scope.py
python/nemo_relay/**/*

⚙️ CodeRabbit configuration file

python/nemo_relay/**/*: Review Python wrapper changes for typed API consistency, contextvars-based scope isolation, async behavior, and parity with the native extension.
Stubs and runtime implementations should stay aligned.

Files:

  • python/nemo_relay/scope.py
🔇 Additional comments (3)
python/nemo_relay/scope.py (2)

19-19: LGTM!


253-256: LGTM!

python/tests/test_scope.py (1)

6-15: LGTM!

Comment thread python/tests/test_scope.py
@github-actions

Copy link
Copy Markdown

@willkill07 willkill07 added this to the 0.7 milestone Jul 31, 2026
@mnajafian-nv

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit 8fcfb1e into NVIDIA:release/0.7 Jul 31, 2026
37 checks passed
@mnajafian-nv
mnajafian-nv deleted the fix/python-cancelled-scope-status branch July 31, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug issue describes bug; PR fixes bug lang:python PR changes/introduces Python code size:S PR is small

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants