Skip to content

ci: split the wire round trips out of the fuzz marker - #181

Merged
ericeil merged 2 commits into
masterfrom
eric/fuzzFix
Aug 19, 2026
Merged

ci: split the wire round trips out of the fuzz marker#181
ericeil merged 2 commits into
masterfrom
eric/fuzzFix

Conversation

@ericeil

@ericeil ericeil commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

The nightly Integration Tests job died in run template fuzz tests the first night the wire
round-trip suite existed — run 32218703320,
13 errors, one cause.

-m 'fuzz' tests selects two unrelated suites, and only one of them is Python. The 36 template
tests the step is named for, and the 13 round trips in tests/test_wire_roundtrip.py, whose
wire_echo fixture builds wire-echo with cargo at test time. The step runs pytest -n 2 and that
fixture is session-scoped per xdist worker, so both workers asked rustup to install the pinned 1.96
toolchain at the same instant. rustup's install path is not safe against concurrent invocations: one
call rolled back (failed to install component: 'rustc-x86_64-unknown-linux-gnu', detected conflict: 'bin/rust-gdb') and the other found what the rollback left behind (the 'cargo' binary ... is not applicable to the '1.96-x86_64-unknown-linux-gnu' toolchain). Not a flake that turned:
rust-toolchain.toml and the suite both landed in #97 on 08-18, after that morning's scheduled run,
so this was the first nightly to select them — 29 fuzz tests on 08-18, 49 on 08-19.

Installing the toolchain first would stop the crash and buy nothing. That is what pytest.yml
does, for exactly this hazard, and it is the wrong answer here: this job would pay an uncached cargo
build inside a 20-minute budget to re-run tests it cannot deepen and does not uniquely cover. They
pin max_examples=200 in their own @settings, so this job's HYPOTHESIS_PROFILE: extended never
reaches them — that profile is registered in test_fuzzed_templates.py alone. And pytest.yml
selects -m 'not expensive', which includes them, on every PR and every push to master, with the
toolchain installed and rust/target cached.

So the marker splits by requirement rather than by kind. wire says nothing about what those tests
do — they are fuzz tests — only about what selecting them commits a job to: having the pinned Rust
toolchain. A job with no Rust can now ask for fuzz and stay that way, which is what this one wants.
The sync-deps comment claiming the job "doesn't compile the Rust crates" stopped being true the
moment those tests carried fuzz; it now says that no toolchain gets installed, and points at the
step that depends on that.

Tests

The selector counts are the whole behavioural change:

-m 'fuzz'          → 36  test_fuzzed_templates.py   the nightly, Rust-free again
-m 'wire'          → 13  test_wire_roundtrip.py
-m 'not expensive' → 35  test_wire_roundtrip.py     pytest.yml, unchanged

pytest -m wire tests/test_wire_roundtrip.py: 13 passed. pytest -m "not expensive": 1036 passed,
11 skipped. pyright: 0 errors.

Nothing in PR CI exercises the nightly's selector — pytest.yml never runs -m fuzz — so the first
real proof is the next scheduled run.

🤖 Generated with Claude Code

@chandrakananandi
chandrakananandi self-requested a review August 19, 2026 16:22

@chandrakananandi chandrakananandi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

The nightly Integration Tests job failed in `run template fuzz tests` the
first night the wire round-trip suite existed. `-m 'fuzz' tests` selected
two unrelated suites: the 36 template tests the step is named for, and the
13 round trips in tests/test_wire_roundtrip.py, whose fixture builds
`wire-echo` with cargo at test time. The step runs `pytest -n 2` and that
fixture is session-scoped *per xdist worker*, so both workers asked rustup
to install the pinned 1.96 toolchain at the same moment. One rolled back
("detected conflict: 'bin/rust-gdb'") and the other found the wreckage:
"the 'cargo' binary ... is not applicable to the
'1.96-x86_64-unknown-linux-gnu' toolchain".

Installing the toolchain up front — pytest.yml already does, for exactly
this reason — would have fixed the crash but left this job paying an
uncached cargo build inside a 20-minute budget, for tests it gains nothing
by running: they pin their own max_examples, so this job's
HYPOTHESIS_PROFILE=extended never reaches them, and pytest.yml runs them at
the same depth on every push to master.

So split the marker instead. `wire` is not a statement about what those
tests do — they are fuzz tests — but about what selecting them commits a
job to: having the pinned Rust toolchain. A Rust-free job can now ask for
`fuzz` and stay Rust-free, which is what this one wants. pytest.yml selects
`-m 'not expensive'` and so keeps running all 35 tests in the file.

The `sync-deps` comment claimed the job "doesn't compile the Rust crates",
which stopped being true the moment those tests carried `fuzz`; it now says
that no toolchain gets installed and points at the step that depends on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ericeil ericeil changed the title CI: install the pinned Rust toolchain before the fuzz tests CI: give the wire round trips their own marker, out of fuzz Aug 19, 2026
@ericeil ericeil changed the title CI: give the wire round trips their own marker, out of fuzz ci: split the wire round trips out of the fuzz marker Aug 19, 2026

@chandrakananandi chandrakananandi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a better fix. Thanks!

@ericeil
ericeil merged commit 30ed3c1 into master Aug 19, 2026
4 checks passed
@ericeil
ericeil deleted the eric/fuzzFix branch August 19, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants