From 9538acd6ccfb5e65032cd9f7a205967caeb45a7b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Thu, 30 Jul 2026 10:03:25 +0200 Subject: [PATCH] docs: the four ways a CI gate can be unable to fail All four look fine on the Actions page and none can turn a merge red. Each has bitten this repo, three inside one week: 1. continue-on-error: true -- gc-stress, for months 2. not a required context -- gc-stress again; #6925's regression landed visibly red and survived three merges 3. unconditional cancel-in-progress -- gc-ratchet had three consecutive main runs cancelled, zero executed 4. the gate runs, its subject doesn't -- the dangerous one, because the job is genuinely green. PERRY_GC_FORCE_EVACUATE was inert for every gc()-driven test (#6942/#6946); the matrix --pressure knob disabled the path it measured (#7024); its moved= counter summed two collectors so a cell could pass having run zero copying minors (#7025) The rule that follows from (4): a gate must assert its subject was live, not merely that nothing threw. Corollary noted for (2): a new gate has never been green, so promoting it to required immediately blocks every open PR. Run it once, then promote -- leaving that second step undone is how (2) happens. --- CLAUDE.md | 11 +++++++++++ changelog.d/7047-gate-cannot-fail-patterns.md | 12 ++++++++++++ 2 files changed, 23 insertions(+) create mode 100644 changelog.d/7047-gate-cannot-fail-patterns.md diff --git a/CLAUDE.md b/CLAUDE.md index fd4f72690a..355e1e5b8d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -202,6 +202,17 @@ Build outputs are invisible to `git status`, so a clean tree tells you nothing a - **`conformance-smoke` shards are flaky.** Before believing a red shard, re-run it and A/B the named tests against a pristine `main` build; several are already in `test-parity/known_failures.json`. - **Integration suites under `crates/*/tests/*.rs` do not run per-PR** (nightly/tag only) — a regression there can land green and sit red for days. Prefer putting acceptance coverage in `cargo-test`-visible unit tests (#5960). +### ★ Four ways a gate can be unable to fail + +All four look fine on the Actions page. None can turn a merge red. When adding or reviewing a gate, check all four — each has bitten this repo, three of them within one week: + +1. **`continue-on-error: true`** — `gc-stress` carried it for months while being the only job covering GC correctness. +2. **Not in branch protection's required contexts** — `gc-stress` again. This is why #6925's `PERRY_PTR_SHAPE_LOCALS=0` regression landed visibly red and survived three merges. A job that reports failure without blocking is documentation, not a gate. +3. **`concurrency` with unconditional `cancel-in-progress`** — on a branch with a slow runner queue, every new merge cancels the previous run before it reaches a runner. `gc-ratchet` had three consecutive `main` runs cancelled, zero executed. Scope cancellation to `pull_request` and let `main` runs queue. +4. **The gate runs but its subject never did** — the most dangerous, because the job is genuinely green. `PERRY_GC_FORCE_EVACUATE` was inert for every `gc()`-driven test (#6942/#6946); the matrix's `--pressure` knob disabled the very path it was measuring (#7024); its `moved=` counter summed two different collectors, so a cell could pass having run zero copying minors (#7025). **A gate must assert its subject was live**, not merely that nothing threw — e.g. `copied_objects > 0` before a green verdict. + +Corollary: a *new* gate has never been green, so promoting it to required immediately blocks every open PR. Run it once, then promote. Leaving that second step undone is how (2) happens. + ### Known-weak areas (symptom is often not the bug) - **Async-to-generator transform, body locals.** It boxes every body local into a shared mutable cell typed `Any`. Two consequences seen in the wild: per-iteration `let`/`const` bindings collapse for closures created in a loop, and computed numeric-key calls (`arr[i](x)`) lose their type proof and silently resolve by *method name*, evaporating the call. - **Native base-class subclassing.** A native base's surface is installed at `super()` time and its parent edge lives in the class registry; keying any of that on a literal `extends` name loses it for fieldless classes, indirect subclasses, and class expressions. diff --git a/changelog.d/7047-gate-cannot-fail-patterns.md b/changelog.d/7047-gate-cannot-fail-patterns.md new file mode 100644 index 0000000000..053d8a5524 --- /dev/null +++ b/changelog.d/7047-gate-cannot-fail-patterns.md @@ -0,0 +1,12 @@ +Documented the four distinct ways a CI gate can be structurally unable to fail, +in `CLAUDE.md`. All four look green-adjacent on the Actions page and none can turn +a merge red: `continue-on-error: true`; absence from branch protection's required +contexts; `concurrency` with unconditional `cancel-in-progress` on a slow-queue +branch; and the subtlest — the gate runs and passes while its subject never +actually executed. + +Each has bitten this repo, three of them inside one week. The fourth is the +dangerous one because the job is genuinely green: `PERRY_GC_FORCE_EVACUATE` was +inert for every `gc()`-driven test, and the GC matrix's `--pressure` knob disabled +the very path it was measuring. The rule that follows is that a gate must assert +its subject was live, not merely that nothing threw.