probe: run the ubuntu 4/4 shard against an unchanged integration/remote (do not merge) - #796
probe: run the ubuntu 4/4 shard against an unchanged integration/remote (do not merge)#796fujibee wants to merge 1 commit into
Conversation
Not for merge. A comment in a shell file, and nothing else. tests.yml runs on pull requests only, so a branch push cannot exercise the matrix, and the failing assertion in that shard does not reproduce on a macOS machine. The shard therefore cannot be observed against the base by any means available here except opening a pull request. A shell file rather than a doc: the workflow skips the suite on a docs-only diff and reports the shard green anyway, which is how a green with no measurement behind it got used as evidence tonight. Closed without merging once the shard has reported.
|
Result: The suite genuinely ran — this is the control that What this does and does not sayIt says: run on its own, this base passed that shard once. n=1. It does not say the base is healthy. A single green cannot say that about an assertion whose own issue (#769) calls it intermittent on CI, and three PRs based on this same commit were failing it within the hour. One sample settles nothing about a race. What it does remove is the reading that the shard is unconditionally red on Also worth recording
That is a separate defect and it is filed separately: a skipped suite and a passing suite are the same colour on the required check. Closed without merging
The next measurement is a real fix rather than another probe: |
1 participant
Declared reviewers: 0
Measurement only. Do not merge — this is closed as soon as the shard reports.
What this measures
Whether
bats (ubuntu-latest 4/4)fails onintegration/remotewith no change applied, which is the one thing nobody has been able to observe tonight.Three PRs (#775, #792, #794) fail that shard. On #792 the same SHA was re-run and failed again with the same assertion —
not ok 265 watch: relaunch with the SAME instance id replaces the previous watcher (#66 preserved)— so it is not a flake by that discriminator.fujibee/agmsg#769says the base reproduces it standalone, but that measurement is from 2026-08-13 and has not been repeated since.Why a pull request, and why not a doc
tests.ymlruns onpush: [main]andpull_request: [main, integration/remote]. A push tointegration/remotedoes not run the matrix — that branch's runs areverify-versionsonly. There is no branch history of this shard to read.d0b762dpasses it locally. So a local run cannot see this failure at all.#776'sbats (ubuntu-latest 4/4)showsconclusion=successwithRun bats suite (this shard)skipped. That green was used as evidence tonight, and it had measured nothing.Reading the result
Either answer settles it. Both are recorded on #769 afterwards, along with the re-run result and the fact that #776 was not a measurement.
Controls already taken
bats (ubuntu-latest 4/4)has run and passed on this lineage — #765 (1743dc8) and #766 (9b05e7a) both showstep 9executed withconclusion=success. So the shard is not uniformly red on ubuntu, which is why the current base has to be measured rather than assumed.