ci: detect unreleased work and silently skipped publishes

[abrichr]

Why

Release and PyPI Publish run 30275153205 was cancelled 22 seconds in and nobody noticed. The same happened to 30226357239 the day before. report-release-failure cannot see either: it tests only for failure, and every needs.*.result of a cancelled run is cancelled.

The sibling failures are the same shape from the other end — openadapt-capture PR #51 removed a path that uploaded raw audio waveforms to a third party, merged to main, and sat unreleased while PyPI's only installable version still contained it; and four openadapt-desktop tags exist with no release object at all.

The design constraint that shaped this

This workflow runs on every push to main under concurrency: release with cancel-in-progress: false, so GitHub cancels the pending run whenever a third push arrives. Those cancellations are constant and usually harmless — a later run releases the same commits. An alert on "a release run was cancelled" would therefore be red most days and worth nothing.

This repository already has one guard that went that way: the platform-manifest drift check fired correctly on genuine drift, but had gone benignly red after every single release (semantic-release bumps ~90s before reconcile; PyPI CDN lag), so main was already red and the real signal was indistinguishable from noise.

So the detector is state-based. It asks whether anything is still unpublished, not whether a run was cancelled.

What this is not

Nothing here publishes, dispatches, or reruns anything. It prints the command a human would run. The deliberate-publication design across these repositories exists because Flow 1.13.0 and 1.14.0 were yanked for shipping AGPL benchmark files.

What it is

scripts/check_release_health.py (stdlib only — no dependency install, no lockfile, no cache) evaluates three detectors per release lane declared in .github/release-health.json:

detector	fires when
unreleased-work	main carries commits past the newest release tag that would bump the version under this repo's own [tool.semantic_release.commit_parser_options], main is green, no release run is in flight, and the oldest such commit is past the grace window
tag-without-release	a release-shaped tag has no release object, no run for it is queued/waiting/running, and it is past the grace window — reporting failure/cancelled/skipped, or that the workflow never started at all
unpublished-release	the newest release tag's version is absent from PyPI past the CDN grace window

False positives are the failure mode

--self-test enumerates every transient this repository actually produces and proves the detector stays quiet on each. It runs offline in CI whenever the detector itself changes.

A. It FIRES on the states it exists to catch
  [PASS] unreleased fix: commit, green main, past grace -> alert
  [PASS] `type!:` breaking change -> alert
  [PASS] `BREAKING CHANGE:` footer -> alert
  [PASS] tag with a cancelled/failed/never-started publish -> alert
  [PASS] conclusion `skipped` (invisible to `if: failure()`) -> alert
  [PASS] tag exists but PyPI never got the version -> alert
B. It stays QUIET on every transient this repository actually produces
  [PASS] semantic-release + reconcile commits are `chore` -> quiet
  [PASS] launcher's bare `1.10.0` release commit is not a bump -> quiet
  [PASS] docs/ci/chore(deps) merges -> quiet
  [PASS] a fix merged 1h ago is inside the 4h grace -> quiet
  [PASS] a release run is in flight -> quiet
  [PASS] main is red -> quiet (a red main is its own signal)
  [PASS] main CI still running -> quiet
  [PASS] PyPI CDN has not propagated yet (10m) -> quiet
  [PASS] PyPI unreachable -> warning, never an alert
  [PASS] tag awaiting a human `native-release` approval -> quiet
  [PASS] tag pushed 90m ago, installers still building -> quiet
  [PASS] tag whose artifact IS on PyPI -> warning, never an alert
  [PASS] explicitly acknowledged historical tag -> quiet
  [PASS] launcher concurrency cancellations that left no gap -> quiet (the exact runs 30275153205/30276970325)
C. Commit parsing honours THIS repository's configured tags
  [PASS] `perf:` bumps only when patch_tags says so
  [PASS] `feat(scope): ...` -> minor
  [PASS] Merge commits and free text -> no bump

The two sharpest transients here are the ~90s window and this repository's unusual release-commit format. reconcile-platform-manifest pushes chore(release): reconcile platform manifest after every release — chore is in allowed_tags but in neither minor_tags nor patch_tags, so no bump. And commit_message = "{version}\n\n..." means the release commit's subject is a bare 1.10.0, which is not a Conventional Commit at all. Both are in the fixtures, and the first one is live on main right now.

Proof it is quiet today

Run against live origin/main, PyPI and the GitHub API:

ok: [launcher] 2 commit(s) since v1.10.0, none of which trigger a version bump.
ok: [launcher] tag v0.36.1 has no release object; acknowledged: 2024-07-06 legacy
    monorepo tag. No release object was ever created and 0.36.1 was never published
    to PyPI (the index jumps 0.35.x -> 0.39.1). Permanently historical: acknowledged
    here rather than left to alert forever, because a guard that is always red is a
    guard nobody reads.
ok: [launcher] openadapt 1.10.0 is on PyPI.

No release-health alerts: every release lane is published and current.

Those "2 commits since v1.10.0" are exactly the transient: chore(release): reconcile platform manifest and ci: reduce launcher PR matrix cost (#1061).

Proof it fires when it should

The same detector, unchanged, run live against openadapt-desktop — it independently reproduced all four silently-skipped releases and their exact causes:

ALERT [unreleased-work/engine] 3 releasable commit(s) unpublished for 19h 25m
ALERT [tag-without-release/native] 4 tag(s) published no release

tag	age	release run	outcome
desktop-v0.1.0	11d 10h	29514474536	failure
desktop-v0.7.0	6d 23h	29756479649	cancelled
desktop-v0.8.0	6d 11h	29799291231	cancelled
desktop-v0.12.0	1d 21h	30169968004	failure

And replaying the exact openadapt-capture miss in the counterfactual where nobody looked:

ALERT [unreleased-work/python] 1 releasable commit(s) unpublished for 9h 47m
  27ccd262 | patch | fix(audio): make narration capture on-device only and fail closed (#51)
  No release workflow run has been recorded for this lane at all.
  The publish did not fail; it never started.

Triggers and cost

• schedule every 3h — the actual detector; with the 4h grace it bounds "how long can a releasable commit sit unnoticed" to ~7h.
• workflow_run on Release and PyPI Publish completed with a non-success conclusion — this is the trigger that sees cancelled and skipped.
• pull_request on the detector's own files — runs --self-test.

One stdlib step, no dependency install, no cache. Python pinned to 3.12 because openadapt-desktop run 29514474536 died on ModuleNotFoundError: No module named 'tomllib'; the script additionally degrades to documented defaults with a warning rather than crashing on an old interpreter.

Issue behaviour

Exactly one open issue per repository, rewritten in place on each run (editing a body does not notify, so a persistent gap is not a daily ping) and closed automatically with a comment once every gap is closed. Never duplicated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM

---

Second commit: chore(release): reconcile platform manifest with capture 1.2.1

Opening this PR immediately turned validate-platform-manifest red — and correctly so:

ERROR: capture version drift: manifest has '1.2.0' but PyPI's latest openadapt-capture
is '1.2.1'. The manifest advertises a superseded release and its digests.

That is not a transient and not caused by this PR. openadapt-capture published 1.2.1 at 15:17 UTC — the very release whose delay motivated the detector above — and reconcile-platform-manifest only runs after a launcher release, so a sibling package's release leaves platform-manifest.json stale until someone regenerates it. main's last manifest run (30277769578, 14:59 UTC) predates the capture publish and was green; the next run on main will fail exactly like this one did.

Regenerated with scripts/generate_platform_manifest.py; the only content change is the capture component's version, filenames, URLs and SHA-256 digests.

python scripts/validate_platform_manifest.py
OK: platform manifest validated (including published artifacts); 0 warning(s).