Let the run-liveness watchdog see Witnessed Runs, not just Conducted ones - #704
Merged
Conversation
…ones A TomoScan-driven scan wedged for 2.5 hours on 2026-08-20 unseen by RunSupervisor, because the conduct_mode filter excluded every Witnessed Run before the liveness rule ever ran. The command-issuing rungs (hold, resume, Rule Q/R acts) rightly stay Conducted-only: CORA has no authority to hold, resume, abort, or stop a Run an external tool drives. But shadow-flagging and advising a human are just observation, and observation is exactly what a Witnessed Run needs when it hangs. Widen the liveness pass to a second list, liveness_candidates, that carries every Running Run regardless of conduct_mode, and rebuild inflight_ids from it so the liveness edge-trigger set still garbage collects correctly. The truncate act stays Conducted-only: a stale Witnessed Run's terminal already belongs to RunWitnessRecorder, which truncates it on the next BEGUN observation, so the supervisor re-deciding that terminal would just be overriding another authority's verdict on a Run it does not drive. Also documents a 2-BM deployment fact next to Rule R: TomoScan cycles the station shutter for dark/flat fields many times per scan, and that same PV feeds beam_open, so Rule R defers through long stretches of every healthy scan. Not a behavior change, just naming the limitation where the next reader will look for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Gate review (2026-08-21, baseline panel) fixes folded in: the Witnessed resume test was decorative, passing because the unpatched envelope assembly failed on the fixture's missing Plan rather than because the conduct_mode gate held. It now mirrors its Conducted sibling exactly (envelope patched good, settle window passed) so only conduct_mode differs. Added a mixed-tick test proving both the wide scope and the act gate are per-item, not all-or-nothing, and an advise-off assertion on the shadow path. Every guard in this commit is mutation-verified: deleting it turns a named test red.
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap a 2026-08-20 incident at APS 2-BM exposed: a TomoScan scan wedged for two and a half hours and the RunSupervisor never saw it.
Mechanism.
_supervise_tickdropped everyconduct_mode != "Conducted"Run in one filter, before any rule ran. 2-BM Runs are Witnessed, so the run-liveness rule, and Rules Q and R, were structurally dead there. The liveness rule is the right detector for a wedge (it is beam-free and keys onrunning_since, which every Witnessed Run has); it simply never got a Run to look at.That one filter is now two scopes.
liveness_candidatescarries every Running Run and feeds only the liveness pass's SHADOW and ADVISE rungs.running/heldstay Conducted-only and feed every command rung: the hold FSM, the gated resume, Rule Q's abort act, Rule R's stop act.inflight_idswidens to match so thelivenessedge-trigger entries are still garbage-collected.Why the truncate act stays Conducted-only.
RunWitnessRecorderalready truncates a stale Witnessed Run on the nextBEGUNfor the same capture code. Re-deciding a terminal another authority owns is whatproject_conjunct_symmetry_designrules out. Worth recording the limit of that argument, which the gate review surfaced: that terminal only fires if a nextBEGUNever arrives. If TomoScan never restarts the code, a wedged Witnessed Run gets no automatic terminal from either authority and the advise Decision is the only signal a human receives. That is the condition under which to revisit the act rung.Scope. One source file and its tests. No new Decision choice (
SupervisionQuietedalready exists), no port, adapter, schema, or setting change. Rules Q and R are deliberately not extended to Witnessed Runs: a Witnessed Run has noexpected_observation_interval_seconds, so Rule R is disabled for one anyway.Also adds a deployment note to the Rule R section recording that 2-BM's station shutter doubles as its fast shutter and is cycled many times per scan, so
beam_openreads False through long stretches of every healthy scan and Rule R defers. Documented limitation, not a behaviour change.Gate review. Baseline panel (architecture, test coverage, cross-BC consistency). Cross-BC found nothing: no projection joins a Decision back to a Run's
conduct_mode, and the record export dropscontextandinputsforDecisionRegisteredunconditionally. Architecture found no P0/P1. Test coverage found one P0, a decorative Witnessed-resume test that passed because the unpatched envelope assembly failed on the fixture's missing Plan rather than because the gate held; it is fixed and now mirrors its Conducted sibling with onlyconduct_modediffering. Every guard in this PR is mutation-verified: deleting it turns a named test red.Testing. 114 supervisor unit tests and 31,288 architecture fitness tests pass.