Give Distribution stale a first-class event, not just a projection flip - #695
Merged
Conversation
An APS storage array failure this week could only be recorded as a read- model side effect: the Attestation projection flips a Distribution row to Stale on a checksum mismatch, but that flip emits no Distribution-stream event, so marked_stale_at/marked_stale_by never populate and a replay of the event stream cannot reproduce the read model's own status. There was also no way to say "the bytes at this location are gone" when nothing probed a checksum at all, which is the actual shape of a storage failure. mark_distribution_stale adds a DistributionMarkedStale event, mirroring discard_distribution's slice shape but NOT its guards: discarding is a deliberate act CORA can refuse (redundancy + parent-Dataset checks); marking stale records a fact that already happened, so the only guard is structural (target exists, not already Discarded). Re-marking an already-Stale copy succeeds rather than raising. The projection gains a second, independent path to 'Stale' alongside the existing AttestationRecorded flip, guarded by the same status != 'Discarded' WHERE clause so neither path resurrects a Discarded row. The event carries `trigger`, resolving watch item W1 of the Distribution design memo, which deferred the TriggerSource shape to this slice. It names what concluded the copy is stale, which `reason` cannot: an operator's failure report and a sweep's re-probe both produce free text. The enum is Data-BC-local rather than an import of Supply's identical one, because W1 puts the hoist to cora.shared.triggers at the rule of three and this is use two. The decider stamps Operator rather than accepting it from the command, mirroring every operator-facing Supply decider, so a principal cannot assert its own report came from a monitor; a test pins both halves. It serializes bare and exports as drop:text, following Supply's five existing trigger entries rather than this aggregate's typed-enum access_protocol carve-out. Deviations from the memo's payload sketch, all deliberate and now recorded in the slice's own design memo: marked_stale_by rather than triggered_by (the sketch and L18 disagreed; the event matches the state), reason required rather than nullable, and the transition set widened to accept an already-Stale target. No context.py: no cross-aggregate lookup exists to carry. No migration: the fold only flips status. naming-r3-reviewer ran clean over the new symbols. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ly does The section claimed the `Marked` prefix discriminates operator observation from monitor measurement, and that a future automated monitor would emit bare past-participle events like `SupplyObservedAvailable`. Shipped code has contradicted that since the monitor slice landed: observe_supply_status emits SupplyMarkedUnavailable / SupplyMarkedRecovering with trigger=TriggerSource.MONITOR, and no SupplyObserved* class exists anywhere in the tree. The real discriminator is the trigger payload field. Left alone, the stale sentence is worse than merely wrong: a reviewer would cite it to reject a correct name. It nearly did. naming-r3-reviewer surfaced it while clearing DistributionMarkedStale, whose monitor-driven reconciliation sweep will reuse the same class exactly as Supply's does. The heading changes too, since the rule was never Supply-specific and is no longer operator-specific. No inbound links to the old anchor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stalewas reachable only in the read model. The Attestation projection flips a Distribution row to'Stale'on a checksum mismatch and emits no Distribution-stream event, somarked_stale_at/marked_stale_bynever populated and a replay could not reproduce the projection's own status. There was also no way to record "the bytes at this location are gone" when nothing probed a checksum, which is the actual shape of a storage failure.mark_distribution_stalegives the transition a first-class event.Why now
On 2026-08-14 an APS storage array failed and
/data2was lost outright, taking every 2-BM dataset that APS Data Management did not already hold. Nothing probed a checksum; the bytes simply ceased to exist. CORA had no way to say so.The design decision this is built around
Discarding is a deliberate act CORA is entitled to refuse, so
discard_distributionguards redundancy and the parent Dataset. Marking stale records a fact that already happened, so it does not. If the array holding the last Verified copy died, refusing to record it would make CORA assert something false. There is a test pinning exactly that.triggerResolves watch item W1 of the Distribution design memo, which deferred the
TriggerSourceshape to this slice. The enum is Data-BC-local rather than an import of Supply's identical one: W1 puts the hoist tocora.shared.triggersat the rule of three, and this is use two.The decider stamps
Operatorrather than accepting it from the command, mirroring every operator-facing Supply decider. A principal cannot assert its own report came from a monitor; a test pins both halves.Deviations from the memo's payload sketch
All deliberate, all now recorded in the slice's own design memo:
marked_stale_byrather thantriggered_by(the sketch and L18 disagreed; the event matches the state),reasonrequired rather than nullable, and the transition set widened to accept an already-Stale target.Second commit
naming-r3-reviewercleared the new symbols but surfaced a stale rule indocs/reference/conventions.md: it claimed a future monitor would emitSupplyObservedAvailable-style names, whenobserve_supply_statushas emittedSupplyMarked*withtrigger=MONITORsince it landed, and noSupplyObserved*class exists. Left alone it would be cited to reject a correct name. Corrected in its own commit.Verification
unit 13,664 · architecture 31,149 · integration 1,243 · contract 3,405 · ruff, pyright, tach clean. Re-run after rebasing onto current main.
No migration: the projection fold only flips
status.🤖 Generated with Claude Code