Skip to content

Make the capture-path vault's upsert monotonic in observed_at - #703

Merged
xmap merged 1 commit into
mainfrom
fix/monotonic-capture-path-upsert
Aug 21, 2026
Merged

Make the capture-path vault's upsert monotonic in observed_at#703
xmap merged 1 commit into
mainfrom
fix/monotonic-capture-path-upsert

Conversation

@xmap

@xmap xmap commented Aug 21, 2026

Copy link
Copy Markdown
Owner

The defect

observed_at on a run_capture_path row is the EPICS PV's own timestamp, not CORA's clock. It reaches the vault straight from RunWitnessRecorder._resolve_capture_path, so a monitor reconnect replaying a stale value, or an IOC whose clock stepped, can deliver an older reading after a newer one.

The upsert overwrote unconditionally, so it believed the stale one.

Why it matters now

It used to be close to harmless. Since #699 a Run can hold one row per storage tier, and _LATEST_SQL orders on observed_at DESC, so a single backwards write no longer corrupts just one row's path. It can make load_run_capture_path resolve to the wrong tier entirely:

   gdata copy    observed at t=5
   local1 copy   observed at t=10   <- newest, get_latest returns this

   stale replay against local1, at t=1
                       |
        before         v
   local1 drops to t=1, now older than gdata's t=5
   get_latest returns the GDATA row

The fix

A WHERE on the whole DO UPDATE, not a GREATEST in the SET. The row moves forward as a unit or not at all, so it can never end up carrying a newer timestamp paired with an older path. The in-memory adapter mirrors the same >= rule, so a declined write is invisible in both.

>= rather than > so a genuine retry of the same observation, carrying a corrected path with no new reading in between, still lands.

A declined write affects zero rows in Postgres. That is correct and the caller ignores the count, which the Protocol docstring now says explicitly so nobody later reads it as a failure.

No migration: this is a statement change.

Tests

Four scenarios, in both the Postgres and in-memory adapters:

  • an older observed_at leaves the row untouched, asserted through updated_at == created_at so it proves nothing was written rather than just that the path survived
  • a newer one still updates, pinning the existing behaviour so the guard cannot over-fire
  • an equal one still updates
  • the tier-level regression drawn above, which fails on the old code by returning the wrong location

observed_at is TIMESTAMPTZ NOT NULL, so the guard cannot silently never-fire on a null comparison.

Written by Claude Sonnet 5, reviewed before submission.

🤖 Generated with Claude Code

observed_at is the EPICS PV's own timestamp, not CORA's clock, so a
monitor reconnect or an IOC clock step can replay an older reading
after a newer one. The old unconditional overwrite let that stale
write clobber a newer observation, and since a Run can now hold one
row per storage tier with get_latest ordering on observed_at DESC,
a single backwards write could make get_latest resolve to the wrong
location entirely.

Guard the row-level UPDATE with a WHERE on the whole SET so a row
only ever moves forward as a unit (never a newer timestamp paired
with an older path), and mirror the same rule in the in-memory
adapter so a declined write is invisible in both.
@github-actions

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  apps/api/src/cora/run/aggregates/run
  capture_path.py
Project Total  

This report was generated by python-coverage-comment-action

@xmap
xmap merged commit accfda0 into main Aug 21, 2026
19 checks passed
@xmap
xmap deleted the fix/monotonic-capture-path-upsert branch August 21, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant