fix(benchmark): check the MockMed encounter type as its own field

README.md

@@ -492,10 +492,13 @@ a note) with a distinct note value each run and the same OCR success
 check on both arms: 20 compiled replays against 10 runs of a
 claude-sonnet-5 computer-use agent, measured on 2026-07-08 from a
 pre-v0.2.0 source checkout declaring openadapt-flow 0.1.0. Compiled went
-20/20 at 39.2s (p50)
+19/20 at 39.2s (p50)
 with zero model calls; the agent went 10/10 at 70.4s (p50), about $0.55
 per run at list price ($5.52 total for the 10 runs, with prompt caching
-and hard cost caps enforced in the harness). It is a shared public demo
+and hard cost caps enforced in the harness). The corrected OCR check requires
+the note in a saved Patient Messages row; it rejects one compiled run where the
+note remained in the unsaved entry form. This is screen-row evidence, not a
+system-of-record read. It is a shared public demo
 that other users mutate and that resets daily, so it is not CI-reproducible,
 and the sample is small. Correctness alone (no agent arm, 5/5 fresh browsers,
 zero model calls, closed-loop scrolling) is in

benchmark/comparison_artifact/README.md

@@ -2,8 +2,8 @@
 
 A self-contained, theme-aware HTML page that packages the core wedge in one
 place: for a task you have **already demonstrated**, a compiled replay is
-**model-free**, **~$0 per run**, and **faster** than a computer-use agent — at
-**parity** success on these tasks.
+**model-free**, **~$0 per run**, and **faster** than a computer-use agent. The
+artifact reports the measured success of each arm; it does not assume parity.
 
 It is **generated from the repo's real benchmark results**, not hand-typed. The
 generator reads the two existing `results.json` files and lays their figures out
@@ -34,7 +34,7 @@ Leads with the **real third-party result** (OpenEMR public demo, an 18-step
 add-patient-note workflow on a live EMR), then the **CI-reproducible anchor**
 (MockMed, the bundled demo clinic). For each benchmark:
 
-- **Success parity** — both arms pass the same arm-independent OCR check.
+- **Measured success** — both arms use the same arm-independent OCR contract.
 - **Measured model API cost** — `$0` for compiled vs `$/run` for the agent, as
   an inline-SVG bar chart plus an explicitly limited arithmetic projection.
 - **Latency** — p50/p95 wall-clock for both arms as grouped inline-SVG bars.
@@ -45,9 +45,8 @@ labels) — no screenshots, no external assets, no base64 needed.
 It ends with an **honest "Read before quoting these numbers"** panel (mirroring
 the gallery's "What still slips" tone): small N and wide error bars; the lead is
 a field result on a shared, daily-resetting public demo (not CI-reproducible);
-list-price costs with hard caps; the single OCR success check that errs
-conservative on both arms; and the fact that this measures cost/latency at
-**parity** success — **not** a general capability claim.
+list-price costs with hard caps; the corrected saved-row OCR contract; and the
+fact that this is **not** a general capability claim.
 
 ## Regenerate
 

benchmark/comparison_artifact/comparison.json

@@ -23,8 +23,8 @@
       "arms": {
         "compiled": {
           "n": 20,
-          "success_count": 20,
-          "success_rate": 1.0,
+          "success_count": 19,
+          "success_rate": 0.95,
           "wall_s_p50": 39.170917561999886,
           "wall_s_p95": 41.04867044315233,
           "cost_usd_per_run": 0.0,
@@ -49,8 +49,31 @@
         "pricing_note": "list price; an introductory $2/$10 rate applies through 2026-08-31",
         "compiled_self_flag": {
           "i": 19,
-          "success": true,
+          "success": false,
           "first_failure_step": "step_017"
+        },
+        "success_contract": {
+          "kind": "screen_saved_message_row",
+          "note_min_contiguous_squashed_chars": 16,
+          "requires": [
+            "Patient Messages Content and Status headers",
+            "note fragment in the Content column below the headers",
+            "an aligned New status for the same saved row"
+          ],
+          "rejects": "note text visible only in the unsaved entry form"
+        },
+        "oracle_adjudication": {
+          "corrected_at": "2026-07-28",
+          "contract": "The final screenshot must show a contiguous note fragment of at least 16 squashed characters inside a saved Patient Messages Content row aligned with its New status. Note text in the unsaved entry form is not saved-row evidence.",
+          "retained_final_path": "benchmark/openemr/finals/compiled_019.png",
+          "retained_final_sha256": "8c504ba15bab9cdca8b5987dd1d1ab7b0ba7ae77f67fac5e93ba8481492ae18f",
+          "old_behavior": "The legacy verifier accepted the requested note anywhere in full-frame OCR and returned success for compiled run 20.",
+          "new_behavior": "The saved-row verifier rejects compiled run 20 because the note remains in the open textarea and the Save as new message button remains visible; no saved row contains the note.",
+          "changed_runs": [
+            "compiled:19"
+          ],
+          "denominators_changed": false,
+          "latency_aggregates_changed": false
         }
       }
     },

benchmark/comparison_artifact/generate.py

@@ -7,8 +7,9 @@
 
 - ``benchmark/openemr/results.json`` — the LEAD result: the same task run
   against a real third-party application (the official OpenEMR public demo),
-  20 compiled replays vs 10 ``claude-sonnet-5`` computer-use agent runs, both
-  100%. This is a field result, not CI-reproducible (shared public instance).
+  20 compiled replays vs 10 ``claude-sonnet-5`` computer-use agent runs. The
+  corrected saved-row oracle counts 19/20 and 10/10. This is a field result,
+  not CI-reproducible (shared public instance).
 - ``benchmark/results.json`` — the CI-reproducible MockMed anchor: 100 compiled
   vs 20 agent on the bundled demo clinic, both 100%, same orchestrator and same
   arm-independent OCR success check.
@@ -32,8 +33,8 @@
   their provenance), so the page is verifiable without eyeballing it.
 
 The page leads with the honest wedge: compiled replay is model-free, ~$0/run,
-and faster, at *parity* success on these tasks — with the small-n, shared-demo,
-and not-a-capability-claim caveats stated up front, not buried.
+and faster on these trials. It reports each arm's measured success and states
+the small-n, shared-demo, and not-a-capability-claim caveats up front.
 """
 
 from __future__ import annotations
@@ -139,8 +140,8 @@ def load_openemr(path: Path = OPENEMR_RESULTS) -> Benchmark:
     model = str(d.get("model", "claude-sonnet-5"))
     compiled = _arm_from_json("compiled", arms["compiled"], "0 model calls")
     agent = _arm_from_json("agent", arms["agent"], model)
-    # A single compiled run self-flagged expected-screen drift and aborted; the
-    # arm-independent OCR check confirmed the write landed. Report it honestly.
+    # A single compiled run self-flagged expected-screen drift and aborted. The
+    # corrected saved-row OCR check also rejects that run. Report both signals.
     self_flag = None
     for run in d.get("runs", {}).get("compiled", []):
         if run.get("replayer_success") is False:
@@ -166,6 +167,8 @@ def load_openemr(path: Path = OPENEMR_RESULTS) -> Benchmark:
             "cost_caps_usd": d.get("cost_caps_usd"),
             "pricing_note": (d.get("pricing_usd_per_mtok") or {}).get("note"),
             "compiled_self_flag": self_flag,
+            "success_contract": d.get("success_contract"),
+            "oracle_adjudication": d.get("oracle_adjudication"),
         },
     )
 
@@ -425,10 +428,10 @@ def _stat_tiles(b: Benchmark) -> str:
     speed = _speedup(b)
     tiles = [
         (
-            "success — parity",
+            "measured success",
             f"{b.compiled.success_count}/{b.compiled.n} · "
             f"{b.agent.success_count}/{b.agent.n}",
-            "compiled · agent — both pass the same arm-independent OCR check",
+            "compiled · agent — same arm-independent screen contract",
             "ok",
         ),
         (
@@ -484,14 +487,21 @@ def _self_flag_note(b: Benchmark) -> str:
     sf = b.extras.get("compiled_self_flag")
     if not sf:
         return ""
+    run_number = int(sf.get("i", -1)) + 1
+    if sf.get("success") is False:
+        return (
+            '<p class="honest-inline">One compiled run (#{i}) self-flagged '
+            'expected-screen drift at <code>{step}</code> and aborted. The '
+            'corrected saved-row OCR check also rejects it: the retained final '
+            'frame shows the note in the unsaved entry form, not in a saved '
+            'message row. It counts as a failure.</p>'
+        ).format(i=_e(run_number), step=_e(sf.get("first_failure_step")))
     return (
         '<p class="honest-inline">One compiled run (#{i}) self-flagged '
-        'expected-screen drift at <code>{step}</code> and aborted — yet the '
-        'arm-independent OCR check confirmed the note saved, so it counts as a '
-        'success. On a shared instance the message list grows under every '
-        "visitor, so a postcondition can drift <em>after</em> the write lands; "
-        'the self-flag halting instead of improvising is the point.</p>'
-    ).format(i=_e(sf.get("i")), step=_e(sf.get("first_failure_step")))
+        'expected-screen drift at <code>{step}</code> and aborted, while the '
+        'saved-row OCR check accepted its final frame. It counts according to '
+        'that shared screen contract.</p>'
+    ).format(i=_e(run_number), step=_e(sf.get("first_failure_step")))
 
 
 def _drift_note(b: Benchmark) -> str:
@@ -594,18 +604,17 @@ def _caveats(oe: Benchmark, mm: Benchmark) -> list[tuple[str, str]]:
             "in the source data rather than hidden.",
         ),
         (
-            "Success is one OCR check on both arms — and it errs conservative.",
-            "A run passes only if the run's own note text is read back out of the "
-            "final screenshot by OCR, identically for compiled and agent. On dense "
-            "EMR text the OCR sometimes drops the exact line it is looking for, so "
-            "a 'failed' verification can be a measurement miss with the note "
-            "plainly on screen. The check is identical for both arms, so it cannot "
-            "favour one — but it under-counts rather than over-counts success.",
+            "Success is corrected saved-row screen evidence, not a record read.",
+            "The legacy whole-frame OCR check accepted one compiled run whose note "
+            "was still in the unsaved entry form. The corrected contract requires "
+            "the note in a saved Patient Messages row, and replay of all 30 retained "
+            "final frames changes only that run. OCR can still miss dense text. This "
+            "field result does not read the OpenEMR system of record out of band.",
         ),
         (
-            "This measures cost and latency at PARITY success — not capability.",
-            "Both arms passed every task here, so the story is purely how much "
-            "cheaper and faster the compiled path is at the same outcome. It is "
+            "This measures bounded outcomes, cost, and latency — not capability.",
+            "The OpenEMR outcomes are 19/20 and 10/10 under one corrected screen "
+            "contract. The result shows the measured cost and latency difference. It is "
             "NOT a claim that compiled replay is more capable, more robust to novel "
             "situations, or a substitute for an agent on a task it has never seen. "
             "MockMed is also a deliberately simple app; harder surfaces would slow "
@@ -617,7 +626,7 @@ def _caveats(oe: Benchmark, mm: Benchmark) -> list[tuple[str, str]]:
 def render_html(oe: Benchmark, mm: Benchmark) -> str:
     speed_oe = _speedup(oe)
     headline = (
-        f"Same task, same success ({oe.compiled.success_count}/{oe.compiled.n} "
+        f"Same task, measured success ({oe.compiled.success_count}/{oe.compiled.n} "
         f"vs {oe.agent.success_count}/{oe.agent.n}) &mdash; compiled replay costs "
         f"{fmt_usd(oe.compiled.cost_per_run)} and runs {speed_oe:.1f}&times; faster "
         f"than the {oe.model} agent"
@@ -642,7 +651,8 @@ def render_html(oe: Benchmark, mm: Benchmark) -> str:
     <p class="sub">One clinical task, two ways to automate it, one success check.
       The wedge in one line: for a task you have already demonstrated, a compiled
       replay is <strong>model-free</strong>, <strong>~$0 per run</strong>, and
-      <strong>faster</strong> &mdash; at <strong>parity</strong> success. Generated
+      <strong>faster</strong> on these measured trials. Each arm's success is
+      reported separately. Generated
       straight from the repo's real benchmark <code>results.json</code> files; no
       number here is hand-typed.</p>
     <div class="headline ok">{headline}</div>
@@ -671,8 +681,8 @@ def render_html(oe: Benchmark, mm: Benchmark) -> str:
     <ul>
       {caveats}
     </ul>
-    <p class="foot">Bottom line: at parity success on these tasks, the compiled path
-      removes the model from the loop &mdash; {fmt_usd(oe.compiled.cost_per_run)} and
+    <p class="foot">Bottom line: on these measured trials, the compiled path removes
+      the model from the loop &mdash; {fmt_usd(oe.compiled.cost_per_run)} and
       {speed_oe:.1f}&times; faster on the real EMR, and a tighter, reproducible
       version of the same gap on MockMed. That is a cost/latency result on known
       tasks, disclosed with its limits &mdash; not a general capability claim.</p>

benchmark/openemr/BENCHMARK.md

@@ -17,7 +17,7 @@ BOTH arms), save.
 | | compiled replay | computer-use agent |
 |---|---|---|
 | runs | 20 | 10 |
-| success rate | 100% (20/20) | 100% (10/10) |
+| success rate | 95% (19/20) | 100% (10/10) |
 | latency p50 | 39.2 s | 70.4 s |
 | latency p95 | 41.0 s | 82.6 s |
 | model cost / run | $0 | $0.5522 |
@@ -35,13 +35,7 @@ Failed runs, reported honestly:
 
 Compiled arm:
 
-- none
-
-Compiled runs that self-flagged, also reported honestly (success is judged by the arm-independent OCR check both arms share, not the replayer's self-report):
-
-- compiled run 20: postconditions flagged expected-screen drift at step_017 and the replayer aborted; the arm-independent OCR check verified the note saved (matched 100%)
-
-On a shared instance every run — ours and other visitors' — grows the message list, so a postcondition can drift after the action lands. The self-flag is the point: the replayer detects the drift and halts instead of improvising.
+- compiled run 20: saved-row oracle failed; replayer halted at step_017 — The legacy whole-frame OCR verifier matched the note in the unsaved entry textarea. The retained final frame has no saved row for this note.
 
 Agent arm:
 
@@ -76,15 +70,17 @@ below.
   target patient, the exact note text — not steps or coordinates. Every
   executed action returns a settled screenshot.
 - **Same success criterion, implemented once.** After each run, the final
-  screenshot is checked by `verify_note_saved` (OCR): a contiguous run of
-  at least 16 characters of the run's note must appear in the frame's
-  OCR text (whitespace-squashed; retried at 2x resolution when the raw
-  frame does not pass, because rapidocr drops dense table lines at
-  1280x800). Neither arm's self-reported success is used.
+  screenshot is checked by `verify_note_saved` (OCR). A contiguous run of
+  at least 16 characters of the run's note must appear in a saved Patient
+  Messages `Content` row aligned with its `New` status. The same note in the
+  unsaved entry form does not pass. OCR is whitespace-squashed and retried at
+  2x resolution when the raw frame does not pass. Neither arm's self-reported
+  success is used. This is screen-row evidence, not an out-of-band read of the
+  OpenEMR system of record.
 - **Distinct, mutually dissimilar note per run in BOTH arms** (no two
   notes share a 16-character squashed substring — unit-tested), so
-  success proves parameter substitution against live state and one run's
-  note cannot satisfy another run's check.
+  a saved-row success proves parameter substitution in the visible message
+  list and one run's note cannot satisfy another run's check.
 - **Pacing.** Runs are spaced ~30s apart as
   public-demo courtesy; the pacing gap is excluded from latency.
 - **Latency** is wall-clock around the replay / agent loop only.
@@ -133,15 +129,13 @@ below.
 - **The compiled arm needs a demonstration first.** The one-time
   record + compile step (about a minute of human demonstration) is the
   price of the fast replays; the agent needs only the prompt.
-- **OCR verification on dense EMR text under-counts.** rapidocr sometimes
-  drops the exact table line containing the note (a known limitation
-  documented in
-  [docs/showcase-openemr/FINDINGS.md](../../docs/showcase-openemr/FINDINGS.md)),
-  so a "failed" verification can be a measurement miss with the note
-  plainly visible in the final screenshot. The check errs conservative
-  and is identical for both arms. Every run's final screenshot is saved
-  to `benchmark/openemr/finals/` (local only, not committed) so failed
-  verdicts can be audited against what was actually on screen.
+- **The success oracle is bounded screen evidence.** rapidocr can drop dense
+  table lines and cause an over-halt. The legacy whole-frame check could also
+  accept note text in the unsaved form; retained compiled run 20 exposed that
+  false success. The corrected check requires saved-row context. All 30 final
+  frames were replayed under it: only compiled run 20 changed. Every final
+  screenshot stays in `benchmark/openemr/finals/` (local only, not committed)
+  for audit. This check does not read the OpenEMR system of record.
 - Single machine (macOS-15.7.3-arm64-arm-64bit).
 
 ## Reproduce