Skip to content

fix: stop fabricating display metrics, window data, and UIA evidence - #61

Merged
abrichr merged 1 commit into
mainfrom
sweep/false-success-capture
Jul 28, 2026
Merged

fix: stop fabricating display metrics, window data, and UIA evidence#61
abrichr merged 1 commit into
mainfrom
sweep/false-success-capture

Conversation

@abrichr

@abrichr abrichr commented Jul 28, 2026

Copy link
Copy Markdown
Member

Five paths in this package returned a plausible value when the measurement or observation could not be taken at all. Each one is written into the recording and is indistinguishable from a real one.

1. get_display_pixel_ratio() returned 1.0 from every failure branch (highest blast radius)

platform/__init__.py:107, platform/darwin.py, platform/windows.py, platform/linux.py — all four ended return 1.0.

1.0 is a real, common answer. A Retina display whose probe failed — no Screen Recording permission, mss.mss() failing in a headless or threaded context, a missing shcore/GetDpiForMonitor on Windows — was recorded as a standard-density display. recorder.py:1313 writes that number onto the Recording; capture.py:540 and visualize/html.py:60 read it back. Every captured coordinate is then rescaled by half, and nothing about the recording looks wrong.

The repo already knows this harm. From CHANGELOG.md:

CaptureSession.pixel_ratio read pixel_ratio off the SQLAlchemy Recording model, but that model had no such column ... so a HiDPI capture whose config JSON lacked it silently scaled at 1.0, under-scaling downstream coordinate mapping.

That change fixed the persistence side and added a nullable column so unknown would be representable — while the measurement side kept fabricating 1.0, so the nullable column could never actually be NULL.

Now: the probe raises DisplayMetricsUnavailable, and create_recording catches it, logs at error level, and persists None.

2. get_screen_dimensions() returned the literal (1920, 1080)

Same shape, in all four places. Now raises DisplayMetricsUnavailable.

3. is_accessibility_enabled() returned True meaning "I did not check"

The Windows implementation is the sharpest example — the check's result was computed and thrown away, and every branch returned True:

try:
    ctypes.windll.shell32.IsUserAnAdmin()
    # Even non-admin can typically capture input
    return True
except Exception:
    return True  # Assume enabled
except Exception:
    return True

A consumer gating "safe to start recording" on this got a green light and then recorded zero input events. Now a tri-state: True verified, False verified, None undetermined, with the docstring stating that None is not a synonym for True.

4. UIA emitted a successful observation for an element it could not read

structural_observer/windows.py:_observe_with_runtime called _as_element(target) unconditionally. When element_info was unreadable — COM failure, element vanished — _element_fields returned {} and the function still returned a fully-formed StructuralObservation(provider="windows_uia", ...) with every identity field None. recorder.py:877 attaches it to the ActionEvent, and downstream compile reads the presence of a structural observation as accessibility ground truth.

The module's own sibling policy is the opposite: _present_string explicitly refuses to emit partial identity ("Never retain a partial identity"), and _candidate_cardinality returns None for unknown. Now _observe_with_runtime returns None when the element could not be read.

5. The window layer could not say "no backend"

  • get_active_window_data() returned {} while its annotation said dict | None and its docstring said "or None if the state is not available".
  • A backend that failed to import left impl = None process-wide behind a logger.warning.
  • read_window_events then polled a backend that could never answer, hit if not window_data: sleep(0.1); continue forever, and never set its readiness event — so _wait_for_tasks_started spun indefinitely (the thread is alive, so stopped_before_ready never fires) and recording startup hung with no stated cause.
  • window/_linux.get_active_element_state returned a fabricated {"x": .., "y": .., "state": "placeholder"}, persisted verbatim onto every ActionEvent as if it were captured accessibility data.

Adds WindowCaptureUnavailable and require_impl(), called at the top of read_window_events; returns None per the documented contract; routes the window reader through _run_task_fail_loud (as the input reader already is) so the failure reaches task_errors with its real exception; and _linux.get_active_element_state now returns None.

Mutation checks

Every test in tests/test_no_fabricated_capture.py was verified against pre-fix code by reverting each production change individually:

Reverted Failing test Failure
platform/__init__.py defaults test_pixel_ratio_raises_instead_of_returning_one Failed: DID NOT RAISE DisplayMetricsUnavailable
platform/__init__.py defaults test_screen_dimensions_raise_instead_of_returning_1920x1080 Failed: DID NOT RAISE DisplayMetricsUnavailable
platform/__init__.py defaults test_accessibility_state_is_none_when_undetermined assert True is None
create_recording -> pixel_ratio = 1.0 test_recording_stores_null_pixel_ratio_when_unmeasurable AssertionError: assert 1.0 is None
require_impl() call removed from reader test_window_reader_refuses_instead_of_polling_forever Failed: Timeout (>15.0s) from pytest-timeout — the original hang, reproduced
window/__init__.py ({} return, no require_impl) test_require_impl_raises_when_no_backend_loaded Failed: DID NOT RAISE WindowCaptureUnavailable
window/__init__.py ({} return) test_active_window_data_is_none_not_empty_dict assert {} is None
_observe_with_runtime element ordering test_unreadable_element_yields_no_observation assert StructuralObservation(... element=StructuralElement(automation_id=None, role=None, ... supported_patterns=[]), process=None, window=None, ancestry=None, candidate_count=None ...) is None
_linux.get_active_element_state placeholder test_element_state_is_none_not_a_placeholder_dict assert {'x': 10, 'y': 20, 'state': 'placeholder'} is None

The Linux test stubs xcffib so it runs on the whole PR matrix rather than skipping everywhere but Linux.

Verification

uv run ruff check openadapt_capture/ clean; uv run pytest tests/ --ignore=tests/test_browser_bridge.py --timeout=120 -> 367 passed, 7 skipped.

Candidates examined and dismissed

  • video.py FFmpeg boundary — exemplary, no change. _run_checked inspects returncode and raises; FFmpegVideoWriter.close() raises "FFmpeg returned success without a non-empty output" when the file is missing or zero-length; _abort_and_reap's except subprocess.TimeoutExpired: pass follows an explicit process.kill() and is a reap, not a swallow.
  • audio.py except ImportError: pass in _get_best_transcription_backend — deliberate and documented: it returns None, and require_local_transcription_backend() converts that to a NoLocalTranscriptionBackend refusal so a recording that cannot be transcribed locally is never started. The docstring states the reason (a hosted fallback would upload a raw waveform).
  • input_observer/base.py, factory.py, linux.py, windows.py — clean. Queue overflow, consumer failure, delivery-thread death and shutdown timeouts all raise; unsupported platforms raise InputObserverUnavailableError; every Linux degradation is named in the module docstring.
  • window_capture.py resolve/capture/translate — deliberately fail-loud with WindowCaptureError.
  • _supported_patterns returning a possibly-short list — pywinauto signals "pattern not supported" by raising, so the broad catch is the idiomatic probe. Left alone; noted below.
  • platform/linux.get_active_window_info degrading title/geometry/pid to ""/0 — a fabricated-success shape, but it has no in-repo callers and no consumer outside this tree. Reported, not fixed.
  • window_capture.py:211 scale = ... if bounds_w else 1.0 on zero-width bounds, and :464 dropping a matching window whose rect cannot be read — plausible but unconfirmed reachable; both are Win32/macOS-only paths that usually fail earlier. Reported, not fixed, to keep this change reviewable.
  • input_observer/darwin.py:552 _is_injected returning False when kCGEventSourceUnixProcessID is absent — requires a PyObjC build missing a long-stable constant. Theoretical.

Fake-gate inventory

.github/workflows/ contains no || true, continue-on-error, exit 0, set +e, or swallowed step across all seven workflows. lint runs ruff check openadapt_capture/ as a real gate, and package-contract runs verify_distribution.py on the built archives.

Two observations, both deliberate rather than fake:

  • test-windows and test-macos carry if: github.event_name != 'pull_request', so on a PR only the Linux 3.10–3.12 matrix, lint, and package-contract run. The workflow comments state this is a cost decision and that cross-platform qualification runs on the exact main commit. Worth knowing when reading a green PR: the Windows UIA and macOS paths in this change are covered by these tests on the post-merge main run, not on the PR.
  • release.yml is workflow_dispatch only, so merging this fix: will not cut a release. Publication stays a deliberate operation gated on the exact main commit having completed test.yml.

Five paths returned a plausible value when the measurement or observation
could not be taken at all, so the recording reported success while carrying
invented data.

1. `platform.get_display_pixel_ratio()` returned 1.0 from every failure
   branch on all three platforms. 1.0 is a real, common answer, so a Retina
   display whose probe failed (no Screen Recording permission, mss failing
   in a headless or threaded context, a missing shcore/GetDpiForMonitor) was
   recorded as a standard-density display and every captured coordinate was
   rescaled by half. The CHANGELOG already describes this exact harm and
   fixed only the persistence side. The probe now raises
   `DisplayMetricsUnavailable`, and `create_recording` persists NULL - the
   column is nullable precisely so that unknown is representable.

2. `platform.get_screen_dimensions()` returned the literal (1920, 1080) when
   every measurement path failed. It now raises.

3. `platform.is_accessibility_enabled()` returned True from every
   "could not check" branch. The Windows implementation called
   `IsUserAnAdmin()`, discarded the result, and returned True from the try,
   the except, and the outer except, so the answer carried no information at
   all. It is now a tri-state: True verified, False verified, None
   undetermined.

4. `structural_observer/windows._observe_with_runtime` emitted a complete
   `StructuralObservation` with an identity-free element when the element's
   `element_info` could not be read. That is positive evidence that
   windows_uia observed the action, indistinguishable from an element that
   legitimately exposes nothing, and downstream compile reads the presence of
   a structural observation as accessibility ground truth. It now returns
   None.

5. `window.get_active_window_data()` returned `{}` despite its annotation and
   docstring both promising None, and a backend that failed to import left
   `impl = None` process-wide behind a warning. `read_window_events` then
   polled a backend that could never answer, never set its readiness event,
   and recording startup hung with no stated cause. Adds
   `WindowCaptureUnavailable` and `require_impl()`, called at the top of the
   reader, and routes the window reader through `_run_task_fail_loud` like
   the input reader. `window/_linux.get_active_element_state` returned a
   fabricated `{"x": .., "y": .., "state": "placeholder"}` that was persisted
   verbatim onto every ActionEvent; it now returns None.

Every test in `tests/test_no_fabricated_capture.py` was mutation-checked:
each production change was reverted individually and the corresponding test
confirmed to fail. The window-reader revert reproduced the original hang as a
pytest-timeout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
@abrichr
abrichr merged commit e9e2d82 into main Jul 28, 2026
12 checks passed
@abrichr
abrichr deleted the sweep/false-success-capture branch July 28, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant