fix: stop fabricating display metrics, window data, and UIA evidence - #61
Merged
Conversation
Five paths returned a plausible value when the measurement or observation
could not be taken at all, so the recording reported success while carrying
invented data.
1. `platform.get_display_pixel_ratio()` returned 1.0 from every failure
branch on all three platforms. 1.0 is a real, common answer, so a Retina
display whose probe failed (no Screen Recording permission, mss failing
in a headless or threaded context, a missing shcore/GetDpiForMonitor) was
recorded as a standard-density display and every captured coordinate was
rescaled by half. The CHANGELOG already describes this exact harm and
fixed only the persistence side. The probe now raises
`DisplayMetricsUnavailable`, and `create_recording` persists NULL - the
column is nullable precisely so that unknown is representable.
2. `platform.get_screen_dimensions()` returned the literal (1920, 1080) when
every measurement path failed. It now raises.
3. `platform.is_accessibility_enabled()` returned True from every
"could not check" branch. The Windows implementation called
`IsUserAnAdmin()`, discarded the result, and returned True from the try,
the except, and the outer except, so the answer carried no information at
all. It is now a tri-state: True verified, False verified, None
undetermined.
4. `structural_observer/windows._observe_with_runtime` emitted a complete
`StructuralObservation` with an identity-free element when the element's
`element_info` could not be read. That is positive evidence that
windows_uia observed the action, indistinguishable from an element that
legitimately exposes nothing, and downstream compile reads the presence of
a structural observation as accessibility ground truth. It now returns
None.
5. `window.get_active_window_data()` returned `{}` despite its annotation and
docstring both promising None, and a backend that failed to import left
`impl = None` process-wide behind a warning. `read_window_events` then
polled a backend that could never answer, never set its readiness event,
and recording startup hung with no stated cause. Adds
`WindowCaptureUnavailable` and `require_impl()`, called at the top of the
reader, and routes the window reader through `_run_task_fail_loud` like
the input reader. `window/_linux.get_active_element_state` returned a
fabricated `{"x": .., "y": .., "state": "placeholder"}` that was persisted
verbatim onto every ActionEvent; it now returns None.
Every test in `tests/test_no_fabricated_capture.py` was mutation-checked:
each production change was reverted individually and the corresponding test
confirmed to fail. The window-reader revert reproduced the original hang as a
pytest-timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five paths in this package returned a plausible value when the measurement or observation could not be taken at all. Each one is written into the recording and is indistinguishable from a real one.
1.
get_display_pixel_ratio()returned 1.0 from every failure branch (highest blast radius)platform/__init__.py:107,platform/darwin.py,platform/windows.py,platform/linux.py— all four endedreturn 1.0.1.0is a real, common answer. A Retina display whose probe failed — no Screen Recording permission,mss.mss()failing in a headless or threaded context, a missingshcore/GetDpiForMonitoron Windows — was recorded as a standard-density display.recorder.py:1313writes that number onto the Recording;capture.py:540andvisualize/html.py:60read it back. Every captured coordinate is then rescaled by half, and nothing about the recording looks wrong.The repo already knows this harm. From
CHANGELOG.md:That change fixed the persistence side and added a nullable column so unknown would be representable — while the measurement side kept fabricating 1.0, so the nullable column could never actually be NULL.
Now: the probe raises
DisplayMetricsUnavailable, andcreate_recordingcatches it, logs at error level, and persistsNone.2.
get_screen_dimensions()returned the literal(1920, 1080)Same shape, in all four places. Now raises
DisplayMetricsUnavailable.3.
is_accessibility_enabled()returnedTruemeaning "I did not check"The Windows implementation is the sharpest example — the check's result was computed and thrown away, and every branch returned True:
A consumer gating "safe to start recording" on this got a green light and then recorded zero input events. Now a tri-state:
Trueverified,Falseverified,Noneundetermined, with the docstring stating thatNoneis not a synonym forTrue.4. UIA emitted a successful observation for an element it could not read
structural_observer/windows.py:_observe_with_runtimecalled_as_element(target)unconditionally. Whenelement_infowas unreadable — COM failure, element vanished —_element_fieldsreturned{}and the function still returned a fully-formedStructuralObservation(provider="windows_uia", ...)with every identity fieldNone.recorder.py:877attaches it to the ActionEvent, and downstream compile reads the presence of a structural observation as accessibility ground truth.The module's own sibling policy is the opposite:
_present_stringexplicitly refuses to emit partial identity ("Never retain a partial identity"), and_candidate_cardinalityreturnsNonefor unknown. Now_observe_with_runtimereturnsNonewhen the element could not be read.5. The window layer could not say "no backend"
get_active_window_data()returned{}while its annotation saiddict | Noneand its docstring said "or None if the state is not available".impl = Noneprocess-wide behind alogger.warning.read_window_eventsthen polled a backend that could never answer, hitif not window_data: sleep(0.1); continueforever, and never set its readiness event — so_wait_for_tasks_startedspun indefinitely (the thread is alive, sostopped_before_readynever fires) and recording startup hung with no stated cause.window/_linux.get_active_element_statereturned a fabricated{"x": .., "y": .., "state": "placeholder"}, persisted verbatim onto every ActionEvent as if it were captured accessibility data.Adds
WindowCaptureUnavailableandrequire_impl(), called at the top ofread_window_events; returnsNoneper the documented contract; routes the window reader through_run_task_fail_loud(as the input reader already is) so the failure reachestask_errorswith its real exception; and_linux.get_active_element_statenow returnsNone.Mutation checks
Every test in
tests/test_no_fabricated_capture.pywas verified against pre-fix code by reverting each production change individually:platform/__init__.pydefaultstest_pixel_ratio_raises_instead_of_returning_oneFailed: DID NOT RAISE DisplayMetricsUnavailableplatform/__init__.pydefaultstest_screen_dimensions_raise_instead_of_returning_1920x1080Failed: DID NOT RAISE DisplayMetricsUnavailableplatform/__init__.pydefaultstest_accessibility_state_is_none_when_undeterminedassert True is Nonecreate_recording->pixel_ratio = 1.0test_recording_stores_null_pixel_ratio_when_unmeasurableAssertionError: assert 1.0 is Nonerequire_impl()call removed from readertest_window_reader_refuses_instead_of_polling_foreverFailed: Timeout (>15.0s) from pytest-timeout— the original hang, reproducedwindow/__init__.py({}return, norequire_impl)test_require_impl_raises_when_no_backend_loadedFailed: DID NOT RAISE WindowCaptureUnavailablewindow/__init__.py({}return)test_active_window_data_is_none_not_empty_dictassert {} is None_observe_with_runtimeelement orderingtest_unreadable_element_yields_no_observationassert StructuralObservation(... element=StructuralElement(automation_id=None, role=None, ... supported_patterns=[]), process=None, window=None, ancestry=None, candidate_count=None ...) is None_linux.get_active_element_stateplaceholdertest_element_state_is_none_not_a_placeholder_dictassert {'x': 10, 'y': 20, 'state': 'placeholder'} is NoneThe Linux test stubs
xcffibso it runs on the whole PR matrix rather than skipping everywhere but Linux.Verification
uv run ruff check openadapt_capture/clean;uv run pytest tests/ --ignore=tests/test_browser_bridge.py --timeout=120-> 367 passed, 7 skipped.Candidates examined and dismissed
video.pyFFmpeg boundary — exemplary, no change._run_checkedinspectsreturncodeand raises;FFmpegVideoWriter.close()raises"FFmpeg returned success without a non-empty output"when the file is missing or zero-length;_abort_and_reap'sexcept subprocess.TimeoutExpired: passfollows an explicitprocess.kill()and is a reap, not a swallow.audio.pyexcept ImportError: passin_get_best_transcription_backend— deliberate and documented: it returnsNone, andrequire_local_transcription_backend()converts that to aNoLocalTranscriptionBackendrefusal so a recording that cannot be transcribed locally is never started. The docstring states the reason (a hosted fallback would upload a raw waveform).input_observer/base.py,factory.py,linux.py,windows.py— clean. Queue overflow, consumer failure, delivery-thread death and shutdown timeouts all raise; unsupported platforms raiseInputObserverUnavailableError; every Linux degradation is named in the module docstring.window_capture.pyresolve/capture/translate — deliberately fail-loud withWindowCaptureError._supported_patternsreturning a possibly-short list —pywinautosignals "pattern not supported" by raising, so the broad catch is the idiomatic probe. Left alone; noted below.platform/linux.get_active_window_infodegrading title/geometry/pid to""/0— a fabricated-success shape, but it has no in-repo callers and no consumer outside this tree. Reported, not fixed.window_capture.py:211scale = ... if bounds_w else 1.0on zero-width bounds, and:464dropping a matching window whose rect cannot be read — plausible but unconfirmed reachable; both are Win32/macOS-only paths that usually fail earlier. Reported, not fixed, to keep this change reviewable.input_observer/darwin.py:552_is_injectedreturningFalsewhenkCGEventSourceUnixProcessIDis absent — requires a PyObjC build missing a long-stable constant. Theoretical.Fake-gate inventory
.github/workflows/contains no|| true,continue-on-error,exit 0,set +e, or swallowed step across all seven workflows.lintrunsruff check openadapt_capture/as a real gate, andpackage-contractrunsverify_distribution.pyon the built archives.Two observations, both deliberate rather than fake:
test-windowsandtest-macoscarryif: github.event_name != 'pull_request', so on a PR only the Linux 3.10–3.12 matrix, lint, and package-contract run. The workflow comments state this is a cost decision and that cross-platform qualification runs on the exact main commit. Worth knowing when reading a green PR: the Windows UIA and macOS paths in this change are covered by these tests on the post-merge main run, not on the PR.release.ymlisworkflow_dispatchonly, so merging thisfix:will not cut a release. Publication stays a deliberate operation gated on the exact main commit having completedtest.yml.