Skip to content

fix: preserve agent and readiness failures - #279

Merged
abrichr merged 3 commits into
mainfrom
agent/evals-false-success-20260728
Jul 28, 2026
Merged

fix: preserve agent and readiness failures#279
abrichr merged 3 commits into
mainfrom
agent/evals-false-success-20260728

Conversation

@abrichr

@abrichr abrichr commented Jul 28, 2026

Copy link
Copy Markdown
Member

What changed

  • Preserve provider, parser, planner, and missing-observation failures as terminal error actions instead of clean done actions.
  • Keep explicit model decisions distinct: DONE remains done, FAIL becomes error, and WAIT remains wait.
  • Require a readable and confirmed unlocked Windows desktop before the WAA recovery helper reports readiness.

Why

The evaluation harness treats done as a model completion claim. Several failure paths used the same action type, which erased the difference between task completion and an agent or infrastructure failure. The WAA helper also reported readiness after a successful probe even when lock-screen inspection failed or unlock input did not take effect.

Validation

  • 376 focused agent and false-success tests passed.
  • 1,739 repository tests passed; 62 skipped.
  • The remaining broad-suite failures were pre-existing environment requirements: local openadapt-capture, a writable home session directory, repository-only synthetic demo assets, and one cross-repository test that expects the currently released openadapt-ml behavior.
  • Ruff passed for every changed Python file.
  • git diff --check passed.

@abrichr
abrichr force-pushed the agent/evals-false-success-20260728 branch from 461e656 to c195188 Compare July 28, 2026 08:34
@abrichr
abrichr marked this pull request as ready for review July 28, 2026 08:39
@abrichr
abrichr merged commit e6083f3 into main Jul 28, 2026
1 check passed
@abrichr
abrichr deleted the agent/evals-false-success-20260728 branch July 28, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant