Skip to content

fix: preserve benchmark agent failures - #72

Merged
abrichr merged 1 commit into
mainfrom
agent/ml-false-success-20260728
Jul 28, 2026
Merged

fix: preserve benchmark agent failures#72
abrichr merged 1 commit into
mainfrom
agent/ml-false-success-20260728

Conversation

@abrichr

@abrichr abrichr commented Jul 28, 2026

Copy link
Copy Markdown
Member

What changed

  • Return terminal error actions when hosted VLM calls fail.
  • Return terminal error actions when API, unified-baseline, or WAA demo responses cannot be parsed.
  • Preserve explicit model-issued completion as done.

Why

The benchmark action schema already supports error, and the evaluation adapters handle it as a distinct terminal state. Returning done for provider and parse failures made task completion and agent failure indistinguishable.

Validation

  • 470 unit tests passed; 5 skipped.
  • 77 focused failure and baseline tests passed.
  • Ruff and formatting checks passed for every changed file.
  • git diff --check passed.
  • The optional integration collection requires openadapt-evals, which is not installed in this worktree environment.

@abrichr
abrichr force-pushed the agent/ml-false-success-20260728 branch from e7810be to e61980a Compare July 28, 2026 08:34
@abrichr
abrichr marked this pull request as ready for review July 28, 2026 08:39
@abrichr
abrichr merged commit 4090310 into main Jul 28, 2026
4 checks passed
@abrichr
abrichr deleted the agent/ml-false-success-20260728 branch July 28, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant