Skip to content

chore: refuse false benchmark completion - #280

Merged
abrichr merged 2 commits into
mainfrom
fix/evals-refuse-false-completion-20260728
Jul 28, 2026
Merged

chore: refuse false benchmark completion#280
abrichr merged 2 commits into
mainfrom
fix/evals-refuse-false-completion-20260728

Conversation

@abrichr

@abrichr abrichr commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

  • reject malformed actions instead of treating them as task completion
  • separate agent failures from unavailable infrastructure and evaluator attempts
  • require explicit action, setup, and evaluator receipts before a result can count
  • exclude unavailable attempts from measured outcome rates while preserving their diagnostics
  • reject malformed persisted results and success-shaped report values

Why

The evaluation stack could turn parser failures, transport failures, missing evaluator responses, and malformed result artifacts into plausible completed outcomes or measured zero scores. This change keeps those states explicit and prevents them from entering success metrics.

Validation

  • 2,136 light tests passed, with 55 skipped and 17 deselected
  • 100 focused evaluator and receipt tests passed after the final receipt correction
  • Ruff passed
  • the wheel and source archive passed the public source-policy boundary check
  • git diff --check passed

Release

This merge does not publish a package release.

@abrichr
abrichr merged commit 8132e60 into main Jul 28, 2026
1 check passed
@abrichr
abrichr deleted the fix/evals-refuse-false-completion-20260728 branch July 28, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant