Add codex and change cost calc - #3
Draft
Intron7 wants to merge 21 commits into
Draft
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3 +/- ##
===========================================
+ Coverage 63.20% 73.91% +10.71%
===========================================
Files 22 26 +4
Lines 2829 4098 +1269
===========================================
+ Hits 1788 3029 +1241
- Misses 1041 1069 +28
🚀 New features to boost your workflow:
|
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
for more information, see https://pre-commit.ci
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Provider usage or credit running out is an infrastructure failure, not evidence about the model, so it must not be scored as an agent failure. Runs that hit provider exhaustion get reason `provider_exhausted` and a `valid: false` result. The pass cancels only that provider's remaining cells, lets other providers finish their running and queued work, and exits non-zero. Invalid cells stay pending, so replenishing the credential and rerunning the same command retries them without `--no-resume`. Reports and `improve` refuse invalid evidence outright. Both adapters now preserve provider stderr on failure so the runner can classify exhaustion from the actual provider error text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`acumen bench` with no arm flag covered only the baseline, so comparing a skill against it took one invocation per version and `--dry-run` never showed what the project as a whole implied. A bare `acumen bench` now covers every arm on disk — the baseline plus each version in `skills/` — benched one after another against a single prepared target, with per-arm counts, per-arm tallies and a combined total. Resume still applies per arm, so adding `skills/v3` and rerunning costs only v3. `--no-skill` / `--skill vN` restrict the pass to one arm as before, and `--dry-run` plans the same set for free. Skills load before target prep, so a version that will not load stops the pass at planning instead of being dropped from the comparison or failing part-way through a paid run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The teal ramp had luna and terra swapped: luna took the mid-dark step and terra the lightest, though terra is the more potent of the two. Published rates put the tiers at sol $5/$30, terra $2/$12, luna $0.20/$1.20, so the ramp now steps sol -> terra -> luna, darkest to lightest, matching the warm Anthropic ramp. _MODEL_ORDER follows, which also puts the legend and bar order in the same sequence config.yaml already scaffolds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prices move, and each run's cost is frozen into its result.json when the run finishes and never recomputed. A table compiled into a release is therefore wrong from whatever date the providers next change a price, and that wrongness gets stored rather than corrected on the next run. So no rates ship with the package at all. They come from the providers' pricing pages, or from a `prices:` pin in config.yaml, which still wins: only the operator knows what a negotiated rate or a gateway markup is. A model neither layer covers is unpriced, which was already a first-class state throughout (cost_usd is None, never 0.0). bench resolves rates before it spends anything and fails the pass if the pages cannot be read: cost is a headline metric, and a pass that cannot establish rates has not earned the numbers it would print. draft, improve, tasks and ship fetch too but degrade to unpriced, since their cost line is progress reporting rather than stored evidence. That degradation warns about its one real consequence: Codex's max_usd cap is derived from these rates, so an unpriced Codex run has no enforceable budget cap. Each run now records price_source and price_rates_as_of alongside the rates themselves, so passes months apart stay individually attributable and one report can mix them. The report flags arms priced on different dates, whose cost gap contains any price change and not only the skill's effect. Two things this surfaced: - A dated snapshot (claude-haiku-4-5-20251001) now falls back to its family rate. Providers publish one rate per family and never list the snapshot IDs, and acumen init scaffolds one, so a default project would otherwise have benched an entirely unpriced model. The old table hid this by hardcoding both IDs. - Three CLI tests began reaching the live network once bench started fetching, and passed only because the machine happened to be online. An autouse fixture now fails any test that touches the network, which is what the suite's docstring already claimed. acumen prices reads the pages too, since there is no offline table left to print; --refresh now reports pins that have drifted from the published price, the only rates that can still go stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Signed-off-by: Intron7 <sdicks@nvidia.com>
The option was reachable but undiscoverable: nothing in a generated config.yaml mentioned it, so the only way to learn egress is configurable was to read the Config dataclass. The comment leads with the failure mode rather than the syntax. Naming any host denies every other one, and the hosts a target reaches are decided at runtime by its own loaders — mirrors, CDN aliases, resource-server fallbacks — so a hand-written list is usually short of what a pass needs. That is now loud rather than silent, but it still costs a pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PauBadiaM
force-pushed
the
add-codex-and-change-cost-calc
branch
from
August 7, 2026 22:36
91a559d to
ef2bc43
Compare
Sandboxing every run also put an egress policy in front of it, and the policy denied the hosts a benchmark target actually needs. A connectivity probe from the decoupler pass found GitHub, PyPI, conda and GCS reachable while EBI, Zenodo, figshare, OmniPath, NCBI, cellxgene, Ensembl and HuggingFace all returned 000 or a proxy 403. A refused host does not stop an agent. It cannot see the refusal as a policy decision either -- the sandbox answers CONNECT with 403, which arrives as an ordinary connection error -- so it improvises from memory and returns a confident wrong answer. 81 of 150 claude-opus-5 transcripts in that pass carry network errors, and the results scored as evidence that the models had got worse. Remove the allowed_domains config key rather than widening it. Which hosts a target reaches is decided at runtime by its own loaders, through mirrors, CDN aliases and resource-server fallbacks, so no list written before a pass can be complete, and the failure mode of an incomplete one is silent. Codex now runs its proxy in `full` rather than `limited` mode. The two modes differ in more than their domain table: under `limited` the proxy also enforces a method policy, answering anything outside GET/HEAD/OPTIONS with 403, which breaks every POST a loader makes. Verified with `codex sandbox` and features.network_proxy=true, where a POST goes 403 -> 200 across the change. Its filesystem profile is untouched, so Codex keeps kernel-level confinement. Claude gets no OS sandbox at all. Its sandbox cannot be configured for open egress: the proxy ignores sandbox.network.allowedDomains coming from the --settings file the SDK passes, whether the entry is "*" or an explicit host, and ignores WebFetch(domain:...) rules too. Measured on CLI 2.1.224 -- with the sandbox on, zenodo.org, ftp.ebi.ac.uk and omnipathdb.org are all unreachable; with it off, all three answer. So the filesystem boundary moves to a PreToolUse guard, which refuses any tool call naming a path outside the run's roots while leaving system paths and the target venv usable. This is a weaker guarantee than the kernel gave, and worth stating plainly: it stops the agent exploring the host, but a program the agent starts can still open what it likes. That is the boundary a benchmark needs -- no discovering another task's data or the operator's files -- rather than a defence against an adversarial agent. The shipper opts out, being the one agent that runs in the operator's real environment on purpose. Also stop scoring a run whose sandbox never started. bubblewrap reports its own startup failure as the command's output rather than on the agent CLI's stderr, which is the only stream the existing check read, so 365 of 443 Codex runs in that pass died before executing anything and still entered the report as graded results. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
for more information, see https://pre-commit.ci
Two things a benchmark can get wrong before a model is ever involved. A task's `answer` was trusted blindly. A typo in one, an answer the generator mis-transcribed, or a pipeline that no longer runs makes every model fail that task: money spent, and the failure reads in the report as the model's fault rather than the task's. So every answer now has to be reproducible. Each task keeps a reproducer at `tasks/<id>-<split>.py` that redoes the analysis in the target venv and writes its answer to `answer.md`, graded by the same comparison a benchmark run gets, and `acumen tasks` writes them as it generates the tasks instead of discarding them. `acumen check` runs them all and reports a row per task and split plus the summary statistics. It probes that the package imports before running anything, since that one failure would otherwise be reported once per task. A task needing no code sets `needs_script: false`. Reproducing an answer only proves the code and the answer agree. It says nothing about the prompt, and a prompt asking for something else fails every agent that reads it correctly: one asking for pathways "sorted by score (ascending)" whose script and answer were both descending graded every correct solver wrong. So a second phase has one agent read every split's prompt, recorded answer and reproducer together and add an `ok`/`mismatch` column, with one line naming the contradiction and one naming the fix. It reads a staged copy with no path back to `tasks.yaml`, never edits a task, and takes its model from the new `check_model`. `--no-review` runs the reproducers alone and spends nothing. Separately, the report's cost-vs-success figure now orders skill versions with arrows rather than marker shapes: each model's marks are joined baseline to v1 to v2 in its own colour, which frees the version labels off the panel and keeps the key at two marks and an arrow however many versions ran. The Pareto staircase runs out to the right edge, where it used to stop at the dearest frontier mark and leave the stretch beyond looking like open ground. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.