ci(agent-image): eval-only pre-merge check for the compass-agent image - #204
Open
seal-agent wants to merge 1 commit into
Open
ci(agent-image): eval-only pre-merge check for the compass-agent image#204seal-agent wants to merge 1 commit into
seal-agent wants to merge 1 commit into
Conversation
seal-agent
force-pushed
the
compass-runner-1749-agent-image-eval-check
branch
from
August 7, 2026 05:03
fb130c0 to
a48d670
Compare
agent-image/ is deliberately not a moon project, so the CI gate never
builds it — the image has zero pre-merge coverage, and an eval-time
break (a bun-pin drift against the toolchain.nix assert) surfaces only
post-merge in the publish workflow while a consumer waits on a tag. This
adds the cheap tripwire that pulls that eval-time class to PR time.
Closes SEA-1749.
## How
The check forces the image derivation's .drvPath:
devenv eval containers.agent.derivation.drvPath
This evaluates the derivation — the .prototools read and the bun-drift
assert (agent-image/toolchain.nix:48-56) — and instantiates the .drv,
but realises none of the heavy closure. So it catches an eval-time break
(a bun-pin drift) in seconds and deliberately does not catch a
realise-time break (an entrypoint.nix FOD-hash invalidation), which only
the post-merge publish build exercises. That realise-time class stays
owned by the publish workflow's main-branch failure notification,
unchanged. This is the accepted tradeoff the frozen record names: a cheap
pre-merge tripwire, not a full image build on the hot path.
## Placement
A separate PR-triggered workflow, contents:read only (an eval needs no
registry credentials), path-filtered to the image's evaluation inputs:
- Not a step in ci.yml — folding a non-moon build into the gate's YAML is
the exact second-source-of-truth staleness the ONE-JOB doctrine forbids.
- Not a step in publish-agent-image.yml — that workflow deliberately
carries no PR trigger, to keep every token/secret off fork PRs. This
check needs a PR trigger and no credentials, so it gets its own
least-privilege workflow.
Its paths filter is the deliberate inverse of the publish workflow's on
one glob: it includes .prototools (which publish excludes), because a
bun-pin move there can only fail the toolchain.nix assert, never silently
change the output — and failing that assert at PR time is precisely this
check's purpose. Root package.json is included because the entrypoint
FOD reads it (fromJSON + workspace-glob readDir) at eval time; bun.lock
and the package sources stay excluded because they are fileset members
whose contents never reach an eval-time read, so they can only fail at
realise time.
The durable operational shape is folded into
docs/architecture/build-and-ci.md beside the publish section.
## Verification
End-to-end, at the exact CI-faithful command (clean env, no direnv/proto,
as on a GitHub-hosted runner):
- Clean .prototools -> exit 0, emits the .drv path.
- Drifted .prototools (bun pinned to a bogus version) -> exit 1, with the
toolchain.nix "has drifted from the .prototools pin" assert message.
Workflow validated with actionlint (0 findings); docs pass
root:markdownlint (0 errors). moon ci green at the pushed head via the
jj-hp gate.
## Open Questions
- [Design fork — parked for Matt, overnight] Placement + required-vs-
advisory. Implemented as a separate PR-triggered least-privilege
workflow, advisory (not a required check) — the stated assumption. It
matches the frozen record's own "separate workflow" reasoning for the
publish lane, and a nix-eval flake should never red a required merge
gate. The frozen record specifies the what (eval-only, nix eval, no
realise) but not the placement, so this is a genuine fork. If required
(branch-protection config) or a different placement is preferred, say
so and I'll adjust — not blocking, the check is correct and green either
way.
- [Scope boundary — parked for Matt, overnight] The paths filter is
scoped to eval-time content-read inputs, so it does not include
packages/** . One remote eval-failure class therefore does not
retrigger the check: the entrypoint FOD's workspace-glob expansion
does builtins.readDir on each glob parent (e.g. packages/), so deleting
an entire glob-parent directory would throw at eval, but a PR touching
only packages/** would not trigger this workflow. Stated assumption:
leave packages/** out — the failure needs deleting a whole workspace
glob-parent dir (very low probability), and widening the trigger to all
of packages/** for it trades a large trigger-surface increase for a
remote edge. If you want the tripwire to also cover glob-parent
structural deletion, I'll add packages/** — not blocking.
Co-authored-by: Matt Wilkinson <matt@sealedsecurity.com>
seal-agent
force-pushed
the
compass-runner-1749-agent-image-eval-check
branch
from
August 7, 2026 05:12
a48d670 to
71e6ee5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
agent-image/ is deliberately not a moon project, so the CI gate never
builds it — the image has zero pre-merge coverage, and an eval-time
break (a bun-pin drift against the toolchain.nix assert) surfaces only
post-merge in the publish workflow while a consumer waits on a tag. This
adds the cheap tripwire that pulls that eval-time class to PR time.
Closes SEA-1749.
How
The check forces the image derivation's .drvPath:
This evaluates the derivation — the .prototools read and the bun-drift
assert (agent-image/toolchain.nix:50-56) — and instantiates the .drv,
but realises none of the heavy closure. So it catches an eval-time break
(a bun-pin drift) in seconds and deliberately does not catch a
realise-time break (an entrypoint.nix FOD-hash invalidation), which only
the post-merge publish build exercises. That realise-time class stays
owned by the publish workflow's main-branch failure notification,
unchanged. This is the accepted tradeoff the frozen record names: a cheap
pre-merge tripwire, not a full image build on the hot path.
Placement
A separate PR-triggered workflow, contents:read only (an eval needs no
registry credentials), path-filtered to the image's evaluation inputs:
the exact second-source-of-truth staleness the ONE-JOB doctrine forbids.
carries no PR trigger, to keep every token/secret off fork PRs. This
check needs a PR trigger and no credentials, so it gets its own
least-privilege workflow.
Its paths filter is the deliberate inverse of the publish workflow's on
one glob: it includes .prototools (which publish excludes), because a
bun-pin move there can only fail the toolchain.nix assert, never silently
change the output — and failing that assert at PR time is precisely this
check's purpose.
The durable operational shape is folded into
docs/architecture/build-and-ci.md beside the publish section.
Verification
End-to-end, at the exact CI-faithful command (clean env, no direnv/proto,
as on a GitHub-hosted runner):
toolchain.nix "has drifted from the .prototools pin" assert message.
Workflow validated with actionlint (0 findings); docs pass
root:markdownlint (0 errors). moon ci green at the pushed head via the
jj-hp gate.
Open Questions
advisory. Implemented as a separate PR-triggered least-privilege
workflow, advisory (not a required check) — the stated assumption. It
matches the frozen record's own "separate workflow" reasoning for the
publish lane, and a nix-eval flake should never red a required merge
gate. The frozen record specifies the what (eval-only, nix eval, no
realise) but not the placement, so this is a genuine fork. If required
(branch-protection config) or a different placement is preferred, say
so and I'll adjust — not blocking, the check is correct and green either
way.
Co-authored-by: Matt Wilkinson matt@sealedsecurity.com