Skip to content

ci(agent-image): eval-only pre-merge check for the compass-agent image - #204

Open
seal-agent wants to merge 1 commit into
mainfrom
compass-runner-1749-agent-image-eval-check
Open

ci(agent-image): eval-only pre-merge check for the compass-agent image#204
seal-agent wants to merge 1 commit into
mainfrom
compass-runner-1749-agent-image-eval-check

Conversation

@seal-agent

Copy link
Copy Markdown
Contributor

agent-image/ is deliberately not a moon project, so the CI gate never
builds it — the image has zero pre-merge coverage, and an eval-time
break (a bun-pin drift against the toolchain.nix assert) surfaces only
post-merge in the publish workflow while a consumer waits on a tag. This
adds the cheap tripwire that pulls that eval-time class to PR time.

Closes SEA-1749.

How

The check forces the image derivation's .drvPath:

devenv eval containers.agent.derivation.drvPath

This evaluates the derivation — the .prototools read and the bun-drift
assert (agent-image/toolchain.nix:50-56) — and instantiates the .drv,
but realises none of the heavy closure. So it catches an eval-time break
(a bun-pin drift) in seconds and deliberately does not catch a
realise-time break (an entrypoint.nix FOD-hash invalidation), which only
the post-merge publish build exercises. That realise-time class stays
owned by the publish workflow's main-branch failure notification,
unchanged. This is the accepted tradeoff the frozen record names: a cheap
pre-merge tripwire, not a full image build on the hot path.

Placement

A separate PR-triggered workflow, contents:read only (an eval needs no
registry credentials), path-filtered to the image's evaluation inputs:

  • Not a step in ci.yml — folding a non-moon build into the gate's YAML is
    the exact second-source-of-truth staleness the ONE-JOB doctrine forbids.
  • Not a step in publish-agent-image.yml — that workflow deliberately
    carries no PR trigger, to keep every token/secret off fork PRs. This
    check needs a PR trigger and no credentials, so it gets its own
    least-privilege workflow.

Its paths filter is the deliberate inverse of the publish workflow's on
one glob: it includes .prototools (which publish excludes), because a
bun-pin move there can only fail the toolchain.nix assert, never silently
change the output — and failing that assert at PR time is precisely this
check's purpose.

The durable operational shape is folded into
docs/architecture/build-and-ci.md beside the publish section.

Verification

End-to-end, at the exact CI-faithful command (clean env, no direnv/proto,
as on a GitHub-hosted runner):

  • Clean .prototools -> exit 0, emits the .drv path.
  • Drifted .prototools (bun pinned to a bogus version) -> exit 1, with the
    toolchain.nix "has drifted from the .prototools pin" assert message.

Workflow validated with actionlint (0 findings); docs pass
root:markdownlint (0 errors). moon ci green at the pushed head via the
jj-hp gate.

Open Questions

  • [Design fork — parked for Matt, overnight] Placement + required-vs-
    advisory. Implemented as a separate PR-triggered least-privilege
    workflow, advisory (not a required check) — the stated assumption. It
    matches the frozen record's own "separate workflow" reasoning for the
    publish lane, and a nix-eval flake should never red a required merge
    gate. The frozen record specifies the what (eval-only, nix eval, no
    realise) but not the placement, so this is a genuine fork. If required
    (branch-protection config) or a different placement is preferred, say
    so and I'll adjust — not blocking, the check is correct and green either
    way.

Co-authored-by: Matt Wilkinson matt@sealedsecurity.com

@linear-code

linear-code Bot commented Aug 7, 2026

Copy link
Copy Markdown

SEA-1749

@seal-agent
seal-agent force-pushed the compass-runner-1749-agent-image-eval-check branch from fb130c0 to a48d670 Compare August 7, 2026 05:03
agent-image/ is deliberately not a moon project, so the CI gate never
builds it — the image has zero pre-merge coverage, and an eval-time
break (a bun-pin drift against the toolchain.nix assert) surfaces only
post-merge in the publish workflow while a consumer waits on a tag. This
adds the cheap tripwire that pulls that eval-time class to PR time.
Closes SEA-1749.
## How
The check forces the image derivation's .drvPath:
    devenv eval containers.agent.derivation.drvPath
This evaluates the derivation — the .prototools read and the bun-drift
assert (agent-image/toolchain.nix:48-56) — and instantiates the .drv,
but realises none of the heavy closure. So it catches an eval-time break
(a bun-pin drift) in seconds and deliberately does not catch a
realise-time break (an entrypoint.nix FOD-hash invalidation), which only
the post-merge publish build exercises. That realise-time class stays
owned by the publish workflow's main-branch failure notification,
unchanged. This is the accepted tradeoff the frozen record names: a cheap
pre-merge tripwire, not a full image build on the hot path.
## Placement
A separate PR-triggered workflow, contents:read only (an eval needs no
registry credentials), path-filtered to the image's evaluation inputs:
- Not a step in ci.yml — folding a non-moon build into the gate's YAML is
  the exact second-source-of-truth staleness the ONE-JOB doctrine forbids.
- Not a step in publish-agent-image.yml — that workflow deliberately
  carries no PR trigger, to keep every token/secret off fork PRs. This
  check needs a PR trigger and no credentials, so it gets its own
  least-privilege workflow.
Its paths filter is the deliberate inverse of the publish workflow's on
one glob: it includes .prototools (which publish excludes), because a
bun-pin move there can only fail the toolchain.nix assert, never silently
change the output — and failing that assert at PR time is precisely this
check's purpose. Root package.json is included because the entrypoint
FOD reads it (fromJSON + workspace-glob readDir) at eval time; bun.lock
and the package sources stay excluded because they are fileset members
whose contents never reach an eval-time read, so they can only fail at
realise time.
The durable operational shape is folded into
docs/architecture/build-and-ci.md beside the publish section.
## Verification
End-to-end, at the exact CI-faithful command (clean env, no direnv/proto,
as on a GitHub-hosted runner):
- Clean .prototools -> exit 0, emits the .drv path.
- Drifted .prototools (bun pinned to a bogus version) -> exit 1, with the
  toolchain.nix "has drifted from the .prototools pin" assert message.
Workflow validated with actionlint (0 findings); docs pass
root:markdownlint (0 errors). moon ci green at the pushed head via the
jj-hp gate.
## Open Questions
- [Design fork — parked for Matt, overnight] Placement + required-vs-
  advisory. Implemented as a separate PR-triggered least-privilege
  workflow, advisory (not a required check) — the stated assumption. It
  matches the frozen record's own "separate workflow" reasoning for the
  publish lane, and a nix-eval flake should never red a required merge
  gate. The frozen record specifies the what (eval-only, nix eval, no
  realise) but not the placement, so this is a genuine fork. If required
  (branch-protection config) or a different placement is preferred, say
  so and I'll adjust — not blocking, the check is correct and green either
  way.
- [Scope boundary — parked for Matt, overnight] The paths filter is
  scoped to eval-time content-read inputs, so it does not include
  packages/** . One remote eval-failure class therefore does not
  retrigger the check: the entrypoint FOD's workspace-glob expansion
  does builtins.readDir on each glob parent (e.g. packages/), so deleting
  an entire glob-parent directory would throw at eval, but a PR touching
  only packages/** would not trigger this workflow. Stated assumption:
  leave packages/** out — the failure needs deleting a whole workspace
  glob-parent dir (very low probability), and widening the trigger to all
  of packages/** for it trades a large trigger-surface increase for a
  remote edge. If you want the tripwire to also cover glob-parent
  structural deletion, I'll add packages/** — not blocking.
Co-authored-by: Matt Wilkinson <matt@sealedsecurity.com>
@seal-agent
seal-agent force-pushed the compass-runner-1749-agent-image-eval-check branch from a48d670 to 71e6ee5 Compare August 7, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant