Skip to content

Run only the CI a pull request can affect - #296

Merged
jserv merged 1 commit into
mainfrom
ci-refine-proof
Aug 15, 2026
Merged

Run only the CI a pull request can affect#296
jserv merged 1 commit into
mainfrom
ci-refine-proof

Conversation

@jserv

@jserv jserv commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Every pull request paid for every workflow whatever it touched: seventeen macOS runners proving seventeen targets, three more analyzing a build, one building and testing, for a branch that edited a test script. The proofs are the bulk of that, and a target whose inputs a branch does not touch already has the verdict the base established.

scripts/proof-scope.py decides which proof targets a set of changed files can reach. It takes each proved source's include closure from the compiler's own -MM, preprocessed with that target's VERIFY_CPP_DEFS and unioned with its VERIFY_SCAN list, so the scan sees what the prover sees. Inputs no closure can see widen the scope to everything: the makefiles, the checkers the recipes run, the Frama-C stub headers, the workflow itself. So does every "cannot tell" answer, whether an unresolvable base ref or a compiler scan that cannot be trusted. Narrowing on a guess turns a speed-up into a correctness problem; proving something twice does not.

verify.yml builds its matrix from that answer on a pull request, so an untouched target gets no runner rather than a runner that skips its work. An empty scope is a verdict rather than a failure, and only the mutation gate decides what counts as a pass; the verify job mirrors it, so the rule lives in one place instead of two shell blocks that have to be edited together. A push to main and a merge queue run prove everything, so the guarantee on the branch a pull request merges into is never the narrowed one.

build.yml and static-analysis.yml take the cheaper mechanism their inputs allow, a paths-ignore list of what cannot reach them, on pull_request only so there is one list per file and nothing can drift. The analysis jobs analyze "make elfuse" and nothing else, which is why tests/ is inert for them. The failure directions differ from the proof scoping, and that is why one needs a self-test and the other does not: a stale entry there matches nothing and the workflow runs more, while a stale harness entry widens nothing and the proofs run less.

lint.yml loses its path filter entirely, which is what makes it the place the self-test can live. It had been ignoring markdown, and check-skill-refs.py validates nothing but the markdown under .claude, which the other two workflows treat as inert, so a documentation-only pull request ran no CI at all. That checker was also wired into no workflow until now, and it failed on its first run, naming two files the CI split had already removed.


Summary by cubic

Runs only the CI jobs a pull request can affect, widening to the full set on any uncertainty. Previously every PR ran all proofs, mutations, builds, and analysis; now proofs/mutations narrow to targets reachable from the diff, builds/analysis use path filters, and lint always runs to cover self-tests and docs checks.

  • Proofs and mutations

    • Adds scripts/proof-scope.py to derive per-target reach from each source’s compiler include closure with VERIFY_<T>_CPP_DEFS unioned with VERIFY_<T>_SCAN; any “cannot tell” widens to all. Separates “scheduling-only” files and self-tests to forbid budgets or make in scheduling workflows.
    • verify.yml reads targets from mk/verify.mk, narrows matrices on PRs, and proves/mutates the full set on push and merge_group. Introduces verify-proofs to backfill anything mutation legs don’t prove; verify-mutants-gate stays the required aggregate and scripts/check-proof-targets.py asserts the matrix is derived and that macOS jobs are reachable from it.
    • Adds composite action ./.github/actions/framac for shared setup and moves to Frama-C 33.0. scripts/check-mutants.py delegates changed-only selection to proof-scope.py; scripts/verify-mk.py exposes targets/SCAN/CPP_DEFS, discovers proof workflows, and rejects multi-source VERIFY_<T>_SRC. scripts/check-char-signedness.py reads the shared table. mk/verify.mk corrects data-model notes (plain char is signed on arm64 macOS) and records why the macos_arm machdep cannot be used.
  • CI and repo

    • build.yml and static-analysis.yml use paths-ignore on pull_request only (tests are inert for analysis). lint.yml drops its path filter, runs package-free checks first with if: !cancelled(), runs proof-scope.py --self-test, and wires scripts/check-skill-refs.py so docs-only PRs still run CI.
    • .ci/check-security.sh now flags “agent scratch markers”; removes one from src/syscall/asyncio.c. Updates SKILL.md to reference mk/verify.mk.
    • Required action: Install Frama-C 33.0. To reproduce CI scoping locally: make verify-mutants MUTANT_SINCE=<base>.

Written for commit 8993631. Summary will update on new commits.

Review in cubic

cubic-dev-ai[bot]

This comment was marked as resolved.

@sysprog21 sysprog21 deleted a comment from cubic-dev-ai Bot Aug 14, 2026
cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv
jserv force-pushed the ci-refine-proof branch 4 times, most recently from a05a4f7 to a84293b Compare August 14, 2026 20:11
Every pull request paid for every workflow whatever it touched:
seventeen macOS runners proving seventeen targets, three more analyzing
a build, one building and testing, for a branch that edited a test
script. The proofs are the bulk of that, and a target whose inputs a
branch does not touch already has the verdict the base established.

scripts/proof-scope.py decides which proof targets a set of changed
files can reach. It takes each proved source's include closure from the
compiler's own -MM, preprocessed with that target's VERIFY_<T>_CPP_DEFS
and unioned with its VERIFY_<T>_SCAN list, so the scan sees what the
prover sees. Inputs no closure can see widen the scope to everything:
the makefiles, the checkers the recipes run, the Frama-C stub headers,
the workflow itself. So does every "cannot tell" answer, whether an
unresolvable base ref or a compiler scan that cannot be trusted.
Narrowing on a guess turns a speed-up into a correctness problem;
proving something twice does not.

verify.yml asks that twice. Which targets to prove is one question;
which mutation sets to re-run is the other, and it is narrower, because
a file that only decides what runs cannot change whether a target
rejects a broken source. SCHEDULING_FILES names those files, everything
else counts as judging so an unclassified newcomer widens both, and the
self-test refuses a scheduling workflow that carries a prover budget or
a make invocation. An empty scope is a verdict rather than a failure. A
push to main and a merge queue run prove and mutate everything, so the
guarantee on the branch a pull request merges into is never the narrowed
one.

The jobs follow the same shape. A mutation leg proves an unmutated
control through the very rule the proofs use, so the proving job takes
only what the matrix will not cover, which over the last 400 commits is
nothing at all: sharding the proofs cost 82 seconds of setup per leg to
do 23 seconds of proving. Both halves report under the check name that
predates the split, which covered proofs and mutations together and
still does, and check-proof-targets.py now asserts every macOS job is
reachable from it. The setup those jobs share, and the toolchain pins
they have to agree on, live in .github/actions/framac; the analyzer
moves to Frama-C 33.0, checked rather than assumed, at 803 of 803
obligations across all 17 targets with the mutation gate still biting.

build.yml and static-analysis.yml take the cheaper mechanism their
inputs allow, a paths-ignore list of what cannot reach them, on
pull_request only so there is one list per file and nothing can drift.
The analysis jobs analyze "make elfuse" and nothing else, which is why
tests/ is inert for them. The failure directions differ from the proof
scoping, and that is why one needs a self-test and the other does not: a
stale entry there matches nothing and the workflow runs more, while a
stale harness entry widens nothing and the proofs run less.

lint.yml loses its path filter entirely, which is what makes it the
place the self-test can live. It had been ignoring markdown, and
check-skill-refs.py validates nothing but the markdown under .claude,
which the other two workflows treat as inert, so a documentation-only
pull request ran no CI at all. That checker was also wired into no
workflow until now, and it failed on its first run, naming two files the
CI split had already removed.

lint.yml runs its package-free checks first and carries if !cancelled()
on every step, the apt setup included, so one failure reports alongside
the rest instead of leaving the tool-dependent checks to fail on a
missing binary. verify-mk.py refuses a VERIFY_<T>_SRC naming two sources
rather than keeping the first, since every consumer assumes one, and
check-char-signedness.py, the last reader with its own regex over those
lines, goes through the shared table now.

Two harness entries answer a question about content rather than about
the path, because the name rule was measurably too blunt: over the last
400 commits the top-level Makefile was the sole reason 36 of the 60
full-scope runs happened, every one of them a rule for a test binary or
a source added to SRCS. It reaches a proof only through its include
lines, since it defines none of the variables the verify recipe reads,
and mk/config.mk qualifies the same way because it defines no rules at
all. mk/common.mk does not: it owns the $(BUILD_DIR) rule every verify
target carries as an order-only prerequisite, which a slicer reading
values cannot see.

That slicer drops what is provably inert rather than keeping what it
recognizes, so an unrecognized construct widens. Reviewers found the
difference the hard way: sorting the slice hid a reordered include, and
override, define, a target-specific assignment, vpath and unexport each
reached a proof while every line an earlier pattern read stayed
byte-identical. All of them are self-test cases now, alongside the
conditions the approach rests on, that neither file defines a verify
rule or a target the verify rules depend on.

Which names count as proof-relevant is a fixpoint, not a union: start
from what mk/verify.mk expands, since that is the only file the recipe
reads, then follow definitions. The union was self-defeating, because
mk/config.mk references its own test lists, so NATIVE_TESTS and its
neighbours counted and any edit to them re-proved everything. The seed
still takes references outside an assignment's right-hand side in the
other three makefiles, or mk/common.mk's "ifeq ($(V),1)", which picks
the Q every recipe expands, would leave V inert. Over the same 400
commits: 231 run no proof at all, 16 run the full set, and the targets
mutated fall from 1103 to 374.

The data model comment was wrong where it mattered most. It claimed
plain char is unsigned on arm64 macOS; it is signed there, and unsigned
on aarch64-linux, which is the cross toolchain for the guest tests, both
answers taken from the compilers rather than from memory. So gcc_x86_64
matches the platform the proved sources compile for on all four
properties, not three. Frama-C 33 does ship a macos_arm machdep and it
cannot be used here: it is not GCC-based, and the atomics stub pulls in
Frama-C's __fc_gcc_builtins.h, which refuses __int128 outside a
GCC-based machdep. Every target aborts at parse time, which is now the
recorded reason for the pin.

The mutation matrix is packed rather than one leg per target. A leg
spends about 90 seconds on Homebrew and the opam switch before it proves
anything, and GitHub bills wall time rounded up to the minute, so 17
legs cost 85 macOS-minutes where 5 buckets cost 61; five also fits one
concurrency wave, so the wall clock improves too. check-mutants.py packs
because it owns the mutation counts it packs by, and each leg still runs
one target at a time so a failure names the target rather than the
group.

Three defects in this machinery came out of review rather than out of
the self-test, and all three are now cases in it. A "printf | grep -q"
under pipefail reports "unchanged" once the diff fills the pipe buffer,
which would have skipped clang-format and cppcheck on exactly the
largest pull requests. check-mutants.py was taking its source table
through proof-scope.py, putting a file classified as unable to affect a
mutation verdict on the path that decides which file gets mutated. And
check-proof-targets.py's job splitter read workflow sub-keys as jobs
while missing any job id not starting lowercase, so the check meant to
notice a prover job leaving the required check could not see one.
@jserv
jserv merged commit f3b2ead into main Aug 15, 2026
14 of 18 checks passed
@jserv
jserv deleted the ci-refine-proof branch August 15, 2026 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant