feat(config): state-ownership model — persist API pools, cluster fan-out, config-drift detection#33
Merged
Merged
Conversation
pools.json makes PUT /v1/pools survive restart (rebuild from operator intent, not the config seed); the mesh gossips a cluster-invariant config digest that warns and raises a gauge on divergence; SetPoolsCluster fans a set across peers; the gossip epoch persists so a backwards clock cannot regress placement. Durable writes use a unique-temp fsync helper with monotonic per-writer guards so concurrent persists can neither lose an update nor race a shared temp into a failed rename.
A restart under a backwards clock re-published an epoch equal to the one peers already held, and merge's strict `>` then rejected the node's fresh pools/templates/digest. UpdateSelf also published the new epoch into the gossiped view before persisting it, so a crash in that window left disk behind what peers had seen — the same stranding. Seed from max(wall, persisted+1), persist the candidate before publishing (hold the old advertised state on write failure), and reject an above-MaxInt64 persisted value as corrupt so the monotonic counter cannot saturate.
CMGS
added a commit
that referenced
this pull request
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #31.
What
Gives each state category one source of truth that a restart rebuilds from, and catches the cluster-invariant config divergences that otherwise surface as an intermittent 401 or a failed cross-node interception.
pools.json— the firstPUT /v1/poolsa node takes is persisted to<data_dir>/pools.json; from then on that file seeds the pools at boot, overriding theconfig.jsonpoolssection (a loud log line notes it; a second warns ifconfig.jsonwas edited afterward). Egress stays config-owned — the API rejects egress specs, sopools.jsonnever carries them and policies re-merge by pool key at boot. Delete the file to return to config-owned pools.api_token, tenant name→token set,preview_secret, egress CA root). HMAC-keyed bycluster_keywhen set; otherwise a token-free digest of tenant names + CA root, so nothing brute-forceable rides cleartext gossip. A divergence logs a warning the moment the node appears (not at the first unlucky redirect) and raises thesandboxd_config_digest_mismatchgauge. Warn-only: a rolling credential rotation is a legitimate transient mismatch and must never partition the mesh.Client.SetPoolsCluster— client-side fan-out of a pool set to the entry node + every peer, returning a per-node result. Pools are legitimately heterogeneous per node, so this is a fan-out, not a gossiped desired state. A non-nil error means peer discovery failed and only the entry node was reached — an incomplete apply to retry, kept distinct from a genuine single-node cluster (nil error).max(persisted, wall-clock)so a restart under a backwards clock cannot regress below peers' last-seen epoch and have its fresh state silently rejected.Durability
utils.WriteFileSync(temp + fsync + rename + dir-fsync) backs the machine-owned files. The temp name is per-call unique so concurrent writers to the same path can't race a shared temp into a failed rename, and each writer (poolStore, the epoch store) carries a monotonic sequence/floor guard so an already-superseded concurrent write can neither win the on-disk state nor return a spurious error for an already-applied request.Not included (deliberately)
SIGHUP/hot-reload, gossiping desired pools, write-back to
config.json, and dynamic tenant CRUD — all out of scope per the issue discussion.Tests
go test -race, both modules): pools.json survives restart / delete-restores-seed / restored egress pool needs an attachment / concurrentSetPoolspersists the latest with no spurious error; epoch round-trip / persisted-floor seed / monotonic under concurrent writes; cluster digest (keyed + token-free);SetPoolsClusterfan-out + peer-discovery-failure surfaces an error.GOWORK=off go build ./...both modules;go test -race -count=1 ./...green both modules;golangci-lint runGOOS=linux and GOOS=darwin, 0 issues, both modules;golangci-lint fmt --diffclean.PUT /v1/pools200,pools.jsonwritten, API pool survives a full daemon restart with the adoption log line (configpools:[]), divergent-tenant restart raises the gauge to 1 and logs the drift warning.