Skip to content

feat(config): state-ownership model — persist API pools, cluster fan-out, config-drift detection#33

Merged
CMGS merged 3 commits into
mainfrom
feat/config-state-ownership
Jul 13, 2026
Merged

feat(config): state-ownership model — persist API pools, cluster fan-out, config-drift detection#33
CMGS merged 3 commits into
mainfrom
feat/config-state-ownership

Conversation

@CMGS

@CMGS CMGS commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Closes #31.

What

Gives each state category one source of truth that a restart rebuilds from, and catches the cluster-invariant config divergences that otherwise surface as an intermittent 401 or a failed cross-node interception.

  • pools.json — the first PUT /v1/pools a node takes is persisted to <data_dir>/pools.json; from then on that file seeds the pools at boot, overriding the config.json pools section (a loud log line notes it; a second warns if config.json was edited afterward). Egress stays config-owned — the API rejects egress specs, so pools.json never carries them and policies re-merge by pool key at boot. Delete the file to return to config-owned pools.
  • Config-drift detection — each node gossips a digest of the cluster-invariant config (api_token, tenant name→token set, preview_secret, egress CA root). HMAC-keyed by cluster_key when set; otherwise a token-free digest of tenant names + CA root, so nothing brute-forceable rides cleartext gossip. A divergence logs a warning the moment the node appears (not at the first unlucky redirect) and raises the sandboxd_config_digest_mismatch gauge. Warn-only: a rolling credential rotation is a legitimate transient mismatch and must never partition the mesh.
  • Client.SetPoolsCluster — client-side fan-out of a pool set to the entry node + every peer, returning a per-node result. Pools are legitimately heterogeneous per node, so this is a fan-out, not a gossiped desired state. A non-nil error means peer discovery failed and only the entry node was reached — an incomplete apply to retry, kept distinct from a genuine single-node cluster (nil error).
  • Gossip epoch persistence — the epoch is persisted and seeded at max(persisted, wall-clock) so a restart under a backwards clock cannot regress below peers' last-seen epoch and have its fresh state silently rejected.

Durability

utils.WriteFileSync (temp + fsync + rename + dir-fsync) backs the machine-owned files. The temp name is per-call unique so concurrent writers to the same path can't race a shared temp into a failed rename, and each writer (poolStore, the epoch store) carries a monotonic sequence/floor guard so an already-superseded concurrent write can neither win the on-disk state nor return a spurious error for an already-applied request.

Not included (deliberately)

SIGHUP/hot-reload, gossiping desired pools, write-back to config.json, and dynamic tenant CRUD — all out of scope per the issue discussion.

Tests

  • Unit (go test -race, both modules): pools.json survives restart / delete-restores-seed / restored egress pool needs an attachment / concurrent SetPools persists the latest with no spurious error; epoch round-trip / persisted-floor seed / monotonic under concurrent writes; cluster digest (keyed + token-free); SetPoolsCluster fan-out + peer-discovery-failure surfaces an error.
  • Gates: GOWORK=off go build ./... both modules; go test -race -count=1 ./... green both modules; golangci-lint run GOOS=linux and GOOS=darwin, 0 issues, both modules; golangci-lint fmt --diff clean.
  • Two-node hardware acceptance (real memberlist mesh + real process restarts, bare metal): 9/9 — mesh join, digest gauge 0 with matching tenants, PUT /v1/pools 200, pools.json written, API pool survives a full daemon restart with the adoption log line (config pools:[]), divergent-tenant restart raises the gauge to 1 and logs the drift warning.

CMGS added 3 commits July 13, 2026 12:48
pools.json makes PUT /v1/pools survive restart (rebuild from operator
intent, not the config seed); the mesh gossips a cluster-invariant config
digest that warns and raises a gauge on divergence; SetPoolsCluster fans a
set across peers; the gossip epoch persists so a backwards clock cannot
regress placement. Durable writes use a unique-temp fsync helper with
monotonic per-writer guards so concurrent persists can neither lose an
update nor race a shared temp into a failed rename.
A restart under a backwards clock re-published an epoch equal to the one
peers already held, and merge's strict `>` then rejected the node's fresh
pools/templates/digest. UpdateSelf also published the new epoch into the
gossiped view before persisting it, so a crash in that window left disk
behind what peers had seen — the same stranding. Seed from
max(wall, persisted+1), persist the candidate before publishing (hold the
old advertised state on write failure), and reject an above-MaxInt64
persisted value as corrupt so the monotonic counter cannot saturate.
@CMGS CMGS merged commit 88fda81 into main Jul 13, 2026
1 check passed
@CMGS CMGS deleted the feat/config-state-ownership branch July 13, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

config: state-ownership model — persist API-applied pools (pools.json), cluster fan-out, config-drift detection

1 participant