feat(config): state-ownership model — persist API pools, cluster fan-out, config-drift detection

docs/cluster.md

@@ -123,10 +123,65 @@ handle `Sandbox.Promote` returns is still owner-bound — `template.New` and
 `template.Delete` dial the owner directly, no gossip involved — and is the
 race-free choice immediately after a promote.
 
+## State ownership
+
+Each kind of state has exactly one source of truth, and a restart rebuilds
+fully from it:
+
+| state | source of truth | survives restart |
+|---|---|---|
+| operator config (`tenants`, `secrets`, egress policies, `bridge`/`network`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot |
+| API-applied pool targets (`PUT /v1/pools`) | `<data_dir>/pools.json` (machine owned) | yes |
+| claims | the claims journal + `Reconcile` | yes |
+| placement hints (warm counts, template sets) | gossip | rebuilt |
+
+Pools are managed API-first. The first time a node takes `PUT /v1/pools`, it
+writes the applied set to `<data_dir>/pools.json` and from then on **that file
+seeds the pools at boot**, overriding the `pools` section of `config.json` (a
+loud log line notes this; if the two disagree because someone edited
+`config.json` afterward, a second warning names it). The file's presence is the
+ownership marker — delete `pools.json` to return the node to config-owned pools.
+Egress stays config-owned regardless: the API rejects egress specs, so
+`pools.json` never carries them, and egress policies re-merge from `config.json`
+by pool key at boot.
+
+Cluster-wide pool changes are a client-side fan-out, not a gossiped desired
+state (pools are legitimately heterogeneous per node): `Client.SetPoolsCluster`
+PUTs the set to the entry node and every peer and returns a per-node result;
+retrying the failed nodes is the whole protocol, because the apply is an
+idempotent declarative replace. A non-nil error means peer discovery itself
+failed and the fan-out reached only the entry node — an incomplete apply to
+retry, kept distinct from a genuine single-node cluster (nil error). It fits a
+homogeneous cluster — a spec set that names an egress-lane pool is refused on a
+node without an attachment, so nodes that differ in capacity or attachment take
+a per-node `Client.SetPools`.
+
+### Cluster-invariant config
+
+Some config must match on every node or the cluster fails in confusing ways:
+
+| config | what breaks on mismatch |
+|---|---|
+| `api_token`, `tenants` | the SDK replays the authorizing token across a redirect; a peer missing that tenant/token answers 401 |
+| `preview_secret` | a preview URL signed on one node fails verification on another |
+| `mesh.cluster_key` | nodes cannot join / decrypt gossip at all |
+| `egress_ca` cluster root | a guest checkpointed/redirected across nodes trusts the root; a divergent root fails interception |
+
+Each node gossips a digest of these (HMAC-keyed by `cluster_key` when set;
+otherwise a token-free digest of tenant names + the CA root, so nothing
+brute-forceable rides cleartext gossip). A mismatch logs a warning at the moment
+the divergent node appears — not at the first unlucky redirect — and raises the
+`sandboxd_config_digest_mismatch` gauge. It is warn-only: a rolling credential
+rotation is a legitimate transient mismatch, so a divergence never partitions
+the mesh.
+
 ## Cluster checklist
 
 - memberlist port (e.g. 7946) open node-to-node, TCP **and** UDP
 - `advertise_addr` set to a routable address on every node (never loopback,
   never a wildcard)
-- same `api_token` and `tenants` set everywhere
+- same `api_token`, `tenants`, `preview_secret`, and `egress_ca` root everywhere
+  (a mismatch warns and shows in `sandboxd_config_digest_mismatch`)
 - `cluster_key` set if the gossip network is not otherwise trusted
+- pool changes via `Client.SetPoolsCluster` (or per-node `SetPools`); the applied
+  set persists to `pools.json` and survives restart

docs/deploy.md

@@ -66,7 +66,7 @@ sandboxd reads one JSON file (`-config`, default
 | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call restores it transparently (a checkpoint restore's latency). Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool |
 | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived, reclaiming storage; the claim is then gone for good. Same node-wide/per-pool split |
 | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node |
-| `pools[]` | — | warm pools. `warm` defaults to 4; `net` is `none` or `egress`; `size` is a tier, below. Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain |
+| `pools[]` | — | warm pools. `warm` defaults to 4; `net` is `none` or `egress`; `size` is a tier, below. Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain. This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. Egress stays config-owned either way. See [state ownership](cluster.md#state-ownership) |
 
 Size tiers (free-form CPU/memory is deliberately not accepted — it would
 fragment the warm pools):

sandboxd/config/config.go

@@ -2,9 +2,15 @@
 package config
 
 import (
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/hex"
+	"encoding/json"
 	"fmt"
 	"os"
 	"slices"
+	"strings"
 
 	"github.com/cocoonstack/sandbox/sandboxd/egress"
 	"github.com/cocoonstack/sandbox/sandboxd/store/s3"
@@ -204,6 +210,34 @@ func (c *Config) HasEgress() bool {
 	return c.Bridge != "" || c.Network != ""
 }
 
+// ClusterDigest fingerprints the must-match config so a divergent node is
+// caught early, not at a later 401. With cluster_key it HMACs token material
+// too; without one only tenant names + CA root ride the (maybe cleartext) wire.
+func (c *Config) ClusterDigest(caFingerprint string) string {
+	names := make([]string, len(c.Tenants))
+	for i, t := range c.Tenants {
+		names[i] = t.Name
+	}
+	slices.Sort(names)
+	if c.Mesh != nil && c.Mesh.ClusterKey != "" {
+		if key, err := base64.StdEncoding.DecodeString(c.Mesh.ClusterKey); err == nil {
+			type auth struct{ Name, Token string }
+			tenants := make([]auth, len(c.Tenants))
+			for i, t := range c.Tenants {
+				tenants[i] = auth{t.Name, t.Token}
+			}
+			slices.SortFunc(tenants, func(a, b auth) int { return strings.Compare(a.Name, b.Name) })
+			raw, _ := json.Marshal([]any{c.APIToken, c.PreviewSecret, caFingerprint, tenants})
+			mac := hmac.New(sha256.New, key)
+			mac.Write(raw)
+			return hex.EncodeToString(mac.Sum(nil))
+		}
+	}
+	raw, _ := json.Marshal([]any{caFingerprint, names})
+	sum := sha256.Sum256(raw)
+	return hex.EncodeToString(sum[:])
+}
+
 // guardsEgressLane: any tenant policy counts (claims mint egress keys without a
 // pool), a none-lane pool policy does not (it locks no tap).
 func (c *Config) guardsEgressLane() bool {

sandboxd/config/config_test.go

@@ -1,6 +1,7 @@
 package config
 
 import (
+	"encoding/base64"
 	"os"
 	"path/filepath"
 	"strings"
@@ -9,6 +10,30 @@ import (
 	"github.com/cocoonstack/sandbox/sandboxd/types"
 )
 
+func TestClusterDigest(t *testing.T) {
+	base := &Config{APIToken: "tok", PreviewSecret: "ps", Tenants: []TenantSpec{{Name: "acme", Token: "t1"}}}
+	d := base.ClusterDigest("ca-fp")
+	if base.ClusterDigest("ca-fp") != d {
+		t.Fatal("digest is not stable for identical config")
+	}
+	if (&Config{APIToken: "tok", PreviewSecret: "ps", Tenants: []TenantSpec{{Name: "beta", Token: "t1"}}}).ClusterDigest("ca-fp") == d {
+		t.Error("a tenant-name change is not reflected")
+	}
+	if base.ClusterDigest("other-fp") == d {
+		t.Error("an egress CA root change is not reflected")
+	}
+	// Keyless: token material must stay off the (maybe cleartext) wire.
+	if (&Config{APIToken: "other", PreviewSecret: "ps", Tenants: base.Tenants}).ClusterDigest("ca-fp") != d {
+		t.Error("api_token leaked into the keyless digest")
+	}
+	key := base64.StdEncoding.EncodeToString(make([]byte, 32))
+	keyed := &Config{APIToken: "tok", PreviewSecret: "ps", Tenants: base.Tenants, Mesh: &MeshConfig{ClusterKey: key}}
+	keyedDiff := &Config{APIToken: "other", PreviewSecret: "ps", Tenants: base.Tenants, Mesh: &MeshConfig{ClusterKey: key}}
+	if keyed.ClusterDigest("ca-fp") == keyedDiff.ClusterDigest("ca-fp") {
+		t.Error("with a cluster_key the api_token must be covered by the digest")
+	}
+}
+
 func TestLoadAppliesDefaults(t *testing.T) {
 	path := writeConfig(t, `{"pools":[{"template":"rt:24.04","net":"none","size":"small"}]}`)
 	cfg, err := Load(path)

sandboxd/main.go

@@ -82,7 +82,7 @@ func main() {
 
 	var placer server.Placer
 	if cfg.Mesh != nil {
-		msh, err := startMesh(cfg)
+		msh, err := startMesh(cfg, mgr)
 		if err != nil {
 			logger.Fatalf(ctx, err, "start mesh")
 		}
@@ -141,7 +141,7 @@ func main() {
 	logger.Info(ctx, "sandboxd stopped; VMs stay alive for the next reconcile")
 }
 
-func startMesh(cfg *config.Config) (*mesh.Mesh, error) {
+func startMesh(cfg *config.Config, mgr *pool.Manager) (*mesh.Mesh, error) {
 	mc := cfg.Mesh
 	mlCfg := memberlist.DefaultLANConfig()
 	host, portStr, err := net.SplitHostPort(mc.Bind)
@@ -167,10 +167,12 @@ func startMesh(cfg *config.Config) (*mesh.Mesh, error) {
 			return nil, fmt.Errorf("mesh cluster key: %w", err)
 		}
 	}
-	msh, err := mesh.New(mlCfg, nodeID, cfg.AdvertiseAddr, key)
+	msh, err := mesh.New(mlCfg, nodeID, cfg.AdvertiseAddr, key, cfg.DataDir)
 	if err != nil {
 		return nil, err
 	}
+	// Publish the config digest before Join, so the first gossip carries it.
+	msh.SetSelfDigest(cfg.ClusterDigest(mgr.EgressCAFingerprint()))
 	if err := msh.Join(mc.Join); err != nil {
 		_ = msh.Shutdown()
 		return nil, err

sandboxd/mesh/epoch.go

@@ -0,0 +1,29 @@
+package mesh
+
+import (
+	"math"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/cocoonstack/sandbox/sandboxd/utils"
+)
+
+// loadEpoch reads the persisted gossip epoch, or 0 when the file is absent,
+// unreadable, or above MaxInt64 — a UnixNano-derived counter cannot legitimately
+// exceed it, so a corrupt value falls back to the wall-clock seed.
+func loadEpoch(path string) uint64 {
+	raw, err := os.ReadFile(path) //nolint:gosec // node-local data-dir path
+	if err != nil {
+		return 0
+	}
+	n, err := strconv.ParseUint(strings.TrimSpace(string(raw)), 10, 64)
+	if err != nil || n > math.MaxInt64 {
+		return 0
+	}
+	return n
+}
+
+func storeEpoch(path string, epoch uint64) error {
+	return utils.WriteFileSync(path, []byte(strconv.FormatUint(epoch, 10)), 0o600)
+}