test(longhaul): add retention pruner to bound workload collection disk usage

[WentingWu666666]

What

Adds a count-based retention pruner to the long-haul test driver so the endurance write load no longer grows the DocumentDB collection (and its PVC) without bound.

Why

An infinite-duration write test on a fixed disk always eventually fills it. The westus2 long-haul cluster exhausted its 10Gi volume after ~24.9M writes, which put Postgres into a CrashLoopBackOff ("Not enough disk space") and tripped the scheduled Long Haul Monitor. This bounds steady-state disk use so it can't recur.

How

• Pruner (test/longhaul/workload/pruner.go): every 5m, per writer, deletes documents older than the most-recent LONGHAUL_RETAIN_PER_WRITER (default 2,000,000; 0 disables). Uses an index-backed range DeleteMany on the existing unique (writer_id, seq) index.
• No false data-loss (key invariant): the pruner deletes strictly below the verifier's confirmed floor, so it never removes a document the durability oracle hasn't already accounted for. The floor is published via atomics so the pruner reads it race-free.
• Restart-safe verifier: on startup the verifier seeds each writer's expected seq from the collection's current minimum seq, so a post-restart scan doesn't misread the pruned prefix as one enormous gap (which would be a false FAIL).
• Wiring: main.go starts the pruner only when RetainPerWriter > 0; each prune (and any freeze-on-failure) is recorded in the run journal; default set in the deployment ConfigMap.

Testing

• go build ./..., go vet ./..., gofmt -l clean
• go test ./... (all packages) + go test -race ./workload/... pass
• New unit tests: pruner logic incl. the "never delete at/above the floor" safety invariant, verifier ConfirmedFloor, config load/validate for the new env var

Notes

The already-running westus2 cluster was recovered separately by expanding its PVC to 64Gi; this PR prevents the refill from happening again. Opening as draft for review.

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: test, enhancement
Project fields suggested: Component test · Priority P1 · Effort L · Status In Progress
Confidence: 0.82 (mixed)

Reasoning

component from path globs (test); effort from diff stats (457+1 LOC, 10 files); LLM: Adds a retention pruner to the long-haul test workload to prevent unbounded disk growth that previously caused a real cluster outage (CrashLoopBackOff), touching multiple files across the test workload package including pruner logic, verifier restart safety, wiring, metrics, and unit tests.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

[Copilot]

Pull request overview

This PR adds count-based retention to the long-haul test workload so the write driver no longer grows the workload collection (and its backing PVC) without bound during infinite-duration runs.

Changes:

• Add a retention pruner that periodically deletes already-verified history beyond a per-writer retention window.
• Make the verifier restart-safe under pruning by seeding expected sequence from the collection’s current minimum seq and publishing a per-writer “confirmed floor” for safe pruning.
• Wire a new LONGHAUL_RETAIN_PER_WRITER configuration knob through docs, deployment defaults, env parsing, validation, and main startup logic.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/longhaul/workload/verifier.go	Publishes a per-writer verified floor (atomics) and seeds expected seq from DB-min on startup to avoid false gaps after pruning + restarts.
test/longhaul/workload/verifier_test.go	Adds unit tests for ConfirmedFloor behavior.
test/longhaul/workload/pruner.go	Introduces the periodic per-writer retention pruner (range DeleteMany based on verifier floor).
test/longhaul/workload/pruner_test.go	Adds unit tests for pruning boundaries and the “never delete at/above the floor” invariant.
test/longhaul/README.md	Documents LONGHAUL_RETAIN_PER_WRITER and its default/disable behavior.
test/longhaul/deploy/deployment.yaml	Sets a default retention window in the long-haul deployment ConfigMap.
test/longhaul/config/config.go	Adds retention env var constant, default, config field, env loading, and validation.
test/longhaul/config/config_test.go	Adds tests for retention env parsing and validation.
test/longhaul/cmd/longhaul/main.go	Starts the pruner when configured and passes the verifier as the floor provider.

[WentingWu666666]

Not a compile error — this is valid Go. Inside a composite literal whose element type is a pointer (*atomic.Int64), the &T is elided, so {} constructs &atomic.Int64{} (Go spec: Composite literals). go vet ./workload/... passes and the test binary compiles/runs green. Leaving as-is.

[WentingWu666666]

Same as above — {} in a map[string]*atomic.Int64 literal elides &atomic.Int64{} and compiles cleanly (per the Go spec on composite literals). Verified with go vet + go test ./workload/.... No change needed.

[WentingWu666666]

Good catch on the discrepancy — but the resolution is to fix the description, not the code. DocsPruned was intentionally removed during review (report row → snapshot field → counter) because it had no production reader once the report row was dropped; the pruner already journals every prune and any freeze-on-failure, which is the real observable. Updated the PR description to drop the stale DocsPruned claim.

[xgerman]

Summary

Reviewed against the actual verifier/pruner semantics (built, vetted, all package tests pass). The core design is genuinely good — floor-based deletion + freeze-on-data-loss is a safe, well-reasoned way to bound disk without ever touching the durability verdict:

Verified correct:

• Pruner only ever deletes seq <= confirmedFloor - retainPerWriter, i.e. strictly below what the verifier has already accounted for; the delete rides the existing (writer_id, seq) unique index (bounded range delete, not a scan). ✅
• Any gap/checksum failure freezes pruning permanently to preserve evidence; freeze is checked before pruning and HasDataLoss() is monotonic, and the verifier increments the loss counters before it publishes the advanced floor — so the pruner can never observe a floor past a gap without also seeing the freeze. ✅
• Concurrency is sound: floor is a read-only map[string]*atomic.Int64 built once at construction; only the atomics are Load/Store-d across goroutines. ✅
• The streaming auditor refactor is O(1) per cycle and now surfaces cursor.Err() — a latent truncated-read → false-tail-loss bug that the old slice path silently swallowed. ✅
• Config gating (RetainPerWriter==0 disables), default (2M docs/writer), and validation all look right. Pruner tests are thorough — boundaries, the never-delete-at-or-above-floor safety invariant, and all three freeze cases.

One blocking bug in the new startup seed, plus a test gap and a nit.

[xgerman]

🟠 Major — the one-time seed is marked done before it succeeds, so a transient error produces a false data-loss FAIL on a pruned+restarted collection.

v.seeded[writerID] = true is set unconditionally, before the minSeq query. If that query errors transiently at startup (network blip, majority-read timeout), the warning is logged but seeded stays true, so the seed is never retried. The very next scan then runs from expectedSeq = 1 against a collection whose pruned prefix no longer exists → the surviving minSeq document registers as one enormous internal gap → HasDataLoss() flips to FAIL for the rest of the run, and (ironically) that freezes the pruner.

This is exactly the restart-after-pruning path this PR introduces — before pruning, a scan-from-1 after restart was always correct, so the seed has to be robust to be safe.

Fix: only mark seeded on success, so a transient error retries next cycle (minSeq returning (0, nil) for an empty collection counts as success):

if !v.seeded[writerID] {
    if minSeq, err := v.minSeq(ctx, writerID); err != nil {
        v.journal.Warn("verifier", fmt.Sprintf("min-seq seed failed for writer %s (will retry): %v", writerID, err))
    } else {
        v.seeded[writerID] = true
        if minSeq > expectedSeq {
            expectedSeq = minSeq
        }
    }
}

[xgerman]

🟡 Minor — the startup min-seq seed has no test, which is precisely where the Major bug above lives. The suite covers ConfirmedFloor/publishFloor and the auditor, but not the restart seed. A test that pre-populates the collection starting at seq N, constructs a fresh verifier (empty nextSeq), runs one cycle, and asserts no gap is counted would exercise the happy path — plus an error-injection variant asserting the seed is retried (not permanently marked done) would lock in the fix.