Move public changelog registry ownership to the scrubber Lambda

[cotti]

Why

Phase 1 of elastic/docs-eng-team#688. The public registry.json is produced client-side against the private bucket and pass-through copied by the scrubber Lambda, which makes it a log of upload operations rather than a description of the public bucket. Every known consistency gap follows from that: ad-hoc uploads leave it stale (the 2026-07-22 incident), --skip-etag-check re-uploads can never reach a stale public manifest, deletions orphan entries, failed scrubs leave phantom entries, and the recorded ETag is documented as useless.

What

The scrubber Lambda becomes the sole producer of the public registry.json, computed by reconciling against the current state of the buckets — registry = f(state), never f(event):

• Events are triggers; state decides. The handler ignores the event type: each distinct key in a batch gets one object-level reconcile against the private bucket (present → scrub current content and copy; absent → delete the public copy), followed by a post-write HEAD validation that redoes the work if the source changed mid-flight (closes the older-read-writes-last race between overlapping invocations). Each distinct group then gets one registry reconcile from the public listing.
• Group reconcile lists the group's prefix (/-delimited so verbatim branches don't sweep nested pools, paginated), reuses entries whose recorded ETag still matches, recomputes the rest from the scrubbed public YAMLs (amends always recomputed — their target depends on the parent), and writes back with optimistic concurrency (If-Match/If-None-Match: *, bounded jittered retries on 412/409, batch-item failure on exhaustion). An empty group's manifest is conditionally deleted — absent ≠ empty for consumers. A manifest with a newer schema_version is reported and left untouched.
• The registry pass-through is retired in the same deploy: registry-key events only schedule a group reconcile, so client-authored JSON no longer reaches the public bucket uninspected (strengthens the single-writer invariant from Changelog: Bundle upload support and Private-Public Bundling SQS Lambda #3163).
• producer field on the manifest identifies the reconcile algorithm version; any mismatch — including every legacy pass-through manifest at cutover — forces a full metadata recompute and a write even when entries come out identical. ETags recorded are now the public object's, finally usable by CDN consumers.
• The Lambda's top-level handler logic moved to testable classes in Elastic.Changelog (ScrubberProcessor, RegistryReconciler, ChangelogContentScrubber); Program.cs is a thin adapter. Per-invocation reconcile metrics are emitted as CloudWatch EMF (the docs-eng-team#688 Phase 0 observability item that could only land with the reconciler). The FakeS3 test fake (stateful, MD5 ETags, real conditional PUT/DELETE semantics, delimiter+pagination, interleaving hooks) is harvested from Add changelog registry state discovery and reconciliation commands #3717 per the disposition table in the issue.

The guarantee is convergence, not atomicity: any successfully processed event for a group repairs all accumulated drift in that group. Consumers' listed-but-missing tolerance stays. The CLI upload path is untouched in this phase (Phase 3 retires the client-side refresh); rollback = revert the Lambda deploy.

Requires the IAM grants from elastic/docs-infra#360 (s3:GetObject/s3:ListBucket on the public bucket) at deploy time.

Part of elastic/docs-eng-team#688 (Phase 1).

🤖 Generated with Claude Code