Add changelog registry reconcile and verify commands - #3741
Open
cotti wants to merge 2 commits into
Open
Conversation
Phase 2 of elastic/docs-eng-team#688. The cutover/heal tooling for the Lambda-owned public registry: - `changelog registry reconcile` plans groups (one scope, or the union of both buckets so orphan public groups are covered) and sends one versioned, discriminated reconcile message per group to the scrubber queue — {kind, version, scope, group, correlation_id}, validated through ChangelogKeys on both ends. The CLI never mutates S3; the Lambda stays the public bucket's single writer. --dry-run prints the plan; the non-dry-run path asks for confirmation (--yes for CI). Each run stamps one correlation id and prints a ledger line per group. - On a reconcile message the Lambda performs a full group heal: object-level reconcile over the union of both buckets' listings (copy what's live, delete what isn't), then the group reconcile — recovering lost/DLQ-expired scrub events. Requires the new optional PRIVATE_BUCKET_NAME Lambda env var; malformed messages are rejected to the DLQ where the Phase 0 alarm surfaces them. - `changelog registry verify` is the read-only sibling and cutover gate: compares each public manifest against what a reconcile would write (same listing spec and entry rules by construction) and reports divergence as missing/stale/corrupt/object-divergent, with unsupported schemas reported distinctly. - Fix the manifest ETag wire format: the snake_case policy serialized the producer-side field as "e_tag" while consumers and the documented format read "etag" — recorded ETags were invisible to every consumer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same suppression RegistryBuilderTests carries: xUnit owns the test class lifetime and TestDiagnosticsCollector needs no disposal in these tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Phase 2 of elastic/docs-eng-team#688, stacked on #3738. Once the scrubber Lambda owns the public
registry.json(Phase 1), two things are still missing: a way to trigger the cutover heal (every existing public manifest is a legacy pass-through copy that needs one authoritative rebuild — and historical drift needs repairing retroactively), and a way to prove convergence before Phase 3 retires the client-side refresh.What
changelog registry reconcileplans the affected groups — a single scope, or the union of groups discovered in both buckets, so orphan public groups (including manifest-only leftovers) are covered — and sends one explicit reconcile message per group to the scrubber queue:{"kind":"reconcile","version":1,"scope":"bundle"|"changelog","group":"…","correlation_id":"…"}, strictly validated throughChangelogKeyson both ends and serialized with the same source-generated context the Lambda parses with (AOT). The CLI never mutates S3 — the Lambda stays the public bucket's single writer.--dry-runprints the plan; the non-dry-run path asks for confirmation (--yesfor non-interactive use). Each run stamps one correlation id and prints a ledger line (group, SQS message id, correlation id) for the cutover checkpoint. Attributes:[RequiresAuth],[CommandIntent(Intent.Destructive | Intent.Idempotent)],[MutationScope(MutationScope.Global)].republish). Reads the (already-provisioned)PRIVATE_BUCKET_NAMELambda env var — S3 events keep working without it since they carry their source bucket. Malformed reconcile messages are rejected to the DLQ, where the Phase 0 alarm makes the sender bug visible — redelivery can't fix a bad message.changelog registry verifyis the read-only sibling and the cutover gate: for each planned group it compares the public manifest against what a reconcile of the current public listing would write — same listing spec and entry rules by construction (RegistryReconciler.VerifyGroupAsync) — and reports divergence using Add changelog registry state discovery and reconciliation commands #3717's taxonomy:Missing/Stale/Corrupt/ObjectDivergent, withUnsupportedSchemareported distinctly. Non-zero exit on any divergence.ETagas"e_tag", while the consumer contract (ChangelogRegistryBundle) and the documented format read"etag"— every recorded ETag has been invisible to every consumer.[JsonPropertyName("etag")]aligns the writer with the documented format; old"e_tag"manifests now parse as corrupt and get rebuilt, which the cutover forces anyway via theproducermismatch.docs/cli-schema.jsonregenerated; supplemental CLI pages added underdocs/cli/changelog/registry/.Part of elastic/docs-eng-team#688 (Phase 2).
🤖 Generated with Claude Code