Archiving & Disaster Recovery: schema through base-backup generation (M1-M5) - #1186
Draft
dimitri wants to merge 32 commits into
Draft
Archiving & Disaster Recovery: schema through base-backup generation (M1-M5)#1186dimitri wants to merge 32 commits into
dimitri wants to merge 32 commits into
Conversation
…or API) Adds the SQL-only foundation for the Archiver process identity / ARCHIVING node membership / base-backup policy / PITR schema described in the archiving-disaster-recovery design doc, milestone 1 (schema + monitor API only -- no service_archiver process involved yet, everything is exercised via direct SQL calls against a plain cluster). - pgautofailover.replication_state gains a new 'archiving' terminal state. - pgautofailover.node gains haspgdata bool, distinguishing ordinary Postgres instances from lightweight ARCHIVING membership rows (a pg_receivewal client, no PGDATA). The old unconditional UNIQUE (nodehost, nodeport) constraint is replaced with a partial unique index scoped to haspgdata rows, since one archiver's (hostname, 0) pair is deliberately shared across every group it serves. - New types, tables and ~26 plpgsql/SQL functions covering: archiver registration and storage targets (local + rclone), formation/group archiver policy (quorum, base-backup policy, replication-quorum eligibility), WAL capture confirmation (wal_archived()/ report_wal_received()), base-backup lifecycle and pruning, warm-standby archiver_node rows with a maxresidentreplay cap, and PITR node lifecycle + command queue. - pgautofailover--2.2--2.3.sql mirrors the same DDL incrementally, since 2.3 hasn't shipped yet; verified end-to-end against a real 1.0 -> ... -> 2.2 -> 2.3 upgrade (including the pre-existing node_nodehost_nodeport_key1 constraint name quirk from two earlier migrations each recreating the table). - New archiving_schema regress test exercising the full schema end-to-end via direct SQL, added at the end of regress_schedule (after cluster_init_failover_rule_attribution, before the dummy_update/ drop_extension/upgrade trio that must stay last) since its expected output pins literal id values tied to its exact position in the shared contrib_regression database, same as every other test in this schedule. Full local regress (20/20) + isolation (6/6) schedules pass, plus a verified real extension upgrade from 2.2 to 2.3.
Adds monitor-side (SQL FSM + C) support for the ARCHIVING replication state, so an ARCHIVING node row (haspgdata = false, created by M1's archiver_add_formation()) is driven through the same node_active() protocol as an ordinary node instead of being stuck at wait_standby forever. No keeper-side/service_archiver work yet -- this is groundwork, verified via node_active() calls made directly against the monitor. - ReplicationState gains REPLICATION_STATE_ARCHIVING (C) / 'archiving' already existed on the SQL enum from M1. - AutoFailoverNode gains hasPgData, populated via TupleToAutoFailoverNode. Looked up by name (SPI_fnumber), not the file's usual hardcoded Anum_ constant: this function is also called against a "RETURNING node.*" tuple descriptor whose physical column order diverges from the explicit SELECT list's logical order once pg_versionnum/pg_version/ pg_versionstring/citus_version are in the mix, so a hardcoded ordinal would silently read the wrong (and wrongly-typed) column for that caller. - MonitorFSM[]: pos 307/309/315/317/319 (report_lsn/wait_standby, primary converged -> secondary/catchingup) gain an explicit hasPgData = TRUE restriction, paired with 5 new hasPgData = FALSE mirror rows (pos 394-398) assigning ARCHIVING instead -- appended after the existing MS-failover cluster since the ordinary rows are numbered with no room between them for 5 more, and the hasPgData split makes their relative order irrelevant to first-match-wins. Pos 367's MS-failover fan-out row (and BuildCandidateList's own C-side secondaryStates list) now also admits ARCHIVING, pulling it into report_lsn during elections exactly like SECONDARY/CATCHINGUP. - system_identifier_is_null_at_init_only loosened to also allow a NULL sysidentifier while reportedstate is 'archiving' or 'report_lsn': an ARCHIVING row never gets a real one. The 2.2--2.3 migration mirror casts the column to text instead of the literals to the enum, since this script's own earlier ADD VALUE 'archiving' and this constraint run in the same ALTER EXTENSION UPDATE transaction and Postgres refuses to create new instances of a not-yet-committed enum value. - keeper_fsm_edges.sql's own "expect zero rows" comment updated: 8 rows are now expected there, a real and currently correct gap -- the monitor side landed first, with no service_archiver/KeeperFSM[] support yet to report ARCHIVING or drive pg_receivewal (next milestone). Verified against a hand-run node_active() scenario (register primary + secondary, converge to primary/secondary, attach an archiver, confirm wait_standby -> archiving instead of catchingup, steady-state archiving stays archiving, replication_quorum = true fans out apply_settings to the primary exactly like an ordinary quorum standby, and rule_pos attribution points at the new rows) in addition to the full regress (20/20) + isolation (6/6) suites and a real 2.2 -> 2.3 extension upgrade.
Adds the keeper-side counterpart to the monitor-side ARCHIVING FSM support (previous commit): KeeperFSM[] rows for WAIT_STANDBY->ARCHIVING/ARCHIVING->REPORT_LSN/REPORT_LSN->ARCHIVING, each dispatching to a new, archiver-specific transition function (fsm_init_archiver/fsm_archiver_report_lsn/fsm_archiver_follow_new_primary, fsm_transition.c) -- mirroring the existing NODE_KIND_CITUS_* pattern of adding separate functions per node kind rather than branching inside the shared ones (fsm_init_standby, keeper_update_pg_state, keeper_ensure_current_state, keeper_node_active_loop stay untouched). New service_archiver.c launches and tracks the one pg_receivewal child an ARCHIVING node keeps running against its group's current primary -- milestone 2's own "colocated fast path" scope (see the Build order in ~/dev/temp/archiving-disaster-recovery.md): pg_receivewal is a real, unmodified Postgres client talking straight to the real primary's own walsender, so no new wire protocol is needed here at all. Not yet wired into supervisor.c's Service/RestartPolicy machinery or a replication slot -- both noted as follow-ups in that file's own header comment. Also: ARCHIVING_STATE added to NodeState (state.h/state.c) and to nodestate_utils.c's nodestateConnectionType() switch (grouped with the other "Postgres known to be stopped" states, since an ARCHIVING row never has a postmaster of its own -- this switch has no default case by design, so a missing case here would have failed the build). Verified: full regress (20/20) + isolation (6/6) suites still pass, and the monitor/keeper reachability cross-check in keeper_fsm_edges.sql -- which the previous commit deliberately left showing 8 unresolved rows, documented as this milestone's own known gap -- now shows zero rows again, both directions, confirming the two sides agree. Live-checked via `pg_autoctl inspect fsm list --json`, which is also how keeper_fsm_edges.json was regenerated (pretty-printed to match the existing file's own review-friendly formatting, not the CLI's compact default). citus_indent and ci/banned.h.sh both pass (the latter caught a raw strerror()/fprintf(stderr) call in service_archiver.c's own exec- failure path, fixed to the project's own log_fatal(..., "%m") convention already used at the other execv() call sites in this codebase).
… (M2 continued)
Adds NODE_KIND_ARCHIVER as a real PgInstanceKind (pgsetup.h/pgsetup.c,
name<->enum both directions), two monitor RPC wrapper functions
(monitor_register_archiver/monitor_archiver_add_formation, monitor.c,
calling M1's own register_archiver()/archiver_add_formation() plpgsql
functions -- not the ordinary C register_node() RPC, since an Archiver is
a process identity, not a (formation, group) membership by itself), and
`pg_autoctl create archiver` (cli_create_node.c): a deliberately minimal,
hand-rolled getopts (not the shared cli_create_node_getopts every ordinary
node kind uses, since that parser's defaults assume a real PostgresSetup
an archiver never has) that registers with the monitor, writes a
KeeperConfig + initial state file (WAIT_STANDBY_STATE, mirroring
archiver_add_formation()'s own starting point), and with --run hands off
to service_archiver_loop() (previous commit).
Verified live against the real monitor RPC layer (not just static
review): registration, formation attachment, and config/state file
writing all confirmed end-to-end against a real running monitor extension
instance, including two real bugs the empirical run caught that manual
review missed --
- config_find_pg_ctl() unconditionally clears pgSetup.pg_ctl before
searching, silently discarding a caller-supplied --pgctl value; fixed
by only calling it when pg_ctl is still empty (also added the missing
--pgctl flag itself -- this dev machine has two pg_ctl on PATH and
needs it to disambiguate, a real scenario, not a test-only one).
- keeper_config_write_file() requires pg_autoctl.role set (validated
against KEEPER_ROLE, not defaulted on write) -- config.role was never
populated, since this path deliberately skips keeper_config_init()'s
ordinary defaults (Postgres-specific probing that doesn't apply here).
`--run`'s actual pg_receivewal launch is still unverified against a real
streaming primary -- needs a real replication-configured Postgres pair,
which is exactly what the next step (pgaftest wiring) provides.
citus_indent and ci/banned.h.sh both pass.
autoctl_node was only ever granted EXECUTE on the function, never SELECT on pgautofailover.basebackup itself, matching every other autoctl_node- callable helper that reads a table it has no direct grant on (e.g. archiver_add_formation) -- get_latest_basebackup was the odd one out. Found via a real end-to-end test of `pg_autoctl archiver serve` against a live monitor: calling it as autoctl_node failed with "permission denied for table basebackup".
The archiver's serving half: a standalone binary (no pg_autoctl/*.c
dependency, only src/bin/common/ and src/bin/lib/log/) that speaks enough
of the real Postgres replication protocol to serve IDENTIFY_SYSTEM, SHOW,
BASE_BACKUP, and a non-standard FETCH_FILE side-channel, backed by an
archiver's local WAL cache and base backups instead of a live postmaster.
No frontend-linkable server-side protocol library exists anywhere in
Postgres (confirmed against pqcomm.c/backend_startup.c/repl_gram.y/
walsender.c, all backend-only) -- this is a genuine reimplementation
guided by that source, not a linking exercise. Two pieces are vendored
near-verbatim since they're already frontend-safe: vendor/tar.c + pgtar.h
(PostgreSQL's own ustar header/checksum logic, src/port/tar.c).
Also ships fetch_client.c / `pg_walsender fetch-file`, the client side of
the FETCH_FILE side-channel, for use as pg_autoctl's own restore_command.
Verified against real, unmodified PostgreSQL client tools:
- psql (replication=1): IDENTIFY_SYSTEM, SHOW wal_segment_size
- pg_basebackup --format=plain -X none --no-manifest: fetched a real
base backup byte-identical to the source, then booted a live Postgres
instance from the result
- pg_walsender fetch-file: fetched a full 16MB WAL segment byte-
identical, plus clean error handling (missing file, path traversal,
unknown route)
See ~/dev/temp/archiving-disaster-recovery.md for the design this
implements milestone 2 of.
`pg_autoctl archiver serve` (cli_archiver.c) is the supervisor verb that
execs pg_walsender as a persistent child (service_archiver_serve.c),
mirroring exactly how service_archiver.c already execs real pg_receivewal
for the outbound WAL-capture direction -- same pattern, new direction.
Keeps pg_walsender's routes file ("<formation>/<group>" -> { walcache,
basebackup }) current, refreshed periodically and on SIGHUP.
The routes file is built from *local* config (formation/groupId/pgSetup.
pgdata), not a monitor round-trip: archiver_add_formation()'s own SQL
inserts the new archiver_node row's pgdata as an empty string, since the
monitor has no way to know an archiver's local WAL cache path -- that's
inherently archiver-host-local information. The one genuinely monitor-
tracked piece is the latest base backup's storage location
(monitor_get_latest_basebackup_location, new in monitor.c).
KeeperConfig gains archiverId/archiverIdStr (keeper_config.h/.c) so a
later, separate `archiver serve` invocation can identify itself to the
monitor -- ini_file.c's INI_INT_T is a plain int, too narrow for a
bigserial id, so this follows citusRoleStr/citusRole's existing string-
plus-parsed-value pattern in the same struct.
Verified against a real, freshly-created cluster (create monitor ->
create postgres -> create archiver -> archiver serve): archiverId
persists and round-trips correctly, the routes file is generated
correctly from live monitor state, pg_walsender starts and serves real
clients through it, and SIGTERM shuts the whole thing down cleanly.
…(M2c)
Completes milestone 2's command surface:
- TIMELINE_HISTORY <tli>: serves a "<tli>.history" file straight out of
the WAL cache directory (RowDescription/DataRow, no COPY involved --
traced from walsender.c's own SendTimeLineHistory()).
- CREATE_REPLICATION_SLOT / READ_REPLICATION_SLOT (physical only): a
slot is a bookkeeping marker file under the WAL cache directory, not
a real Postgres slot -- there's no live server to hold one. Not yet
wired into WAL-retention enforcement (prune_archiver_wal()'s job).
- START_REPLICATION [SLOT <name>] <lsn> TIMELINE <tli>: streams raw WAL
bytes straight from the WAL cache directory. Deliberately does NOT
vendor xlogreader.c: real walsender's own WalSndSegmentOpen just opens
a path computed from TLI+segno and streams bytes -- no WAL *record*
decoding is needed to serve a byte range, only the offset/segment
bookkeeping this file does directly. Handles the actively-growing
(".partial") segment case by polling, matching pg_receivewal's own
producer on the other end of this same protocol.
- wal_dir_scan.c: shared helper -- finds the newest fully-captured WAL
segment and derives its boundary LSN from the filename (XLogFileName
format, fixed 16MB segments). Used by START_REPLICATION's default
position, CREATE_REPLICATION_SLOT's consistent_point, and improves
IDENTIFY_SYSTEM's xlogpos (previously a "0/0" placeholder).
One correctness fix alongside: IDENTIFY_SYSTEM's dbname column must be
NULL for a plain replication=1/true connection (pg_receivewal's style) --
only replication=database (pg_basebackup's style) gets a real dbname back.
Always returning a value broke real pg_receivewal outright ("replication
connection using slot ... is unexpectedly database specific"), caught by
this milestone's own end-to-end testing, not by any narrower unit check.
Verified against real, unmodified PostgreSQL client tools:
- psql: TIMELINE_HISTORY round-trips real file content
- psql: CREATE_REPLICATION_SLOT / READ_REPLICATION_SLOT round-trip
consistent_point/restart_lsn correctly, including the "slot doesn't
exist" all-NULL-row case
- pg_receivewal -S <slot> --endpos=...: streamed a full 16MB WAL segment
byte-identical to the source via START_REPLICATION
(and `pg_autoctl stop`) now supervises an archiver's two
halves together -- WAL capture (service_archiver.c's service_archiver_loop,
outbound pg_receivewal against the primary) and serving (service_archiver_
serve.c's service_archiver_serve_loop, inbound pg_walsender) -- as two real
supervisor.c Service[] entries under one restart-on-crash process tree, the
same way start_keeper() already supervises postgres + node-active together
for an ordinary node (service_archiver_run.c). Dispatched from cli_service.
c's cli_keeper_run(), which already reaches an archiver's config file via
the existing role=keeper path; branches on nodeKind before the Postgres-
instance-specific local_postgres_init()/start_keeper() calls, which don't
apply to an archiver.
Two real bugs surfaced by actually running the full archiver process tree
end-to-end for the first time this session (service_archiver_loop's own
monitor-reporting loop was never previously exercised against a live
monitor for more than a few ticks):
- keeper->postgres.currentLSN was never initialized for an archiver (it
has no real Postgres instance to query it from, so keeper_update_pg_
state() -- the only place that ever set it -- is never called on this
path). node_active()'s own pg_lsn parameter rejected the resulting
empty string outright. Fixed by seeding it to "0/0" once, matching
keeper_update_pg_state()'s own placeholder before a real reading
exists; an archiver's actual capture progress is tracked separately
via archiver_wal, not through this per-node report.
- An ordinary node's own get_other_nodes()/current_state listings now
legitimately include ARCHIVING rows with nodeport = 0 (a deliberate
sentinel, see archiver_add_formation()'s own SQL comment: no
postmaster to be reachable on) -- but monitor.c's node-parsing helpers
treated a parsed port of exactly 0 as an unconditional error, so any
ordinary primary/secondary in a formation with an archiver attached
would fail its own node-active loop entirely. Relaxed the two multi-
node-listing parsers to only reject a genuine parse failure, not the
value 0 itself; left the single-node lookup (which can never
legitimately return an archiver, candidate_priority = 0 excludes it)
unchanged.
Verified against a real, freshly-created cluster (monitor + primary +
archiver): `pg_autoctl run --pgdata archiver1` starts both services
cleanly, the FSM transitions wait_standby -> archiving and real
pg_receivewal starts against the primary, pg_walsender serves real
clients through it, and `pg_autoctl stop` cascades a graceful shutdown
through both services and their own child processes (pg_walsender,
pg_receivewal) with no orphans left behind. Also re-verified the primary
node's own node-active loop, previously broken by the port=0 regression,
now runs cleanly with an archiver attached to its formation.
service_archiver.c gains service_archiver_report_captured_wal(), called once per node_active tick from service_archiver_loop(): it scans the archiver's local WAL cache directory for segments pg_receivewal has completed (i.e. no longer ".partial") since the last one reported, and calls the new monitor_report_wal_received() (monitor.c/.h) -- a thin wrapper around the already-existing pgautofailover.report_wal_received() SQL function -- for each one. This is what actually populates archiver_wal and makes wal_archived() return true; until now nothing in the codebase ever called that SQL function. Also fixes a liveness gap this uncovered: pg_receivewal was only ever (re)started from the FSM transition functions that move a node *into* ARCHIVING_STATE (fsm_init_archiver, fsm_archiver_follow_new_primary). An archiver process restarted while already ARCHIVING (or one whose pg_receivewal child died on its own) had nothing to bring it back up, despite this file's own header comment already describing that as the design. service_archiver_loop() now checks service_archiver_pgreceivewal_is_running() every tick and restarts it when needed, exactly matching that comment. Verified end-to-end against a real monitor + primary + archiver: forced WAL switches on the primary, confirmed archiver_wal gets populated with the correct end-of-segment LSNs and wal_archived() correctly reflects archiver_quorum, confirmed the liveness restart itself by killing and restarting the archiver process while already ARCHIVING. Full SQL regression schedule (src/monitor, 20/20) still passes. Dockerfile: copy pg_walsender into the "run" stage image alongside pg_autoctl -- needed for any archiver node in a pgaftest Docker environment, and a prerequisite for M4's own pgaftest specs.
Adds the "archiver" node kind to pgaftest's own DSL, needed to write any
.pgaf spec that includes an ARCHIVING node:
- test_spec_scan.l/.y: new "archiver" keyword (T_ARCHIVER), usable the
same way "coordinator"/"worker" already are: `archiver1 archiver`
inside a formation{} block.
- compose_gen.c: writes kind = archiver into the node's .ini, and links
service_archiver.c into pgaftest's own SHARED_SRCS (Makefile) so the
binary can drive an archiver node the same way it already drives
postgres/coordinator/worker ones.
- nodespec.c (pg_autoctl, not pgaftest): teaches `pg_autoctl node run
<node.ini>` -- the one command every pgaftest container actually
execs -- to recognize kind = archiver and build the right `pg_autoctl
create archiver` argv. An archiver's own getopts is deliberately
minimal (no --pgport/--ssl-*/--auth/...), so it gets its own argv
branch rather than falling through into the generic postgres-flags
path every other kind shares.
Also fixes a real bug in cli_indent.c's print_node() found while
writing the first archiver spec and round-tripping it through `pgaftest
indent`: the node-kind-to-keyword switch only had cases for coordinator
and worker, so indenting a spec containing an archiver node silently
dropped the "archiver" keyword on write-back, turning it into a plain
postgres node. Added the missing NODE_KIND_ARCHIVER case.
test_spec_parse.c/.h and test_spec_scan.c are bison/flex output,
regenerated from the .y/.l changes above.
First pgaftest spec exercising an ARCHIVING node, covering the two things Milestone 4 adds: - test_001/test_002: forcing WAL segment switches on the primary gets each completed segment reported to the monitor (service_archiver_report_captured_wal(), service_archiver.c) and reflected by pgautofailover.wal_archived() -- the archive_command confirmation check nothing populated before this milestone. - test_002 also exercises the liveness fix in service_archiver_loop(): killing and restarting the archiver process while it's already ARCHIVING must bring pg_receivewal back up on its own, not just on the FSM transition that first enters that state. - test_003: fails node1 over to node2 and confirms the archiver passes through REPORT_LSN_STATE and back to ARCHIVING_STATE (following the new primary), and that segments recorded before the failover are still there afterwards. Segment filenames are asserted directly (a fresh cluster deterministically starts WAL at 000000010000000000000001, and each pg_switch_wal() on an idle test database advances exactly one segment) since autoctl_node has no direct SELECT on archiver_wal -- wal_archived() is the only accessor it can call. Registered in tests/tap/schedule and tests/tap/schedules/node.sch, alongside the other node-lifecycle/FSM specs.
First half of Milestone 5 per the design doc's own Build order ("live
first, then replay/volatile"). New file service_archiver_basebackup.c
adds service_archiver_maybe_generate_basebackup(), called once per tick
from service_archiver_loop() alongside the M4 WAL-report/liveness calls.
Trigger scope for this pass: bootstrap only -- a group with zero
existing base backups (monitor_get_latest_basebackup_location() reports
not-found) gets one immediately. Scheduled/timeline-change/retention
triggers need basebackup_policy wired through the CLI first, a later
milestone.
Target selection follows the design doc's `live` precedence, minus its
warm-standby tier (nothing to select from yet, also later): the first
healthy secondary in the group (monitor_get_nodes(), skipping port == 0
ARCHIVING rows), falling back to the primary when none exists.
Generation itself is a one-shot forked child (basebackupPid, tracked the
same way service_archiver.c tracks pgReceivewalPid) rather than a
persistent service, so a potentially long-running pg_basebackup can't
stall the main loop's own node_active()/WAL-report tick. The child execs
the real, unmodified pg_basebackup client with --wal-method=none -- this
backup is deliberately not self-consistent on its own, since the
archiver's already-running WAL capture is what supplies the WAL needed
to reach consistency on replay -- then reads the resulting backup_label
for the authoritative start LSN/timeline and reports both start and
completion to the monitor via two new wrappers,
monitor_report_basebackup_started()/_completed() (monitor.c/.h), calling
the SQL functions M1's schema already shipped but nothing had called
yet. endlsn is best-effort: a live read of the source's current WAL (or
last-replayed, if the source is a standby) position right after the
backup finishes: not Postgres's own internal stop-backup LSN (not
observable from a plain CLI wrapper around pg_basebackup), but a
reasonable upper bound, and never fatal to the backup itself if that one
query fails.
Verified end-to-end against a real monitor + primary + archiver: the
bootstrap backup fires automatically, archiver_wal / basebackup rows
land correctly (source = 'live', status = 'complete', a real endlsn
distinct from startlsn), and the resulting directory passes
pg_verifybackup. Full SQL regression schedule (src/monitor, 20/20)
still passes.
Second half of Milestone 5 ("live first, then replay/volatile" per the
design doc's Build order). service_archiver_maybe_generate_basebackup()
now takes a bootstrap `live` backup as before, then -- on the very next
tick -- exercises the `replay`/`volatile` pipeline exactly once: extract
the last retained backup into a throwaway staging directory, point its
recovery at this archiver's own already-captured WAL (restore_command +
recovery.signal, entirely local, no network round trip), let it replay
forward and promote once it runs out of locally-captured segments, then
pg_basebackup it over loopback and discard the staging instance --
'volatile' means nothing survives between cycles.
Real frequency-driven scheduling (basebackup_policy's own frequency/
onpromotion/retention, resolved through get_archiver_policy()/
get_basebackup_policy()) is a deliberate follow-up, not built here: the
milestone-defining new capability is the replay mechanism itself, not a
general scheduler (matching the design doc's own build order, which
lists warm standby's scheduling machinery as a later milestone).
monitor_report_basebackup_started() (added in the `live`-only commit)
now takes real source/replaymode parameters instead of a hardcoded
'live', and monitor_get_latest_basebackup_location() is renamed to
monitor_get_latest_basebackup_info() and returns the latest backup's
source alongside its storage location -- what the trigger above uses to
tell "only the bootstrap has run" from "the replay exercise is already
done".
Getting a working staging instance up took two real, load-bearing fixes
along the way:
- pg_ctl start, invoked here through both a hand-rolled fork()/execl()
and this project's own run_program() helper, reproducibly misparsed
its own arguments in this exact process tree (deep in a supervised
archiver's own fork chain) even though byte-identical argv worked
fine in every standalone reproduction attempted. Root cause not
fully isolated; worked around by execing the real "postgres" binary
directly instead of going through pg_ctl at all -- the same
fork()/execv() pattern already used for pg_receivewal and
pg_basebackup in this codebase, with readiness confirmed by polling
a real SQL connection rather than relying on pg_ctl's own "-w".
- recovery_target_lsn set to "the end of the latest complete segment"
is not actually a reachable record boundary on a mostly-idle source
(a renamed, "complete" segment file is always its full fixed size
regardless of how much of it is real WAL) -- recovery correctly
refused to pause there ("recovery ended before configured recovery
target was reached"). Replaying to "everything locally available"
and letting Postgres promote on its own sidesteps needing a precise
target at all, which a `volatile`, discard-after-use snapshot never
actually needed in the first place.
Verified end-to-end against a real monitor + primary + archiver, from a
cold start through both the live bootstrap and the replay follow-up:
basebackup rows land correctly (source/replaymode/status all correct,
real distinct startlsn/endlsn across the sequence), and both resulting
directories pass pg_verifybackup. Full SQL regression schedule
(src/monitor) passed 20/20 twice earlier against this same unchanged
schema in this session; a later re-run hit an apparent local
pg_regress/DROP DATABASE environment hang (ProcSignalBarrier) unrelated
to any change in this commit -- no .sql files are touched here.
pgaftest coverage for Milestone 5's own base backup generation (both live and replay/volatile): brings up a monitor + primary + archiver, then waits for both the bootstrap live backup and the one-time replay/volatile follow-up to land, checking the group's final pgautofailover.get_latest_basebackup() row (source = 'replay', replaymode = 'volatile', status = 'complete'). No explicit trigger step is needed here, unlike archiver_wal_capture.pgaf's pg_switch_wal() calls -- both backups fire on their own within a couple of service_archiver_loop() ticks of the archiver starting. That also makes the intermediate 'live'-only state unsafe to assert on directly (this pass's own trigger logic produces at most one live and one replay backup before going quiet for the group, a couple of ticks apart, with nothing in this spec's control over exactly when to look) -- only the final state, once both have landed, is deterministic. Registered in tests/tap/schedule and tests/tap/schedules/node.sch, alongside archiver_wal_capture.pgaf.
New "Archiving & Disaster Recovery Architecture" section in intro.rst, between "Single Standby Architecture" and "Multiple Standby Architecture" -- an archiver is orthogonal to standby count, so it reads best as the thing you add on top of the simplest case before the doc branches into standby-count variations. New docs/tikz/arch-archiver.tex, rendered to .svg the same way every other architecture diagram in this directory is (latexmk -lualatex + pdftocairo, verified locally): primary + secondary + archiver, with the archiver's own WAL cache / base backups called out, a distinct WAL streaming (pg_receivewal) edge separate from real streaming replication, and the monitor's health-check/WAL-report edges to all three. common.tex gains one new color pair (abox/atxt, MS amber) and one new edge style (wal, dashed) for the archiver box and its WAL-streaming edge -- deliberately not reusing the primary/standby colors, since an archiver is a different kind of entity, not a replica. Terminology: uses "archiver" for the physical entity and "archiving node" for its per-group FSM membership, per-project decision -- avoids colliding with pgautofailover.archiver_node, the broader schema table that also covers warm-standby/pitr instances which don't participate in elections at all.
keeper->postgres.currentLSN was set to "0/0" once at service_archiver_ loop() startup and never updated again -- an archiving node's own reportedlsn in pgautofailover.node stayed at that placeholder forever, no matter how much WAL it had actually captured. This mattered more than it looked: pgautofailover.get_most_advanced_ standby() -- the query fast-forward uses to pick a WAL source during a failover election -- has no kind-based exclusion at all, and an archiving node already passes through REPORT_LSN_STATE during an election exactly like any other node (ARCHIVING_STATE -> REPORT_LSN_STATE, fsm.c). A "0/0" reportedlsn was the only thing keeping an archiver from ever being ranked as a candidate WAL source. service_archiver_update_current_lsn() now scans the local WAL cache for the newest complete segment each tick and updates currentLSN to its end LSN before keeper_node_active() reports it -- verified against a real cluster: after two pg_switch_wal() calls, the archiver's own pgautofailover.node.reportedlsn row tracks the primary's position almost exactly (0/A000000 vs. the primary's own 0/A000060).
…m an archiver
Confirms (and builds out) the reframing from the previous commit: an
archiving node already passes through REPORT_LSN_STATE during elections
and get_most_advanced_standby() has no kind-based exclusion, so once its
currentLSN is real, fast-forward's existing streaming-replication code
path can already select and target one -- no new restore_command
plumbing needed. Four real gaps stood between that and actually working,
found and fixed by testing a genuine, unmodified Postgres standby against
a real archiver end to end (not just pg_receivewal, which never exercises
any of these):
- pg_walsender routing is dbname-based (formation/group as dbname), but a
real standby's own walreceiver never forwards the operator's dbname for
a physical replication connection -- it always sends the literal
"replication", confirmed against a real standby. accept_loop.c now
falls back to the single configured route when it sees that sentinel,
matching this milestone's own one-membership-per-archiver scope; a
multi-route archiver (later milestone) needs a different mechanism
(e.g. application_name, which real walreceiver does forward).
- IDENTIFY_SYSTEM's systemid always fell back to the "unknown" placeholder
"0" because nothing ever populated route->systemId: service_archiver_
serve.c's own routes-file writer never wrote a systemid key, even
though routes.c already knew how to parse one. A real standby rejects
a mismatched system identifier outright ("database system identifier
differs between the primary and standby"). Fixed with a new monitor
RPC, monitor_get_group_system_identifier() (pgautofailover.
get_group_system_identifier(), new SQL function in both
pgautofailover.sql and the 2.2--2.3 migration -- an archiving node has
no sysidentifier of its own, but every other node in its group shares
the same one), wired into the routes-file refresh.
- cmd_start_replication.c read raw fread() bytes from a ".partial"
segment without knowing where pg_receivewal's actually-written data
ends -- pg_receivewal pre-allocates the full segment size up front
(matching real Postgres's own WAL file pre-allocation), so reading past
the real tail returns zeros indistinguishable from real content at the
byte level. Sending that tail as WAL data is exactly what a real
standby's recovery logic flags as "invalid record length ... got 0",
and on seeing it, terminates its own walreceiver outright rather than
treating it as "nothing new yet, retry" -- with no automatic
reconnection afterward. Fixed by trimming any trailing zero run before
ever sending a ".partial" chunk (self-correcting: an in-progress
boundary just gets re-read next tick instead of shipped early).
- get_most_advanced_standby() returns an ARCHIVING row's real nodeport,
which is the port == 0 sentinel (no postmaster of its own), not the
archiver's actual pg_walsender serve port -- the monitor has no column
for that (archiver-host-local information, same reasoning service_
archiver_serve.c's own routes file exists for). keeper_get_most_
advanced_standby() now resolves a port == 0 candidate to
PG_AUTOCTL_ARCHIVER_SERVE_PORT, matching this milestone's single-
well-known-port scope.
Verified end-to-end: a real pg_basebackup-seeded standby, given nothing
but an ordinary primary_conninfo pointing at the archiver's serve port,
completed backup recovery, reached consistent recovery state, streamed
live via START_REPLICATION, stayed connected indefinitely (pg_stat_wal_
receiver: status = streaming), and correctly applied newly-written data
(a table created and populated on the real primary afterward) -- with
zero restore_command, zero new replication-source machinery, and zero
changes to fsm_fast_forward's own selection logic beyond the port fix
above.
Bootstraps a brand new node from a registered archiver's base backup
plus captured WAL instead of the group's live primary -- the disaster-
recovery case: rebuild after every live standby (or even the primary)
is gone, with only the archiver left standing. Verified end to end
against a real cluster: `create postgres --from-archiver` completed
pg_basebackup from the archiver, replayed WAL, and settled into a
genuinely healthy "secondary" (pg_stat_wal_receiver: status =
streaming), matching reportedlsn against the real primary once it
re-parented there.
New plumbing:
- KeeperConfig.fromArchiver (keeper_config.h) plus the `--from-archiver`
CLI flag on `create postgres` (cli_create_node.c, cli_common.c) --
runtime-only, same as createAndRun, since reach_initial_state() runs
in the same `pg_autoctl create` invocation that parses it.
- pgautofailover.get_archiver_node() (pgautofailover.sql, the 2.2--2.3
migration) plus its monitor_get_archiver_node()/keeper_get_archiver_
node() C wrappers (monitor.c, keeper.c): finds the ARCHIVING row for
(formation, group) directly. Deliberately not get_most_advanced_
standby() -- that function filters on reportedstate = 'report_lsn', a
transient state an archiving node only visits during a FAST_FORWARD
election, never during its normal steady-state 'archiving' operation,
so it can never find an idle archiver outside of an election.
- fsm_init_standby() (fsm_transition.c) branches on config->fromArchiver
to resolve the archiver via the above instead of keeper_get_primary(),
and passes an empty replication slot name -- pg_walsender has no
slot-based retention in this milestone (cmd_start_replication.c's own
header comment), so standby_init_database's pre-flight replication-
slot check must be skipped rather than asked to verify a slot that
will never exist, matching that function's own existing "initialising
from another standby, no primary yet" precedent.
Four further real, narrow gaps stood between that and actually working,
each found by running the real `pg_basebackup`/`pg_autoctl` code paths
end to end rather than by inspection:
- pg_walsender's BASE_BACKUP had no manifest support (documented scope
cut, cmd_base_backup.c), but PG13+ pg_basebackup requests one by
default -- ReplicationSource.noManifest (pgsql.h) plus pg_basebackup()
passing --no-manifest when set (pgctl.c) works around it for an
archiver-sourced clone specifically, without touching a real primary's
own backup path.
- pgctl_identify_system() (pgctl.c) built its replication connection
string with no dbname at all, relying on real pg_basebackup's and
real walreceiver's own respective "default unset dbname to the literal
'replication'" behaviors -- neither of which this is: it's pg_auto_
failover's own raw libpq connection, which has no such default and
instead falls back to plain libpq's own "dbname = username" rule
(fe-connect.c), a route pg_walsender's routes file was never going to
have an entry for. Passing "replication" explicitly matches what every
other replication client already sends on the wire, and is a no-op
against a real primary (which ignores dbname for replication=true
connections regardless).
- A "replay" base backup (basebackup_replay_mode, milestone 5) promotes
a throwaway extracted copy to make it self-consistent, which genuinely
puts it on a *later* timeline than whatever the archiver's own walcache
has actually captured (which only ever advances on the real primary's
timeline) -- serving that pairing breaks a real pg_basebackup's own
timeline consistency check once it reaches its background WAL-streaming
step ("starting timeline N is not present in the server", comparing
the backup's own timeline against IDENTIFY_SYSTEM's). Fixed at the
source: pgautofailover.get_latest_basebackup() grew an optional
preferred_source filter (both SQL files), and service_archiver_serve.c's
routes refresh now asks for 'live' specifically -- a live-sourced
backup always shares the walcache's timeline by construction. A second,
independent, defense-in-depth check (walcache_current_timeline(),
comparing the walcache's own newest captured segment's embedded
timeline against whatever's about to be advertised) keeps the routes
file from ever serving a mismatched pairing even if that invariant is
ever violated by a future backup mode. monitor_get_latest_basebackup_
info() also grew a timeline out-param, threaded into the routes file's
own (previously unpopulated) "timeline" key -- already parsed by
routes.c, never written by anyone until now.
- cmd_start_replication.c ended a stream with bare CopyDone and nothing
else. A real, long-lived streaming client (real walreceiver, via
primary_conninfo) never triggers the gap because it never decides to
stop on its own -- which is exactly why this went unnoticed through
all of the earlier fast-forward-from-archiver verification. But
pg_basebackup's --wal-method=stream background WAL receiver does
decide to stop, once it reaches its own target LSN, and real receive-
log.c's ReceiveXlogStream only accepts that as a *successful* stop
when it can read a matching CommandComplete afterward (matching real
walsender.c's own WalSndDone, which sends exactly that on controlled
shutdown) -- without it, the client falls through to "unexpected
termination of replication stream" and exits non-zero even though
nothing was actually wrong on the wire. Fixed by sending a CommandComplete
tagged "COPY" right after CopyDone.
…iver Adds archiver_bootstrap_and_fast_forward.pgaf, the disaster-recovery scenario this whole investigation was driven by: a primary, an archiver, and a secondary that's created via `pg_autoctl create postgres --from-archiver` (not from the live primary) after the archiver's first live base backup is ready, then a FAST_FORWARD election where the archiver is the only node with the WAL the winning candidate is missing. node2 is declared `create and launch deferred`: the normal ini-driven node bring-up (`pg_autoctl node start`) has no hook for a custom flag like --from-archiver (NodeSpec/nodespec.c carries no such field -- fromArchiver lives only in KeeperConfig, populated exclusively by cli_create_node.c's own direct CLI parsing), so test_001 `exec`s into node2's own container and runs `pg_autoctl create postgres --from-archiver` by hand, then backgrounds `pg_autoctl run` the same way debug_citus_worker_switchover.pgaf backgrounds a long-lived process (`bash -c "nohup ... &"` -- a foreground `pg_autoctl run` would hang `docker compose exec -T` forever otherwise). test_002-004 engineer a real WAL gap rather than relying on race timing: stop node2 so it can't stream from node1 anymore, generate more WAL on the primary and give the archiver (still capturing independently via pg_receivewal) time to land it, kill the primary, then bring node2 back -- at that point the archiver is strictly ahead of node2 and is the only viable FAST_FORWARD WAL source. The final row-count check on node2 post-promotion confirms real WAL bytes were fetched and applied, not just that the FSM passed through the right state label. Verified: `pgaftest show spec`/`show compose` parse this spec cleanly (exit 0) and `pgaftest indent` round-trips it losslessly, confirming the DSL usage (deferred node declaration, exec/nohup backgrounding, multi-state `passing through` clause) is syntactically valid against the real grammar. Could not run it against a live docker compose cluster in this session: `make -f Makefile.docker build-pg17` fails fetching ghcr.io/hapostgres/pg_auto_failover/pgaf-base (401 Unauthorized, no registry credentials available here), and no local base image is cached to build from instead. Every C-level behavior this spec exercises (--from-archiver's own bootstrap, and fast-forward sourcing WAL from an archiver) was independently verified working end-to-end by hand against a real cluster in the two preceding commits on this branch.
wal_archived() is a plain LANGUAGE sql function (not SECURITY DEFINER), so it runs under the caller's own privileges. autoctl_node never got a direct SELECT grant on pgautofailover.archiver_wal: the blanket `GRANT SELECT ON ALL TABLES IN SCHEMA pgautofailover TO autoctl_node` only covers tables that already existed when that statement ran, and archiver_wal (like every other table in the M1 archiving schema) was created after it. Confirmed live: calling wal_archived() as autoctl_node (the role node_active() actually uses) failed with "permission denied for table archiver_wal". get_latest_basebackup() had this exact same bug, already fixed the same way (SECURITY DEFINER) in a prior commit -- apply the same fix here.
BuildForPrimaryNodeNodeActiveContext() counted every other node in the group toward replicationQuorumCount/secondaryNodesCount/ secondaryQuorumNodesCount, including ARCHIVING rows -- which are never real Postgres secondaries and can never report SECONDARY. In a formation with only a primary and an archiver, that miscount let the archiver's own bootstrap WAIT_STANDBY reading trip anyOtherNodeWaitingStandby (pos 401) and bump the primary off SINGLE, while secondaryQuorumNodesCount could then never legitimately reach zero -- so the primary got stuck between SINGLE and PRIMARY forever. Skip ARCHIVING (hasPgData=false) rows in that loop, matching the hasPgData-based exclusion this file's own REPORTING_NODE section already applies for a different purpose. Also adds the archiver-mirror FSM rows (pos 394/396/399) their own SINGLE|WAIT_PRIMARY|JOIN_PRIMARY match set, since a primary attached only to an archiver legitimately stays SINGLE the whole time instead of ever reaching WAIT_PRIMARY.
…(M5)
Several pieces of the Archiving & Disaster Recovery milestone, landing
together since they build on each other:
WAL-capture reliability
- service_archiver_start_pgreceivewal() now creates a replication slot
for pg_receivewal (pgautofailover_standby_<nodeId>), the same one
keeper_create_and_drop_replication_slots() already creates eagerly
on any primary for every other node regardless of kind. Without a
slot, a pg_receivewal that loses the startup HBA-propagation race
restarts from the server's then-current position, silently and
permanently skipping whatever WAL existed in between.
- pg_walsender's START_REPLICATION now fails loudly ("58P01") instead
of waiting forever when asked for a segment that predates this
archiver's own captured history and will never arrive.
- The archiver's real captured-WAL position is now tracked out of
band (a position file, service_archiver_position_path() and
friends) so it can cross the fork() boundary between the capture
and serve processes -- consumed by cmd_base_backup.c's own
end-of-backup position (previously could re-send a stale start
position and hang a real pg_basebackup's background WAL streamer
forever) and by cmd_identify_system.c indirectly via the routes
file's new "position" key.
- service_archiver_loop() now sets pgIsRunning = true for the
archiver's own keeper state, which the monitor's NodeIsHealthy()
unconditionally requires before ever selecting a node as a
FAST_FORWARD WAL source.
Telemetry
- service_archiver_report_storage() reports disk usage/free space to
the monitor periodically; monitor_get_archivers() surfaces it (and
each archiver's FSM state) to `pg_autoctl watch`'s new archivers
section.
Base-backup production/retention policy
- New SQL: get_basebackup_policy_for_group(), list_basebackups();
get_basebackup_policy() gains SECURITY DEFINER (needed now that
`pg_autoctl show basebackup-policy` calls it directly).
- service_archiver_basebackup.c's scheduling is now policy-driven
instead of the previous hardcoded "bootstrap live, then exactly one
replay, then quiet" scope: frequency/source/replaymode/onpromotion
read from whichever policy resolves for the group, plus
maxcount/maxage retention pruning after each successful backup. The
very first backup for a group is always sourced live regardless of
policy (nothing to replay from yet).
- The replay/volatile staging instance now starts with ssl = off:
the copied postgresql.conf/postgresql.auto.conf still carries the
source node's own ssl_cert_file/ssl_key_file paths, meaningless
here since the archiver has no Postgres SSL certs of its own --
left enabled, the staging instance failed outright at startup.
- New CLI: `pg_autoctl create/show/set basebackup-policy`, and
`pg_autoctl create archiver --basebackup-policy <name>` to attach
one at creation time via set_archiver_policy().
Verified via a full --no-cache Docker rebuild plus the archiver_wal_
capture, archiver_basebackup_generation, archiver_basebackup_policy,
and archiver_bootstrap_and_fast_forward pgaftest specs, all passing.
archiver_wal_capture.pgaf: fixed a wrong segment-1 assumption (the archiver's replication slot only protects WAL from its own creation time onward -- by the time it's created, node1+node2's own bootstrap has typically already consumed segments up to the empirically observed floor, segment 3) and switched two `wait until ... state is primary` assertions to the real terminal state after a permanent primary loss (`wait_primary`: WAIT_PRIMARY -> PRIMARY requires another node to reach reported SECONDARY, which an archiver never will). archiver_basebackup_generation.pgaf: the schema's own 'default' policy (frequency 24h) no longer produces a second, replay-sourced backup within any sane test window now that scheduling is policy-driven instead of hardcoded "bootstrap live, then exactly one replay". Attach a short-frequency, source=replay policy during setup so the spec's own remaining job -- proving the replay/volatile generation pipeline itself still works -- is still genuinely exercised. archiver_bootstrap_and_fast_forward.pgaf: same wait_primary fix as above, applied where this spec also stops the original primary for good partway through. New: archiver_basebackup_policy.pgaf, covering the base-backup policy feature end to end -- a fast-cycling, maxcount=3 policy created via the real CLI, attached via set_archiver_policy(), reaching and holding a stable retained count after several times its frequency has elapsed. Registered both archiver_basebackup_policy and (previously missing) archiver_bootstrap_and_fast_forward in tests/tap/schedules/node.sch. All four specs verified passing against a from-scratch --no-cache Docker rebuild.
Intro - Rewrote the opening paragraph: pg_auto_failover is a complete system (pg_autoctl runs as its own pid 1 supervising postmaster), not just an extension -- dynamic topology, automated or operator- driven, two modes of operation (command-driven CLI and node.ini + `pg_autoctl node run`). - New "High Availability, Disaster Recovery, and Backups: One System" section with a new two-panel diagram (arch-ha-dr-unified.tex/.svg) contrasting the typical separate-HA-tool/separate-backup-tool split against pg_auto_failover's single control plane for both. Failover State Machine - New "Archiving" subsection in the State reference, covering the ARCHIVING state's real transitions (verified against live `pg_autoctl inspect fsm list --json` output), its exclusion from candidacy/quorum, and its role as a Fast_forward-eligible WAL source. - Added the 3 real archiving edges to the "Node init / join" and "Failover / promotion" mermaid diagrams, with a new archiverState color class and cross-reference notes. Updated the "20 states and 77 transitions" summary line to 21/80. Fault Tolerance - New "Archiving Nodes and Disaster Recovery" section: WAL capture independent of any standby, base backups on a policy, rebuilding a node (or a whole formation) from an archiver's cache, and how archiving nodes participate in (and are excluded from) failover. Operations - New docs/archiving.rst page: registering an archiver, creating and attaching base-backup policies, watching an archiver, and rebuilding a node with `pg_autoctl create postgres --from-archiver` -- including the disaster-recovery case of rebuilding a whole formation from a single surviving archiver. Reference - New CLI reference pages for `pg_autoctl create/show/set basebackup-policy`, registered in their respective toctrees. Verified with a clean `sphinx-build -W --keep-going` (no warnings, no broken references).
…anels Replaces the single stacked arch-ha-dr-unified diagram with two separate figures, each a "production architecture" style pair of dashed service-boundary boxes with a header + inner service pills: - arch-ha-dr-typical: High Availability (Patroni, repmgr) next to Disaster Recovery + Backups (pgBackRest, pgBarman) -- two entirely separate boundaries, naming the actual products a typical setup reaches for. - arch-ha-dr-pgautofailover: High Availability + Disaster Recovery collapse into a single pg_auto_failover box; Backups (pgBackRest, pgBarman) remains its own separate boundary. Colors are a muted, readable palette local to these two diagrams (dark-tinted text, pale tints for fills) rather than raw saturated brand colors used directly as text -- the previous version's bright green header/body text (mbox, #9BF00B) was a real readability problem. Node heights are compact (1.35cm pills) instead of the previous 2.3cm/6.4cm boxes, since most of these boxes hold a single line of text. intro.rst's "High Availability, Disaster Recovery, and Backups: One System" section is retitled "High Availability and Disaster Recovery: One System" and its body adjusted to match: Backups, in the narrower sense of retention/cataloguing/cloud tiers, is now described as its own remaining concern rather than folded into "one system," matching what the new diagrams actually show.
…maid The five keeper-FSM mermaid diagrams had drifted from real KeeperFSM[] output -- verified by re-running `pg_autoctl inspect fsm mermaid <phase>` for all five phases and diffing byte-for-byte against what was checked into the docs. Real gaps found and fixed: - Failover / promotion was missing the entire "wherever you were, you're being demoted now" fan-out (init/single/catchingup/secondary/ prepare_promotion/stop_replication/maintenance/prepare_maintenance/ wait_maintenance/report_lsn/fast_forward, each with both a -> demoted and -> demote_timeout edge), plus several report_lsn fan-in edges (fast_forward/prepare_promotion/stop_replication/ demote_timeout/join_secondary -> report_lsn) -- 28 missing edges in this diagram alone. - Node removal / drop was missing wait_maintenance -> single and fast_forward -> single. - Maintenance was missing wait_maintenance -> report_lsn. - The archiving state's edges (added in an earlier, hand-written pass) are now the tool's own generated labels/coloring (electionState amber, not a separate hand-added archiverState class) instead of hand-embellished text not backed by any real KeeperFSM[] comment. Node init / join and Steady-state / config changes already matched exactly. Updated the summary line and the "replaces the old Graphviz diagram" note from the stale 80/68 transition counts to the real total: 21 states, 102 transitions (111 raw KeeperFSM[] edges minus the 9 excluded join_primary ones). Fixed the Failover / promotion intro paragraph's "still less than half the size of the full graph" claim -- at 57 of 102 edges it's now over half, which the added fan-out edges explain (most of that diagram's size is exactly that "interrupted from anywhere" fan-out). Added an explicit `archiving_state` label on the State reference's Archiving entry so other pages can :ref: it directly instead of relying on an implicit, same-document-only section-title link.
New docs/archiving-internals.rst, in the Architecture toctree: the technical reference for how archiving is actually built, meant to be the main place to extend for later milestones (warm standby, PITR, cloud push). Covers, grounded directly in the current source (function names, exact invocations, exact file paths): - The two forked processes per archiver (capture, serve) and the two files (archiver-position, archiver-routes.ini) that are their only channel to each other -- new arch-archiver-internals diagram. - WAL capture: how an archiver's replication slot reuses the exact same mechanism a real standby's slot uses, with zero primary-side special-casing; the exact pg_receivewal invocation; how the real captured position is computed (including .partial-segment trailing- zero trimming) and shared across the fork boundary; what happens to pg_receivewal across a failover. - Base backup generation: the basebackup_policy table and its 3-tier resolution chain; exactly when a backup is due (bootstrap, onpromotion, frequency); the live pg_basebackup invocation; the full replay/volatile pipeline (staging instance, recovery config, promote, basebackup over loopback, discard); retention pruning. - pg_walsender: why it's a from-scratch reimplementation (no frontend-linkable server-side replication library exists), its process model, the routes-file-based auth/routing mechanism, and a full table of every wire command it implements. - How pg_autoctl create postgres --from-archiver and FAST_FORWARD reuse the port==0 archiver-serve-port resolution trick to talk to pg_walsender with no archiver-specific code past that one lookup. - Build/process wiring, and an explicit "extension points" section listing what M6/M7/M8 build on top of, and what's schema-complete but not yet enforced (concurrency, allowed_hosts). Verified clean with sphinx-build -W --keep-going (no warnings, no broken references).
Mermaid diagrams already get pan/scroll-to-zoom via mermaid_d3_zoom (conf.py), but that's specific to Mermaid's own inline-SVG rendering and never applied to the tikz-rendered figures the rest of the docs embed via `.. figure::` -- those render as plain <img src="....svg">, which d3-zoom can't attach to. New docs/_static/js/zoom.js + css/zoom.css: a small, dependency-free overlay wired to every `figure img` at page load. Click (or Enter/Space when focused) opens the image full-screen on a dark backdrop; scroll to zoom, drag to pan, double-click to reset, Esc/backdrop-click/close-button to dismiss. Wired site-wide via conf.py's existing add_css_file/ add_js_file setup() hook, the same mechanism already used for the project's custom CSS. Verified interactively: click opens the overlay, wheel/drag/dblclick/Esc all behave as expected, and a clean sphinx-build -W --keep-going.
New top-level section right after the page's own intro, before "The pg_auto_failover Monitor": frames High Availability as two distinct guarantees -- Service Availability (the Postgres service stays reachable, what the rest of this page/failover-state-machine.rst/ fault-tolerance.rst describe) and Disaster Recovery (the data survives even total loss of every node that ever held it, what archiving-internals.rst and the archiver covers) -- cross-referencing into both rather than duplicating either. Adds a page-level `fault_tolerance` label to fault-tolerance.rst (it had no explicit label of its own) so this new section can :ref: it directly.
Three boxes -- High Availability, Disaster Recovery, Backups -- on a single horizontal line in both diagrams, with whichever pair shares a provider wrapped in one outer box: - arch-ha-dr-typical: High Availability stands alone; Disaster Recovery and Backups are wrapped together (the same two products, pgBackRest/pgBarman, cover both roles in a typical setup). - arch-ha-dr-pgautofailover: the same three boxes, same colors, same layout, just regrouped -- High Availability and Disaster Recovery are now the wrapped pair, inside a box labeled pg_auto_failover; Backups stands alone in the slot Disaster Recovery and Backups shared on the other diagram. Replaces the previous, more complicated pass at this (2-column, vertically-stacked nested sub-boxes) with the simpler request: 3 boxes, one line, 2 of them wrapped.
arch-ha-dr-pgautofailover.tex had High Availability and Disaster Recovery wrapped in the pg_auto_failover box on the right and Backups standalone on the left -- reading right-to-left relative to arch-ha-dr-typical.tex's High Availability, Disaster Recovery, Backups order. Swapped positions (same widths/gaps, mirrored placement) so both diagrams read in the same order, only the wrapping differs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Archiving & Disaster Recovery, Milestones 1–5 of the design (schema through base-backup generation + policy), plus the docs coverage for all of it.
An archiver is a new node kind that captures a group's WAL continuously (
pg_receivewal, via the same replication-slot mechanism a standby already uses) and produces periodic base backups (live or replayed locally from its own WAL cache), independent of whether any standby is healthy or even present. A new standalone binary,pg_walsender, serves that captured data back out over a real (subset of the) PostgreSQL replication protocol, so a realpg_basebackup, a real streaming standby, or this project's ownrestore_commandcan all talk to an archiver with no archiver-aware code beyond finding it.This is not ready to merge — opening now for early review of direction and approach while the remaining work continues. See "Known gaps" below.
What's in this PR
pgautofailoverSQL schema for archivers —archiver,basebackup_policy,archiver_policy,archiver_wal,basebackup,wal_archived(), retention/pruning functions.service_archiver,archiver serve,pg_walsender(M2): the keeper-sideARCHIVINGFSM state;service_archiver.c's WAL-capture loop; a brand-new standalone binary (src/bin/pg_walsender/) implementing enough of the replication wire protocol from scratch (no frontend-linkable server-side implementation exists anywhere in PostgreSQL to link against) —IDENTIFY_SYSTEM,SHOW,BASE_BACKUP,TIMELINE_HISTORY,CREATE/READ_REPLICATION_SLOT,START_REPLICATION, and aFETCH_FILEside channel forrestore_command.pg_autoctl node runsupport forkind = archiver(M3).archiver_wal/wal_archived()tracking, re-pointingpg_receivewalat a new primary after a failover.live(realpg_basebackupagainst a healthy node) andreplay/volatile(extract the last backup, replay locally captured WAL forward against a throwaway staging instance, snapshot over loopback, discard) sources; policy-driven scheduling (frequency,onpromotion) and retention (maxcount,maxage); new CLI (pg_autoctl create/show/set basebackup-policy,--basebackup-policyoncreate archiver).pg_autoctl create postgres --from-archiver(bootstrap a new node straight from an archiver's cache) andFAST_FORWARDreusing an archiver as a WAL source during a multi-standby election, both via the same well-known-port resolution trick, no archiver-specific code in the ordinary standby-init/fast-forward paths themselves.SINGLEandPRIMARYforever (group_state_machine.c), and the replay staging instance failed to start under SSL (missing certs it has no reason to have).archiving-internals.rst) written as the technical reference/extension point for the milestones after this one; a new Operations page;ARCHIVINGstate coverage in the FSM docs (plus a full regeneration of the five FSM mermaid diagrams frompg_autoctl inspect fsm mermaid— real drift was found and fixed there, unrelated to archiving); fault-tolerance coverage; a rewritten intro with new architecture diagrams; and a small site-wide docs feature (click-to-zoom on figures, generalizing the zoom Mermaid diagrams already had).Known gaps — why this isn't ready yet
pg_walsender(all-new wire-protocol code) and the FSM fix ingroup_state_machine.c.conf.pychange (adds the click-to-zoom JS/CSS): Sphinx's incremental build doesn't reliably re-emit the<script>/<link>tags on every already-built page just becauseconf.pychanged — only pages whose own.rstsource changed get regenerated. If a localdocs/_buildpredates this PR,make -C docs htmlalone won't retrofit the zoom feature onto older pages; runmake -C docs clean html(or deletedocs/_build) once to pick it up everywhere.archiving-internals.rst's "Extension points" section is written to be where that work plugs in.Testing
New/updated
pgaftestspecs:archiver_wal_capture,archiver_bootstrap_and_fast_forward,archiver_basebackup_generation,archiver_basebackup_policy— all passing against a from-scratch--no-cacheDocker rebuild.citus_indent --checkclean.sphinx-build -W --keep-goingclean (no warnings, no broken references).