pgaftest/pg_autoctl: step-mode tooling (no-autopilot, fsm step, report/advance split)

docs/ref/pg_autoctl_manual.rst

@@ -16,6 +16,7 @@ stopped or stuck, and for low-level diagnostic work. Using them while
 .. toctree::
    :maxdepth: 1
 
+   pg_autoctl_manual_fsm_step
    pg_autoctl_manual_service_restart
 
 ``pg_autoctl manual`` provides the following sub-command groups::
@@ -31,16 +32,20 @@ stopped or stuck, and for low-level diagnostic work. Using them while
     pg_autoctl manual fsm
       init    Initialize the keeper's state on-disk
       assign  Assign a new goal state to the keeper
-      step    Make a state transition if instructed by the monitor
+    + step    Make a state transition if instructed by the monitor
     + nodes   Manually manage the keeper's nodes list
 
+    pg_autoctl manual fsm step
+      report   Report the current state to the monitor without transitioning
+      advance  Attempt the transition already assigned by the monitor
+
     pg_autoctl manual fsm nodes
       get  Get the list of nodes from file (see --disable-monitor)
       set  Set the list of nodes to file (see --disable-monitor)
 
     pg_autoctl manual service
     + restart  Restart pg_autoctl sub-processes (services)
-    + pgctl    Signal the pg_autoctl postgres controller
+    + pgctl    Signal the pg_autoctl postgres service
 
     pg_autoctl manual service restart
       postgres     Restart the pg_autoctl postgres controller service
@@ -67,8 +72,8 @@ stopped or stuck, and for low-level diagnostic work. Using them while
       drop    Drop a replication slot on the primary server
 
     pg_autoctl manual primary adduser
-      monitor  Add a local user for queries from the monitor
-      replica  Add a local user with replication privileges
+      monitor  add a local user for queries from the monitor
+      replica  add a local user with replication privileges
 
     pg_autoctl manual standby
       init            Initialize the standby server using pg_basebackup
@@ -77,12 +82,12 @@ stopped or stuck, and for low-level diagnostic work. Using them while
       promote         Promote a standby server to become writable
 
     pg_autoctl manual coordinator
-      add       Add this node to its formation's coordinator
-      activate  Activate this node on its formation's coordinator
-      remove    Remove this node from its formation's coordinator
+      add       Add this pg_auto_failover node to its formation's coordinator.
+      activate  Activate this pg_auto_failover node to its formation's coordinator.
+      remove    Remove this pg_auto_failover node to its formation's coordinator.
     + update    Update current node's host:port on the coordinator
 
     pg_autoctl manual coordinator update
-      prepare   Prepare a Citus coordinator metadata update
-      commit    Commit a Citus coordinator metadata update
-      rollback  Rollback a Citus coordinator metadata update
+      prepare   Prepare transaction for master_update_node on the coordinator
+      commit    Commit prepared transaction for master_update_node on the coordinator
+      rollback  Rollback prepared transaction for master_update_node on the coordinator

docs/ref/pg_autoctl_manual_fsm_step.rst

@@ -0,0 +1,95 @@
+.. _pg_autoctl_manual_fsm_step:
+
+pg_autoctl manual fsm step
+===========================
+
+pg_autoctl manual fsm step - Make a state transition if instructed by the monitor
+
+Synopsis
+--------
+
+::
+
+  usage: pg_autoctl manual fsm step  [ --pgdata ] [ --json ] [report|advance]
+
+  --pgdata      path to data directory
+  --json        output data in the JSON format
+
+  Available commands:
+    pg_autoctl manual fsm step
+      report   Report the current state to the monitor without transitioning
+      advance  Attempt the transition already assigned by the monitor
+
+Description
+-----------
+
+Called with no argument, ``pg_autoctl manual fsm step`` does both halves of
+a step in a single, atomic call: it reports the node's current state to the
+monitor, and then immediately attempts whatever transition the monitor
+assigns back. This is the same thing the node-active service's own
+autopilot loop does on every tick, exposed here as a one-shot command for
+manual recovery.
+
+Passing ``report`` or ``advance`` as an argument splits that combined call
+into its two independently-issuable halves:
+
+``report``
+  Reports the node's current state to the monitor and persists whatever
+  goal state the monitor assigns back, without attempting the transition.
+
+``advance``
+  Attempts the transition already on file (typically from an earlier
+  ``report`` call) without talking to the monitor again.
+
+Neither half re-runs the other, so observing the effect of ``advance`` on
+the monitor's own view still needs a following ``report`` (or plain
+``step``) call.
+
+If the node's node-active service is currently suspended
+(``PG_AUTOCTL_SUSPENDED``), it owns the keeper's FSM already, so ``step``,
+``report``, and ``advance`` are all dispatched over a small Unix-domain
+control socket to that running service instead of stepping the FSM from
+this one-shot process, which would otherwise race the running service. See
+the ``suspended`` node modifier in :ref:`pgaftest <pgaftest>`, which
+starts a ``pgaftest`` node suspended so that a test spec can freeze its
+FSM and advance it one transition at a time via the ``fsm step <node>`` DSL
+command.
+
+Options
+-------
+
+--pgdata
+
+  Location of the Postgres node being managed locally. Defaults to the
+  environment variable ``PGDATA``.
+
+--json
+
+  Output JSON formatted data. Not currently supported by this command; a
+  warning is printed and plain-text output is used instead.
+
+Examples
+--------
+
+Combined step, on a node whose goal state the monitor has just bumped to
+``single``::
+
+  $ pg_autoctl manual fsm step --pgdata node2
+  catchingup ➜ single
+
+Splitting that same transition into its two halves — first observe what
+the monitor assigns, without moving::
+
+  $ pg_autoctl manual fsm step report --pgdata node2
+  catchingup ➜ single
+
+then perform it::
+
+  $ pg_autoctl manual fsm step advance --pgdata node2
+  catchingup ➜ single
+
+In both examples the printed pair is ``<state before this call>
+➜ <state after this call>`` — for ``report`` that's the node's own,
+unchanged current state on the left and the monitor's newly assigned goal
+state on the right; for ``advance`` it's the state the node started this
+call at on the left and the state it just transitioned to on the right.

docs/ref/pgaftest.rst

@@ -440,6 +440,10 @@ Node modifiers:
                                               (``--region``; default: ``default``)
 ``launch deferred``                           Container starts with ``sleep infinity``;
                                               use ``exec node  pg_autoctl node start``
+``suspended``                                 The node-active service never transitions
+                                              on its own; drive it explicitly with the
+                                              ``fsm step <node>`` DSL command (see
+                                              `Suspended nodes`_ below)
 ``coordinator`` / ``worker group <N>``        Citus role
 ``no-monitor``                                Standalone node (no monitor)
 ``listen``                                    Bind all interfaces (``--listen 0.0.0.0``)
@@ -478,6 +482,38 @@ See "Deterministic node registration order" in
 and ``cli_node.c``).
 
 
+Suspended nodes
+~~~~~~~~~~~~~~~~
+
+A node declared with the ``suspended`` modifier starts with
+``PG_AUTOCTL_SUSPENDED`` set, which changes what its node-active service does
+on each FSM tick: instead of reporting its current state to the monitor and
+immediately attempting whatever transition the monitor assigns back —
+atomically, on every tick, with no way to observe or freeze the moment in
+between — it opens a small Unix-domain-socket server and waits. Nothing
+happens to that node's FSM until the spec explicitly drives it with the
+``fsm step <node>`` command, documented under "Commands inside ``setup``,
+``teardown``, and ``step`` blocks" below.
+
+This exists so a test can hold a node frozen at a specific reported state on
+purpose — e.g. to prove the monitor assigns the right next state before the
+node itself races off to reach it, or to reproduce a specific ordering
+between two nodes that would otherwise be a race under the normal, freely
+ticking FSM. Every other node in the same spec (without the modifier) keeps
+autopiloting normally; ``suspended`` only affects the node(s) it's
+declared on.
+
+``fsm step <node>`` is sugar for the combined
+``pg_autoctl manual fsm step`` command (report the current state, then
+immediately attempt whatever transition the monitor assigns). The underlying
+CLI also exposes the two halves separately as ``manual fsm step report`` and
+``manual fsm step advance`` — see :ref:`pg_autoctl_manual_fsm_step` — for
+scenarios that need to observe the monitor's assigned goal state before
+deciding whether, or when, to actually attempt the transition; the DSL does
+not yet have separate sugar for the split, so use ``exec <node>  pg_autoctl
+manual fsm step report`` / ``... advance`` directly for that.
+
+
 Commands inside ``setup``, ``teardown``, and ``step`` blocks
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -575,6 +611,12 @@ node ...`` which queries the running node/monitor instead of the file.
    stop postgres  <node>
    start postgres <node>
 
+**FSM step** (``suspended`` nodes only — see `Suspended nodes`_ above)
+
+.. code-block:: text
+
+   fsm step <node>
+
 **Failover**
 
 .. code-block:: text
@@ -762,6 +804,12 @@ Schedules under ``tests/tap/schedules/*.sch`` group these into CI jobs.
 ``fast_forward``
     Test fast-forward stuck detection and recovery.
 
+``fsm_step_report_advance``
+    Test the ``pg_autoctl manual fsm step report``/``... advance`` split
+    using a ``suspended`` node: report a stale state to the monitor
+    without transitioning, then advance to the transition the monitor
+    already assigned, proving the two halves work independently.
+
 ``guard_data_loss``
     Test ``pgautofailover.guard_data_loss`` /
     ``pg_autoctl perform failover --allow-data-loss``.