feat(pgbouncer): peer so_reuseport workers to forward cross-worker query cancels

[jaredLunde]

Problem

beyond-pg fronts Postgres with PgBouncer and runs 1..N so_reuseport pooler workers (PgbScaler scales up on many-vCPU boxes). so_reuseport was enabled, but peering was never configured. Peering is what forwards a query-cancel that the kernel routes to the "wrong" worker to the one actually holding the session.

Without it, on a box that has scaled past one worker, a CancelRequest (Ctrl-C, client statement-timeout, PQcancel) opens a fresh TCP connection that the kernel may route to any worker. If it lands on a worker that doesn't hold the session — probability (N-1)/N — the cancel is silently dropped and the query keeps running. pool_mode = transaction doesn't help; cancelling a mid-statement query is a normal operation.

This is the third component of the ClickHouse "scaling PgBouncer" design (fleet + so_reuseport + peering); the first two were already implemented here, only peering was missing.

Fix

Configure PgBouncer peering, generated per worker at boot:

• Each worker gets a stable peer_id, a distinct unix_socket_dir, and an identical [peers] map pre-declaring every possible worker (1..=max_workers).
• The map is pre-declared for the full worker cap, so the reactive scaler never rewrites config as it grows/shrinks the live set. Forwarding to an idle (not-yet-running) peer simply fails — exactly as a dropped cancel would have before — so pre-declaring idle peers is safe (no regression).
• Single-worker boxes (max_workers == 1, i.e. < 8 vCPU) can never misroute a cancel, so peering is omitted and unix_socket_dir stays empty — byte-compatible with today.

Changes

• src/config.rs — pgbouncer_ini(…, peer_id) emits unix_socket_dir / peer_id / [peers] when max_workers > 1; new pgb_peer_id, pgb_peer_socket_dir, pgbouncer_ini_path helpers. peer_id 1 keeps the canonical /etc/pgbouncer/pgbouncer.ini.
• src/boot.rs — writes one config per possible worker and creates each peer socket dir (owned by postgres) up front.
• src/supervisor.rs — spawn_pgbouncer selects the per-worker config from the worker's stable name.
• packer/files/pgbouncer/pgbouncer.ini — drops the hardcoded empty unix_socket_dir (now generated per worker).
• tests/pgbouncer_peering.rs + mise test:peering — e2e regression test.

Empirical validation

Against real PgBouncer 1.25.2 (the version prod's Ubuntu-noble image and the beyond-pg-test image both ship), two workers in front of one Postgres, a CancelRequest for worker A's session sent deliberately to worker B:

Peering	Result
OFF (pre-fix)	cancel dropped — pg_sleep(30) runs to completion
ON (this PR)	cancel forwarded — query cancelled (57014) immediately

tests/pgbouncer_peering.rs encodes both cases (the OFF case is a control proving the test discriminates). Config-key correctness is covered by config unit tests; the e2e proves pgbouncer honors those keys.

Note: the e2e gives each worker its own port for deterministic routing — so_reuseport port-sharing is orthogonal to the forwarding logic and already runs in prod.

🤖 Generated with Claude Code