Skip to content

fix(security): harden the replication surface (#103 + #104 + #105, rebased) - #107

Merged
kwhorne merged 4 commits into
mainfrom
security/replication-hardening
Aug 21, 2026
Merged

fix(security): harden the replication surface (#103 + #104 + #105, rebased)#107
kwhorne merged 4 commits into
mainfrom
security/replication-hardening

Conversation

@kwhorne

@kwhorne kwhorne commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Supersedes #103, #104 and #105 — same commits, same authorship (@HelgeSverre),
sequenced and with their mutual conflicts resolved. Opening it here rather than on
the fork because Actions never started a single check-suite for any of the
three
(total_count: 0, nothing awaiting approval), and a security fix should
not land on local verification alone.

Why this is urgent

The first one is exploitable in 1.9.8, released today. I reproduced it before
touching anything:

serve --listen 127.0.0.1:3460 --replication-listen 127.0.0.1:3461 \
      --auth root:adminpw:admin

→ connect to 3461, send zero bytes, receive 560 bytes containing the canary row

No handshake. The MySQL listener demands a password; the replication port
demanded nothing. The old guard only covered non-loopback binds, so
--replication-listen 127.0.0.1:… without a cluster secret let any local process
— or an SSRF payload that can open a TCP connection — read the entire database.

The other two are as real: a replica never verified its primary's identity, so
anything answering on the primary's address could inject fabricated rows even
with ELYRASQL_CLUSTER_SECRET set
(auth ran one direction only); and
elyrasql replica had no auth flags at all, so its MySQL listener always ran open
— any username logged in as Admin over replicated production data. I confirmed
the last one in the CLI definition: Replica took only primary, data,
listen.

What I changed while sequencing them

The three were written independently against the same base and conflict with
each other
, so they could not simply be merged in order:

One real interaction, in its own commit: replica_auth.rs spawns
elyrasql replica, which after #104 refuses to start without a cluster secret.
Both its tests now set one. This only fails with all three applied — each PR
passes the suite alone, which is exactly the class of problem sequencing them
surfaces.

Testing

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo nextest run --workspace --locked — 448/448
  • Verified against a MySQL client

Each PR also verified individually merged onto main first: 443/443, 443/443,
441/441.

The exploit is closed, checked both ways:

  • no cluster secret → the endpoint refuses to start, logs the reason, and the
    port is not listening (ConnectionRefusedError)
  • secret set → an unauthenticated peer receives 16 bytes (the challenge) and no
    data; the canary does not leak

One design question for you

Without a secret, serve_replication returns Err and the caller logs
ERROR replication endpoint stopped — but the process keeps running and
serves normally, just without replication. That is fail-closed on the endpoint,
which is right, but an operator who passed --replication-listen and does not
read logs now has a silently non-replicating primary. Worth deciding whether that
should be fatal at startup instead. Not changed here.

Checklist

  • Docs updated under docs/
  • No internal engine/dependency names leak into SQL, errors, CLI, or the wire handshake
  • Limitations documented honestly
  • Scoped to a single logical change (one surface: replication auth)

HelgeSverre and others added 4 commits August 21, 2026 16:25
… every bind

The replication endpoint hands a full copy of the database to every
connecting peer. It previously refused only non-loopback binds without
ELYRASQL_CLUSTER_SECRET, so a listener on 127.0.0.1 without a secret let
any local process (or an SSRF payload able to open a TCP connection)
exfiltrate the entire data set with a bare connection and zero bytes sent.

Require ELYRASQL_CLUSTER_SECRET on every bind address; keep
ELYRASQL_ALLOW_OPEN_AUTH=1 as the explicit opt-out. Add guard unit tests
and update security/configuration/limitations docs.

(cherry picked from commit d0b491b)
…ted primaries

A replica applies everything its primary streams to it, but it never
verified the primary's identity: auth_connect only proves the replica to
the primary. Any host accepting TCP connections on the primary's address
could impersonate it (the transport is plaintext unless cluster TLS is
configured) and feed the replica arbitrary snapshot/write frames --
fabricated tables, tampered rows, corrupted catalog state -- which the
replica then served to clients. Demonstrated end-to-end with a fake
primary against a stock replica.

- Add auth_verify_peer/auth_respond_to_peer: after the existing
  replica->primary proof, the primary must prove knowledge of
  ELYRASQL_CLUSTER_SECRET back to the replica before any data flows.
  Handshake is bounded by a 5s timeout so a stalling peer cannot hang
  the reconnect loop.
- run_replica fails closed when no cluster secret is configured unless
  ELYRASQL_ALLOW_OPEN_AUTH=1 explicitly opts in.

Primary and replica must be upgraded together (handshake changed).
Adds handshake unit tests plus an end-to-end primary/replica test.

(cherry picked from commit 9b2fe19)
elyrasql replica had no auth flags at all and always started its MySQL
listener with open authentication: any username/password logged in as
Admin with full read access to the replicated (often production) data
set. serve mode already defaulted to 127.0.0.1 with a safe-by-default
refusal for remote binds; replicas are typically part of real
deployments, so a credential-less default there is a much sharper edge.

- Add --user/--password/--auth USER:PASS:ROLE to the replica subcommand,
  mirroring serve.
- Refuse to start a credential-less replica listener unless
  ELYRASQL_ALLOW_OPEN_AUTH=1 explicitly opts in.
- Make elyra_server::env_flag public so the CLI reuses the same truthy
  parsing as the library guards.
- Add CLI tests pinning both the refusal and authenticated startup.

(cherry picked from commit 7c57db8)
Both tests in replica_auth.rs spawn `elyrasql replica`, which now refuses to
start without ELYRASQL_CLUSTER_SECRET -- replication authentication became mutual
and mandatory in the commit before this one. Neither test is about that guard:
one pins the MySQL listener's refusal to run credential-less, the other that it
starts once accounts are configured. Without a secret the first would have passed
for the wrong reason and the second could not get far enough to bind at all.

This interaction only appears with all three security commits applied; each one
passes the suite on its own.
@kwhorne
kwhorne merged commit 185cd7d into main Aug 21, 2026
8 checks passed
@kwhorne
kwhorne deleted the security/replication-hardening branch August 21, 2026 15:41
kwhorne added a commit that referenced this pull request Aug 21, 2026
## Summary

**A security release.** Three independent holes in the replication
surface
(#107, superseding #103#105), all reachable in 1.9.8.

The first needs no credentials and no handshake: connecting to the
replication
port returned a full copy of the database. Reproduced against 1.9.8 —
zero bytes
sent, 560 bytes back with the canary row in them. The guard only covered
non-loopback binds, so `--replication-listen 127.0.0.1:...` without a
cluster
secret was readable by any local process.

## The upgrade note is a `danger` block, not a `warning`

Because the fixes are deliberately breaking and the exposure is real:

- a primary with `--replication-listen` and a replica **both** now
require
  `ELYRASQL_CLUSTER_SECRET`
- **primary and replica must be upgraded together** — the handshake
gained a
  step, so a 1.9.9 replica will not accept a 1.9.8 primary
- a replica now needs accounts (`--user`/`--password`/`--auth`)
- a refused endpoint **keeps the server running**, so `replication
endpoint
stopped` in the log is the only signal that replication is not happening
—
  the note says to watch for it rather than assume
- anyone who ran an exposed endpoint should rotate what the data would
have
  revealed: the port left no access log

`ELYRASQL_ALLOW_OPEN_AUTH=1` opts out of all four, described as the
honest way to
say "this port is on a network I control" rather than as a convenience.

## The three things the release workflow does not validate

Per `CONTRIBUTING.md`: `SERVER_VERSION` → `8.0.12-ElyraSQL-1.9.9`
(confirmed with
`elyrasql version`), `testbench/sql-dump/Cargo.lock` refreshed with
`cargo metadata --locked` green in both workspaces, and the version
strings in
README, installation, deployment and mysql-compatibility. The
`Upgrading to 1.9.8` block keeps its title, and the 1.9.8 changelog
section is
byte-identical (verified by digest).

## Testing

- [x] `cargo fmt --all --check`
- [x] `cargo clippy --workspace --all-targets --all-features --locked --
-D warnings`
- [x] `cargo nextest run --workspace --locked` — 448/448
- [x] `cargo metadata --locked` in both workspaces
- [x] Verified against a MySQL client

The fix itself was verified both ways before merging: without a secret
the
endpoint refuses to start and its port does not listen; with one, an
unauthenticated peer receives the 16-byte challenge and no data.

## Checklist

- [x] Docs updated under `docs/`
- [x] No internal engine/dependency names leak into SQL, errors, CLI, or
the wire handshake
- [x] Limitations documented honestly
- [x] Scoped to a single logical change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants