Gateway is unrecoverable after a fail-closed purge: Core never re-adopts it (gated on wizard.completed)

[valentinvilar-conexa]

Describe the bug

If the Gateway loses its connection to the Core for longer than core_disconnect_grace_period (default 30s), it runs a fail-closed purge that tears down wg0 and deletes its own gRPC certificates, returning to setup mode (plaintext gRPC).

The Core, however, still has that Gateway recorded as adopted in its gateway table — with a certificate_serial and a stored core_client_cert_der — so it keeps taking the mTLS client path and never attempts adoption again. The result is an indefinite loop and a VPN that is down until someone edits the database by hand:

Failed to connect to Gateway http://10.0.0.10:50066/, retrying: code: 'The service is currently
unavailable', message: "received corrupt message of type InvalidContentType", source:
tonic::transport::Error(Transport, ConnectError(Custom { kind: Other, error: Custom { kind:
InvalidData, error: InvalidMessage(InvalidContentType) } }))

The Core is speaking mTLS; the Gateway is now a plaintext setup server. InvalidContentType is that mismatch.

The trigger is very low-bar: a Core restart that takes over 30 seconds is enough. No instance replacement or misconfiguration required — routine patching, an upgrade, a config change or a slow service restart all do it.

We filed this against the gateway repo first as DefGuard/gateway#361, but it belongs here: two of the fixes below are Core-side, and the gate that prevents recovery is Core code.

To Reproduce

1. Deploy Core + Gateway and let auto-adoption complete (DEFGUARD_ADOPT_GATEWAY). Gateway healthy, wg0 up, traffic flowing.
2. Stop the Core for more than 30 seconds — systemctl stop defguard, a container restart, an OS patching window, anything.
3. Start the Core again.

Gateway log at step 2:

Core still disconnected after 30s; purging (fail-closed)
Interface wg0 removed successfully
Removed gRPC certificate at /etc/defguard/certs/gateway_grpc_cert.pem
Removed gRPC key at /etc/defguard/certs/gateway_grpc_key.pem
Removed CA certificate at /etc/defguard/certs/grpc_ca_cert.pem
Removed Core client certificate at /etc/defguard/certs/core_client_cert.pem
Removed gRPC certificate files; entering setup mode
Restarting setup server after purge request
Starting Gateway setup server on 0.0.0.0:50066 and awaiting configuration from Core for 10 min

Core log at step 3: the InvalidContentType line above, every 10 seconds, indefinitely.

Expected behavior

Either the Gateway keeps its mTLS identity across a Core outage so it can simply reconnect, or the Core is able to re-provision a component it already knows. Today neither happens, so the system cannot return to a working state on its own.

Version information

• Defguard Core version: 2.0.3+0015912
• Defguard Gateway version: 2.0.4+ebac60b
  • Operating system running the gateway: Ubuntu 24.04, kernel WireGuard
• Defguard Proxy/Edge version: 2.0.2+322ac3f
• Database: PostgreSQL 18
• Packages from apt.defguard.net (trixie release-2.0); deployed with the official Terraform modules (DefGuard/deployment, terraform2.0)
• Browser: n/a (not a UI bug)

Additional context

Why it does not recover — the gate is wizard.completed. On startup the Core logs:

Wizard already completed, skipping initialization

Deleting the gateway row is not enough: with wizard.completed = true the Core skips adoption entirely and the Gateway is simply left unconfigured. Restarting either or both services does not help — the Gateway reopens its 10-minute setup window, but nothing on the Core side ever dials it in setup mode.

The only recovery we found is unsupported and touches the database directly:

DELETE FROM gateway;
UPDATE wizard SET completed = false, active_wizard = 'auto_adoption', auto_adoption_state = NULL;
-- restart the Core within the Gateway's 10-minute setup window, then:
UPDATE wizard SET completed = true, active_wizard = 'none';

That works and preserves the location, users and devices — the Core logs Auto-adoption: reusing existing network location name=Gateway id=1 for new gateway — but it should not be the answer.

Why DefGuard/gateway#357 does not cover this. #357 (which closed DefGuard/gateway#353, the same user-visible failure) adds core_disconnect_grace_period and makes configure() recreate the interface if missing. Both are real improvements. But purge() still removes the certificates and re-enters setup mode — the code comment is explicit that this is intended (otherwise we fail closed). So the failure is only delayed by the grace period, not removed: an outage longer than the window reproduces it exactly as on 2.0.2. Since #353 is closed, there is currently no open tracking for it.

There is no documented recovery procedure. The gateway troubleshooting guide only says "Make sure Defguard Gateway has been properly adopted", without explaining how to re-adopt one that has lost its certificates.

Suggested fixes, in rough order of preference:

1. Do not delete the certificates on purge (gateway-side). Tearing down wg0 is a reasonable fail-closed action; discarding the mTLS identity is what makes it unrecoverable. Keeping the certs would let the Gateway reconnect as soon as the Core is back — which is what chore(deps-dev): bump postcss from 8.4.27 to 8.4.31 in /web #357's grace period is already trying to achieve.
2. Let the Core re-adopt a component it already knows (Core-side). An InvalidContentType — or any TLS-vs-plaintext mismatch — against a Gateway the Core believes is adopted is a reliable signal that the peer is in setup mode. The Core could fall back to the adoption path and re-issue certificates, independently of wizard.completed.
3. Expose a supported re-adoption action in the web UI / API for a component that has lost its certificates, so recovery does not require SQL.
4. At minimum: raise the default grace period well above a typical restart, and document the recovery procedure.

Happy to provide more logs or test a patch — we have a reproducible environment.