Fail-closed purge deletes gRPC certificates, leaving Gateway unrecoverable: Core never re-adopts (2.0.4)

[valentinvilar-conexa]

Summary

On gateway 2.0.4, any Core outage longer than core_disconnect_grace_period (default 30s) puts the Gateway into a state it cannot recover from without manual database surgery on the Core. The fail-closed purge deletes the Gateway's gRPC certificates and returns it to setup mode (plaintext), but the Core still has the Gateway recorded as adopted, so it keeps dialing mTLS forever. Neither side recovers.

This is the same user-visible failure as #353, which was closed as fixed in 2.0.4 by #357. #357 does fix the missing-interface symptom, but not this.

Environment

• Gateway 2.0.4+ebac60b, Core 2.0.3+0015912, Proxy/Edge 2.0.2+322ac3f
• Ubuntu 24.04, .deb packages from apt.defguard.net (trixie release-2.0), kernel WireGuard
• AWS EC2, Core and Gateway on separate instances, PostgreSQL 18 on RDS
• Deployed with the official Terraform modules (DefGuard/deployment, terraform2.0), auto-adoption via DEFGUARD_ADOPT_GATEWAY / DEFGUARD_ADOPT_EDGE

Reproduction

1. Deploy Core + Gateway and let auto-adoption complete. Gateway is healthy, wg0 up, traffic flowing.
2. Stop the Core for more than 30 seconds (systemctl stop defguard, a container restart, an instance replacement, an OS patching window — anything).
3. Start the Core again.

What happens

Gateway log:

Core still disconnected after 30s; purging (fail-closed)
Interface wg0 removed successfully
Removed gRPC certificate at /etc/defguard/certs/gateway_grpc_cert.pem
Removed gRPC key at /etc/defguard/certs/gateway_grpc_key.pem
Removed CA certificate at /etc/defguard/certs/grpc_ca_cert.pem
Removed Core client certificate at /etc/defguard/certs/core_client_cert.pem
Removed gRPC certificate files; entering setup mode
Restarting setup server after purge request
Starting Gateway setup server on 0.0.0.0:50066 and awaiting configuration from Core for 10 min

Core log, then, every 10 seconds indefinitely:

Failed to connect to Gateway http://10.0.0.10:50066/, retrying: code: 'The service is currently
unavailable', message: "received corrupt message of type InvalidContentType", source:
tonic::transport::Error(Transport, ConnectError(Custom { kind: Other, error: Custom { kind:
InvalidData, error: InvalidMessage(InvalidContentType) } }))

The Core is speaking mTLS; the Gateway is now a plaintext setup server. InvalidContentType is that mismatch.

Why it does not recover

The Core still has the Gateway in its gateway table, adopted, with a certificate_serial and a stored core_client_cert_der. So it takes the mTLS client path and never attempts adoption again. On startup it logs:

Wizard already completed, skipping initialization

We confirmed the gate is wizard.completed. Deleting the gateway row is not enough — with wizard.completed = true the Core still skips adoption entirely, and the Gateway is simply left unconfigured. Restarting either or both services does not help: the Gateway reopens its 10-minute setup window, but nothing on the Core side ever dials it in setup mode.

The only recovery we found is unsupported and touches the database:

DELETE FROM gateway;
UPDATE wizard SET completed = false, active_wizard = 'auto_adoption', auto_adoption_state = NULL;
-- restart Core (within the Gateway's 10-minute setup window), then:
UPDATE wizard SET completed = true, active_wizard = 'none';

That works and preserves the location, users and devices (Auto-adoption: reusing existing network location name=Gateway id=1 for new gateway), but it should not be the answer.

Why #357 does not cover this

#357 adds core_disconnect_grace_period and makes configure() recreate the interface if missing. Both are real improvements. But purge() still removes the certificates and re-enters setup mode — the code comment is explicit that this is intended (otherwise we fail closed). So the failure is only delayed by the grace period, not removed: a Core outage longer than the window reproduces it exactly as on 2.0.2.

Since #353 reports this same scenario ("After rebooting core instances or changing core configuration, all gateways will be disconnected / getting errors until restart") and is closed, there is currently no open tracking for it.

Impact

The trigger is very low-bar: a Core restart that takes over 30 seconds is enough. No instance replacement or misconfiguration required. Routine operations — patching, upgrades, config changes, a slow service restart — take the VPN down in a way that requires manual DB intervention to fix, and there is no documented recovery procedure (troubleshooting guide only says "make sure Defguard Gateway has been properly adopted").

Suggested fixes (in rough order of preference)

1. Do not delete the certificates on purge. Tearing down wg0 is a reasonable fail-closed action; discarding the mTLS identity is what makes it unrecoverable. Keeping the certs would let the Gateway reconnect as soon as the Core is back — which is what fix: handle missing interface on disconnect recovery #357's grace period is already trying to achieve.
2. Let the Core re-adopt a component it already knows. If the Core gets InvalidContentType (or any TLS-vs-plaintext mismatch) against a Gateway it believes is adopted, that is a reliable signal the peer is in setup mode. It could fall back to the adoption path and re-issue certificates, independently of wizard.completed.
3. Expose a supported re-adoption action in the web UI / API for a component that has lost its certificates, so recovery does not require SQL.
4. At minimum: raise the default grace period well above a typical restart, and document the recovery procedure.

Happy to provide more logs or test a patch — we have a reproducible environment.

[valentinvilar-conexa]

Re-filed in the central tracker as DefGuard/defguard#3514, following the bug_report.md template there.

Two reasons it belongs in that repo rather than here: the recovery gate we identified (wizard.completed) is Core code, and two of the four suggested fixes are Core-side. The gateway-side half — purge() deleting the mTLS certificates — is of course still this repo's.

Happy for this one to be closed in favour of #3514, or kept open to track the gateway-side fix specifically. Whichever you prefer.