Skip to content

fix(terraform2.0/gateway): restart the service so it applies the written config; expose core_disconnect_grace_period - #173

Open
valentinvilar-conexa wants to merge 1 commit into
DefGuard:mainfrom
valentinvilar-conexa:fix/gateway-restart-and-grace-period
Open

fix(terraform2.0/gateway): restart the service so it applies the written config; expose core_disconnect_grace_period#173
valentinvilar-conexa wants to merge 1 commit into
DefGuard:mainfrom
valentinvilar-conexa:fix/gateway-restart-and-grace-period

Conversation

@valentinvilar-conexa

Copy link
Copy Markdown

Problem 1 — the Gateway never reads the config this module writes

terraform2.0/modules/gateway/setup.sh does, in order:

  1. line 32: apt install -y defguard-gateway — the package's postinst starts the service, using the gateway.toml shipped with the .deb
  2. lines 41-60: writes the module's gateway.toml
  3. line 79: systemctl start defguard-gateway

systemctl start on an already-active unit is a no-op, so step 3 does not apply step 2. The Gateway keeps running with the package's default configuration for the entire life of the instance, unless someone restarts it by hand.

The most damaging consequence is that masquerade stays false even when nat = true. The nft table DEFGUARD-wg0 is created but its POSTROUTING chain is empty:

table inet DEFGUARD-wg0 {
        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
        }
}

On AWS this is fatal and silent. Forwarded VPN packets leave the ENI with a source address in the client range, the ENI's source/destination check drops them in the hypervisor — no security group, no log — and nothing inside the VPC answers. The tunnel still connects (the handshake does not involve forwarding), the client gets its address and routes, so it looks healthy. Because the client sends all DNS to the VPN resolver, name resolution fails completely, which sends you looking at DNS instead of NAT. It took us a full day.

An easy way to tell whether a Gateway is affected, from the startup log line Starting Defguard Gateway ... with configuration: Config { ... }:

  • stats_period: 60, masquerade: false → running the .deb default config
  • stats_period: 30, masquerade: true → running this module's config

Fix: systemctl startsystemctl restart.

Problem 2 — core_disconnect_grace_period is not configurable

The Gateway purges fail-closed when the Core has been unreachable for core_disconnect_grace_period, whose default is 30s. The purge tears down wg0 and deletes the Gateway's gRPC certificates, dropping it back to setup mode. The Core still has it recorded as adopted, so it keeps dialing mTLS and loops on received corrupt message of type InvalidContentType indefinitely, with no automatic recovery. Filed separately as DefGuard/gateway#361.

Until that is resolved, the practical mitigation is a grace period longer than any normal Core restart — but this module does not write the key, so there is no way to set it without editing the file on the instance, which is lost on every instance replacement (and user_data_replace_on_change = true makes replacements routine).

This PR adds the variable and writes the key, defaulting to 300. Happy to change the default to 30 to preserve current behaviour if you prefer the module to stay neutral.

The core and edge modules have the same bug — they only appear to work

Both end their scripts with systemctl start after writing the config, exactly like gateway. We checked all three on a live deployment:

Module Restarted after config write? Outcome
gateway no Broken. Runs the .deb config for the life of the instance. masquerade stays false.
core yes — by crashing Works by accident.
edge no Works by coincidence.

core is masked by a crash loop. The packaged core.conf has no database configured, so the service the postinst started panics and core-dumps:

Started defguard.service - Defguard Core service.
message=Initializing DB pool
thread 'main' panicked at crates/defguard_common/src/db/mod.rs:30:10:
Database connection failed: PoolTimedOut
defguard.service: Main process exited, code=dumped, status=6/ABRT
defguard.service: Failed with result 'core-dump'.
defguard.service: Scheduled restart job, restart counter is at 1.

By the time systemd's restart policy brings it back, setup.sh has written the real core.conf, so it comes up correctly. It works because the packaged config is unusable — not by design.

edge never restarts at all (0 restarts across the boot) and therefore runs the packaged proxy.toml for the life of the instance. It happens to behave correctly only because every value the module sets coincides with a default:

--- packaged /etc/defguard/proxy.toml
+++ module-written /etc/defguard/proxy.toml
  http_port = 8080          # same
  grpc_port = 50051         # same
+ https_port = 443          # absent from the package — binary default happens to be 443
+ cert_dir = "/etc/defguard/certs"   # absent from the package — binary default matches

So the module's edge_http_port / edge_https_port variables are effectively ignored: set either to a non-default value and the Edge will silently keep listening on the old ports. Same class of silent failure as the Gateway, just currently invisible.

The one-word startrestart fix should be applied to all three modules. This PR only changes gateway (the one that is actively broken) to keep the diff focused — happy to extend it to core and edge in this PR or a follow-up, whichever you prefer.

Testing

Verified on Ubuntu 24.04 / EC2, gateway 2.0.4, Core 2.0.3: after the change the startup log shows masquerade: true, stats_period: 30 and core_disconnect_grace_period: 300, the POSTROUTING chain contains the masquerade rule, and restarting the Core no longer triggers a purge.

…ten config

The package postinst starts defguard-gateway with the gateway.toml shipped in
the .deb, before setup.sh writes the module's configuration. The final
`systemctl start` is a no-op on an already-active unit, so the service keeps
running with the packaged defaults for the life of the instance -- most
importantly `masquerade = false` even when `nat = true`, which on AWS makes the
ENI's source/destination check silently drop all forwarded VPN traffic.

Also expose core_disconnect_grace_period, which the module did not write, so the
fail-closed purge window can be raised above a normal Core restart
(see DefGuard/gateway#361).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant