Shut down workers on idle instead of a static maximum lifetime — Closes #100

[conradbzura]

Summary

A worker exits once it has been continuously idle past a threshold, rather than once a static wall-clock ceiling expires. wool 0.14 (wool-labs/wool#269) gives every worker an idle RPC reporting seconds since its in-flight task set last became empty, read through WorkerConnection.idle(). The worker_main serve loop dials its own worker over loopback — the same channel-back-to-own-subprocess pattern wool uses for graceful stop, so mTLS credentials and the identity SAN verify unchanged — and polls that value on a 15 s cadence, exiting once it crosses CFDB_WORKER_IDLE_TIMEOUT_SECONDS (default 600 s; 0 disables). A busy worker reports zero idle, so the idle exit never fires while a task is running.

Both self-termination paths stop the worker with a drain grace rather than wool's default immediate cancel. An idle reading is a snapshot, so it cannot rule out a dispatch accepted between the final poll and the teardown, and a max-lifetime expiry can land mid-job on a worker running jobs back to back. In either case the API has already marked that task running, where a graceless cancel is finalized as a terminal failure instead of being re-queued. CFDB_WORKER_MAX_LIFETIME_GRACE_SECONDS (default 6 h) returns instantly on an empty docket and is sized above the API's 4 h per-job duration cap, so the only work it can cancel is work already past that bound.

That drain is pinned against real wool rather than argued from its contract: an integration test dispatches a task to a live worker and stops it mid-flight, asserting the task completes, paired with a graceless stop asserting it is cancelled.

The grace is what lets the ceiling rise to 12 h. CFDB_WORKER_MAX_LIFETIME_SECONDS is no longer a periodic reaper that eventually preempts a healthy worker; it bounds the two cases idle reporting cannot see — a wedged or unimplemented idle RPC, and a stuck job that holds the in-flight set non-empty, since a hung subprocess reports zero idle exactly like a working one. Worst-case worker uptime is lifetime plus grace, 18 h at the defaults.

Closes #100

Proposed changes

Track wool 0.14

Move pyproject.toml from wool~=0.13.0 to wool~=0.14.0. The LocalWorker construction surface and every other wool import cfdb uses are unchanged between the two versions.

The specifier accepts patch releases but not 0.15, since a minor bump may move the wire protocol the dispatch channel rides on. Because wool admits a worker only when the proxy's version is at most the worker's, any lock refresh that moves the resolved version is a fleet-rejection event rather than a routine dependency update — refresh the lock deliberately and drain the fleet when it moves.

Idle-based shutdown with a draining exit

Add the idle poll to worker_main.serve, along with DEFAULT_IDLE_TIMEOUT_SECONDS (600 s), a 15 s poll cadence, and a 5 s per-poll gRPC deadline. The loop checks max lifetime, then the signal-drain path, then the idle poll, so a SIGTERM drain always takes precedence over an idle exit.

Track the stop grace as loop state rather than a fixed argument: the idle and max-lifetime exits set it to max_lifetime_grace_seconds, and the signal paths leave it at wool's immediate cancel, which is correct there because ECS bounds SIGTERM with SIGKILL and a long drain could never complete.

Raise DEFAULT_MAX_LIFETIME_SECONDS to 12 h and add DEFAULT_MAX_LIFETIME_GRACE_SECONDS at 6 h.

Poll-failure handling

A poll failure must neither kill a healthy worker nor flood the logs for hours. IdleUnavailable disables polling for the process lifetime, since a version-skew scenario must not crash-loop the poll. An isolated failure retries on the next cadence. After CFDB_WORKER_IDLE_POLL_FAILURE_LIMIT consecutive failures — 20 by default, roughly 5 minutes at the standard cadence — the worker escalates once to ERROR and stops polling, which turns a permanent TLS misconfiguration or a dead subprocess into one actionable CloudWatch record instead of hundreds of identical warnings. Both messages name the failing exception type, and both describe the remaining bound honestly rather than reporting a max-lifetime backstop that is disabled.

Worker knobs

Expose --idle-timeout-seconds, --idle-poll-interval-seconds, --idle-poll-failure-limit, and --max-lifetime-grace-seconds, each mirroring a CFDB_WORKER_* environment variable per the module's existing convention.

Wire the reaping knobs through the workers stack

Add WorkerIdleTimeoutSeconds (default "600") and WorkerMaxLifetimeGraceSeconds (default "21600") to cloudformation/workers.yml, both feeding the container environment, and raise WorkerMaxLifetimeSeconds to "43200" to match the code default. Template tests pin the env wiring and default-agreement for all three.

The WorkerIdleTimeoutSeconds description also records a tuning constraint that is otherwise discoverable only by hitting it: an idle timeout at or below CFDB_WORKFLOW_RETRY_INTERVAL_S plus worker cold-start time lets overflow-spawned workers reap themselves before the scheduler's next retry tick reaches them, and the jobs then ride the queue to the capacity: deadline while every dashboard looks healthy.

Pin the stop-grace contract against a real worker

Add an integration test that starts a real worker, dispatches a real task, and stops it mid-flight. Every unit test on this path mocks wool.LocalWorker, so the suite otherwise asserts only that cfdb passes a grace — nothing verifies wool honors it, and a wool change that cancelled regardless would leave the suite green while restoring the defect the grace exists to prevent. The graceless case is tested alongside it, so the two form a controlled pair differing only in the grace.

Documentation

Reframe the README's leaky-bucket and version-bump-wedge sections and the provisioner's orphan-worker note around idle-based reaping: rejected or orphaned workers self-reap in minutes rather than hours once the fleet runs an idle-aware image. State the reaping guarantee, the resulting worst-case uptime, and the idle-timeout sizing constraint in the worker-knobs paragraph. Name the idle poll as a third consumer of the mTLS identity contract alongside dispatch and the drain channel, since a worker now dials its own subprocess over loopback — under the address-verification opt-out (CFDB_WORKER_TLS_IDENTITY="") the worker leaf needs a loopback SAN, or every poll fails its handshake and idle shutdown degrades to the backstop with only poll warnings as the symptom.

Deployment notes

This PR is itself a wool version bump, so on deploy the rolled API rejects in-flight 0.13 workers until fleet turnover. Drain the dev worker fleet as part of the deploy, per the README's version-bump runbook: the old workers predate the idle RPC and cannot self-reap early. The same applies before the first prod promotion carrying this change.

Landing the two new parameters requires the usual one-time privileged workers.yml deploy per environment. Until it runs, deployed workers use the identical code defaults, so the deploy order is not load-bearing here.

Because the specifier is ~=0.14.0, a later lock refresh that picks up a wool patch release is itself a version bump and repeats the same drain procedure — the hazard is not limited to deliberate upgrades.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestMainCli	No CLI arguments and no overriding environment variables	worker_main.main is invoked	serve receives the documented defaults for every idle and lifetime knob	CLI defaults
2	TestMainCli	Idle, poll, lifetime, and grace environment variables set to non-default values	worker_main.main is invoked with no flags	serve receives the env-driven values	Env overrides
3	TestMainCli	Env vars set and conflicting CLI flags supplied	worker_main.main is invoked	CLI flags win over the environment	Flag precedence
4	TestServeIdleShutdown	A worker whose idle RPC reports more idle time than the timeout	serve runs	Exits through the idle path on the first poll, with a bounded RPC deadline, stopping the worker with the drain grace	Idle exit path
5	TestServeIdleShutdown	A worker always reporting zero idle and a short max lifetime	serve runs	Polls idle without exiting on it and terminates via max-lifetime, stopping the worker with the drain grace	Busy-worker immunity
6	TestServeIdleShutdown	A worker reporting idle strictly between zero and the threshold	serve runs	Keeps serving through those polls and exits via max-lifetime	Partial idle accumulation
7	TestServeIdleShutdown	idle_timeout_seconds set to 0	serve runs	Never constructs a WorkerConnection	Disable sentinel
8	TestServeIdleShutdown	An idle RPC raising IdleUnavailable	serve runs	Polls exactly once and exits via the max-lifetime backstop	Version-skew fallback
9	TestServeIdleShutdown	An idle RPC failing transiently once, then reporting idle beyond the threshold	serve runs	Survives the failed poll and exits via the idle path	Transient poll failure
10	TestServeIdleShutdown	An idle RPC failing non-transiently, succeeding, failing again, then reporting idle beyond the threshold, under a failure limit of two	serve runs	Retries through both isolated failures without escalating and exits via the idle path	Consecutive-failure counter reset
11	TestServeIdleShutdown	An idle RPC failing on every poll under a failure limit of two	serve runs	Emits exactly one ERROR naming the remaining bound and stops polling	Sustained-failure escalation
12	TestServeIdleShutdown	A credentials builder returning a sentinel and a non-default port	serve runs to idle exit	Constructs the connection against loopback at the worker port with those credentials	mTLS wiring
13	TestServeIdleShutdown	A worker exiting via the idle path	serve returns	Closes the WorkerConnection	Resource teardown
14	TestServeIdleShutdown	A WorkerConnection whose close raises	serve runs to completion	Still returns 0 and stops the worker	Teardown failure isolation
15	test_cloudformation	The workers template's idle-timeout parameter and container definition	The Environment entries and parameter default are read	CFDB_WORKER_IDLE_TIMEOUT_SECONDS references the parameter and the default equals the code constant	Infra wiring
16	test_cloudformation	The workers template's max-lifetime parameter and container definition	The Environment entries and parameter default are read	CFDB_WORKER_MAX_LIFETIME_SECONDS references the parameter and the default equals the code constant	Infra wiring
17	test_cloudformation	The workers template's lifetime-grace parameter and container definition	The Environment entries and parameter default are read	CFDB_WORKER_MAX_LIFETIME_GRACE_SECONDS references the parameter and the default equals the code constant	Infra wiring
18	TestWorkerStopGrace	A real worker running a dispatched task that outlasts the stop	The worker is stopped with a grace longer than the task's remaining runtime	Lets the task run to completion and return its result	Drain contract (integration)
19	TestWorkerStopGrace	A real worker running a dispatched task that outlasts the stop	The worker is stopped without a grace, wool's default	Cancels the task rather than returning its value	Cancel contract (integration)