When one of the device-only cliff asserts fires under real VRAM pressure (see #927), the prover process can hang indefinitely instead of crashing: observed ~15 times across three rented 5090s — the panicked thread dies, sibling table threads briefly continue, then the process goes fully quiescent (thread census: everything futex-parked plus 3 threads in CUDA driver poll waits, zero CPU) while holding ~19.5 GB of VRAM. Longest observed: 12+ minutes before being killed externally; it never exits on its own.
Partial mechanism (ptrace was unavailable on the boxes, so this is wchan-level + code reading): std::thread::scope cannot join because sibling threads are blocked in driver waits — plausibly on events whose producing thread died with the panic. The VramGate permit itself is unwind-safe (RAII drop), so it is not a leaked-permit deadlock.
Impact: any orchestration that waits on the prover (CI merge queue on rented hardware, production continuation drivers) waits forever. The merge-queue exposure is mitigated by per-target timeouts (#924), but the process-level bug remains.
Fix candidates, roughly in order: a panic hook that aborts the process when a prover worker thread panics (a dead prove has nothing to protect); auditing cross-stream event waits for producer-death (bounded waits); both.
Repro: any cliff panic under pressure — real block 25368371 at epoch 2^22 with a 5 GB VRAM ballast triggers cliffs in ~35-40% of proves.
When one of the device-only cliff asserts fires under real VRAM pressure (see #927), the prover process can hang indefinitely instead of crashing: observed ~15 times across three rented 5090s — the panicked thread dies, sibling table threads briefly continue, then the process goes fully quiescent (thread census: everything futex-parked plus 3 threads in CUDA driver poll waits, zero CPU) while holding ~19.5 GB of VRAM. Longest observed: 12+ minutes before being killed externally; it never exits on its own.
Partial mechanism (ptrace was unavailable on the boxes, so this is wchan-level + code reading):
std::thread::scopecannot join because sibling threads are blocked in driver waits — plausibly on events whose producing thread died with the panic. TheVramGatepermit itself is unwind-safe (RAII drop), so it is not a leaked-permit deadlock.Impact: any orchestration that waits on the prover (CI merge queue on rented hardware, production continuation drivers) waits forever. The merge-queue exposure is mitigated by per-target timeouts (#924), but the process-level bug remains.
Fix candidates, roughly in order: a panic hook that aborts the process when a prover worker thread panics (a dead prove has nothing to protect); auditing cross-stream event waits for producer-death (bounded waits); both.
Repro: any cliff panic under pressure — real block 25368371 at epoch 2^22 with a 5 GB VRAM ballast triggers cliffs in ~35-40% of proves.