fix(gpu): recover device-only declines at the remaining cliff sites (R4 DEEP, comp-tree, R3 barycentric) - #935
Open
ColoCarletti wants to merge 2 commits into
Open
fix(gpu): recover device-only declines at the remaining cliff sites (R4 DEEP, comp-tree, R3 barycentric)#935ColoCarletti wants to merge 2 commits into
ColoCarletti wants to merge 2 commits into
Conversation
…R2 commit, R3 OOD, R4 DEEP) Under VRAM pressure a device dispatch can decline after the device-only gate already skipped the host drain, and the host fallbacks at the R2 comp-poly commit, the R3 parts/trace OOD and the R4 DEEP loop hard-abort on the empty host buffers. Download the resident data instead: the trace LDEs via materialize_lde_trace_host, the H part evaluations via a new download off the resident R2 parts handle. The asserts remain only for handles that cannot serve the data. The R4 DEEP host loop reads both the trace and the part evals, so it recovers both sides. Also adds sticky fault-injection hooks (test-faults) to the cuda barycentric, DEEP and comp-tree entries: the drain-and-retry absorbs one-shot faults, so the cliff paths need a fault that keeps firing.
Three prove+verify runs under sticky faults (comp-tree, barycentric, DEEP), each requiring the device-only path to fire on the warm-up and the recovery counters to move.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #927.
Extends the #914 device-decline recovery to the downstream cliff sites, which today hard-abort under the same transient VRAM pressure (census on rented 5090s: R4 DEEP ×97, comp-tree ×42, R3 ×3):
materialize_composition_parts_host) instead of asserting. The parts handle is folded into the trace session before the commit so the recovery and every downstream consumer read it from one place.materialize_lde_trace_host, threaded&mutthroughget_trace_evaluations_from_lde.round_3/round_4/compute_deepnow take&mut Round1/&mut Round2(single caller).The asserts survive only for the case where a resident handle cannot serve the data, with the device-only contract's message. New counter
GPU_COMPOSITION_PARTS_DOWNLOADSmirrors the downgrade counters (parts side).Testing: sticky fault hooks in math-cuda (
test-faults) on the barycentric/DEEP/comp-tree entries — sticky because the drain-and-retry absorbs one-shot faults — plus three end-to-end tests incuda_fallback_tests.rsthat arm them, require the device-only path on warm-up, and assert the recovery counters moved and the proof verifies.Validation (RTX 5090): fault tests 5/5 (3 new + 2 existing),
cuda_path_integration7/7 (happy path, zero downgrades), stark cuda lib 220/220; CPU-only stark suite 217/217 and clippy clean (cuda / no-cuda / test-faults).