Fix MGARD-X HIP Huffman lossless (codec correctness + code-length OOB)#257
Open
gqian-coder wants to merge 1 commit into
Open
Fix MGARD-X HIP Huffman lossless (codec correctness + code-length OOB)#257gqian-coder wants to merge 1 commit into
gqian-coder wants to merge 1 commit into
Conversation
Four independent bugs in the MGARD-X HIP ParallelHuffman lossless path: three in the Huffman codec (canonical-codeword generation, header serialization, and codeword-length overflow handling) made GPU-compressed streams undecodable/oversized or abort, and one out-of-bounds device read in the parallel code-length generation faulted on the GPU. All verified on gfx90a (Frontier MI250X/MI210): HIP-compressed streams now round-trip and are byte-compatible with the SERIAL decoder. HuffmanWorkspace.hpp: allocate `status` as non-managed device memory. It was managed (hipMallocManaged); atomicMin on fine-grained memory is unreliable on ROCm/gfx90a, so GenerateCW::Operation4's atomic min never updated newCDPI. The Huffman length-group boundary was never found, all symbols collapsed into one group, and the canonical codewords came out with a constant offset -- codebook and decodebook mutually inconsistent, so the stream was undecodable by any backend. Huffman.hpp: default-initialize `outlier_count = 0`. ComposedLosslessCompressor::Compress -> CompressPrimary never writes outlier_count, but Serialize reads it to size compressed_data.resize(). The uninitialized value produced a garbage-sized allocation (hipMalloc OOM, or an invalid Copy1D). GetCodebook.hpp: throw instead of exit(1) when the longest codeword exceeds the H-type budget (sizeof(H)*8 - 8 bits), so callers can catch it and fall back to another lossless backend or a smaller dict_size. GenerateCL.hpp: bounds-check histogram[_lNodesCur + _curLeavesNum] in the parallel code-length generation; when every remaining leaf joins the batch the index reached dict_size (one past end of the nz_dict_size histogram), faulting on GPU. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gqian-coder
force-pushed
the
hip-lossless-huffman-fixes
branch
from
July 22, 2026 15:44
ae8d8e6 to
177b1b3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four independent bugs in the MGARD-X HIP ParallelHuffman lossless path: three in the Huffman codec (canonical-codeword generation, header serialization, and codeword-length overflow handling) made GPU-compressed streams undecodable/oversized or abort, and one out-of-bounds device read in the parallel code-length generation faulted on the GPU. All verified on gfx90a (Frontier MI250X/MI210): HIP-compressed streams now round-trip and are byte-compatible with the SERIAL decoder.
HuffmanWorkspace.hpp: allocate
statusas non-managed device memory.It was managed (hipMallocManaged); atomicMin on fine-grained memory is
unreliable on ROCm/gfx90a, so GenerateCW::Operation4's atomic min never
updated newCDPI. The Huffman length-group boundary was never found, all
symbols collapsed into one group, and the canonical codewords came out
with a constant offset -- codebook and decodebook mutually inconsistent,
so the stream was undecodable by any backend.
Huffman.hpp: default-initialize
outlier_count = 0.ComposedLosslessCompressor::Compress -> CompressPrimary never writes
outlier_count, but Serialize reads it to size compressed_data.resize().
The uninitialized value produced a garbage-sized allocation (hipMalloc
OOM, or an invalid Copy1D).
GetCodebook.hpp: throw instead of exit(1) when the longest codeword
exceeds the H-type budget (sizeof(H)*8 - 8 bits), so callers can catch
it and fall back to another lossless backend or a smaller dict_size.
GenerateCL.hpp: bounds-check histogram[_lNodesCur + _curLeavesNum] in the
parallel code-length generation; when every remaining leaf joins the
batch the index reached dict_size (one past end of the nz_dict_size
histogram), faulting on GPU.