Fix loading of BF16 bare nn.Parameters (LTX-2 learnable_registers)#467
Open
hc20k wants to merge 1 commit into
Open
Fix loading of BF16 bare nn.Parameters (LTX-2 learnable_registers)#467hc20k wants to merge 1 commit into
hc20k wants to merge 1 commit into
Conversation
GGMLLayer only handles `weight` and `bias`, so bare nn.Parameters fall through to torch's default _load_from_state_dict, which needs a real float tensor. BF16 tensors skip the F32/F16 reshape and stay as raw uint8 byte buffers, so LTX-2's *_embeddings_connector.learnable_registers fail with "only Tensors of floating point dtype can require gradients" and a doubled final dim. Dequantize quantized tensors whose keys GGMLLayer cannot intercept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LTX-2 GGUFs (
sulphur-2-distilled,ltx-2.3-*) fail to load:Root cause
learnable_registersis a barenn.Parameteron a plain module (comfy/ldm/lightricks/embeddings_connector.py), not a Linear/Conv/Embedding/LayerNorm.GGMLOpsnever wraps it, soGGMLLayer._load_from_state_dictnever runs and it reaches torch's default path, which builds aParameter(..., requires_grad=...).These tensors are BF16.
loader.pyonly reshapes F32/F16, so BF16 stays as the raw uint8 mmap buffer — 2 bytes/element, hence the doubled last dim. The BF16 rescue directly below only coverslen(shape) <= 1, and these are 2D. Torch therefore receives uint8 and raises.The reported dim mismatch is a red herring:
GGMLTensor.shapereturns the logical shape (so torch's shape check passes), while the error text printsinput_param.size()— the raw byte size.Fix
GGMLLayercan only ever interceptweight/bias— everything else goes tounexpected_keys. So any other key must reach torch as a real float tensor.Testing
On
sulphur-2-distilled-Q4_K_M.gguf(archltxv, 4444 tensors):is_quantizedguard) and 2 are the BF16 registers.UnetLoaderGGUFnow loadsLTXAVModel; registers land as bfloat16 at(128, 2048)/(128, 4096).is_quantizedis False.Note
#392 addresses this same class of bug for lumina2 by matching key names. Happy to narrow this to
learnable_registersspecifically if you'd prefer the surgical approach.