Skip to content

Fix loading of BF16 bare nn.Parameters (LTX-2 learnable_registers)#467

Open
hc20k wants to merge 1 commit into
city96:mainfrom
hc20k:fix-bf16-bare-parameters
Open

Fix loading of BF16 bare nn.Parameters (LTX-2 learnable_registers)#467
hc20k wants to merge 1 commit into
city96:mainfrom
hc20k:fix-bf16-bare-parameters

Conversation

@hc20k

@hc20k hc20k commented Jul 16, 2026

Copy link
Copy Markdown

Problem

LTX-2 GGUFs (sulphur-2-distilled, ltx-2.3-*) fail to load:

RuntimeError: Error(s) in loading state_dict for LTXAVModel:
  While copying the parameter named "audio_embeddings_connector.learnable_registers",
  whose dimensions in the model are torch.Size([128, 2048]) and whose dimensions in
  the checkpoint are torch.Size([128, 4096]), an exception occurred :
  ('only Tensors of floating point dtype can require gradients',).

Root cause

learnable_registers is a bare nn.Parameter on a plain module (comfy/ldm/lightricks/embeddings_connector.py), not a Linear/Conv/Embedding/LayerNorm. GGMLOps never wraps it, so GGMLLayer._load_from_state_dict never runs and it reaches torch's default path, which builds a Parameter(..., requires_grad=...).

These tensors are BF16. loader.py only reshapes F32/F16, so BF16 stays as the raw uint8 mmap buffer — 2 bytes/element, hence the doubled last dim. The BF16 rescue directly below only covers len(shape) <= 1, and these are 2D. Torch therefore receives uint8 and raises.

The reported dim mismatch is a red herring: GGMLTensor.shape returns the logical shape (so torch's shape check passes), while the error text prints input_param.size() — the raw byte size.

Fix

GGMLLayer can only ever intercept weight/bias — everything else goes to unexpected_keys. So any other key must reach torch as a real float tensor.

Testing

On sulphur-2-distilled-Q4_K_M.gguf (arch ltxv, 4444 tensors):

  • Changes exactly 2 tensors. Of 292 non-weight/bias keys, 290 are F32 (skipped by the is_quantized guard) and 2 are the BF16 registers.
  • ~3 MB fp32 in a 14.1 GB model; 1772 tensors remain lazily quantized.
  • UnetLoaderGGUF now loads LTXAVModel; registers land as bfloat16 at (128, 2048) / (128, 4096).
  • Other archs unaffected — their non-weight/bias keys are F32, so is_quantized is False.

Note

#392 addresses this same class of bug for lumina2 by matching key names. Happy to narrow this to learnable_registers specifically if you'd prefer the surgical approach.

GGMLLayer only handles `weight` and `bias`, so bare nn.Parameters fall
through to torch's default _load_from_state_dict, which needs a real
float tensor. BF16 tensors skip the F32/F16 reshape and stay as raw
uint8 byte buffers, so LTX-2's *_embeddings_connector.learnable_registers
fail with "only Tensors of floating point dtype can require gradients"
and a doubled final dim.

Dequantize quantized tensors whose keys GGMLLayer cannot intercept.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant