server: do not propagate --kv-mean-center bias to draft model contexts#81
Open
cnndabbler wants to merge 1 commit into
Open
server: do not propagate --kv-mean-center bias to draft model contexts#81cnndabbler wants to merge 1 commit into
cnndabbler wants to merge 1 commit into
Conversation
The bias GGUF is calibrated for the target model's K geometry. When a draft model is loaded (-md), copying common_params wholesale made both the VRAM-measurement probe and the real draft context inherit kv_mean_center_path, failing with 'bias tensor ... has 1024 elements, expected 512' and disabling speculative decoding. Clear the path on the draft params in both sites so bias + drafter can coexist.
There was a problem hiding this comment.
Pull request overview
Prevents target-model KV mean-centering bias from being applied to incompatible draft-model contexts.
Changes:
- Clears the inherited bias path during draft VRAM measurement.
- Clears it when creating the actual draft context.
- Preserves bias behavior for the target and MTP contexts.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
--kv-mean-centersetscommon_params.kv_mean_center_pathglobally. The server builds draft-model params by copying the whole struct (auto params_dft = params_base;), so when a draft model is loaded via-md, both the draft VRAM-measurement probe and the real draft context inherit the bias path. The bias GGUF is calibrated for the target model's K geometry, so draft context creation fails:llama_init_from_model: path_kv_mean_center requires the K cache type to be Q4_0 (got f16)-ctkd q4_0:load_kv_mean_center: bias tensor kv_bar.blk.3.k has 1024 elements, expected 512 (n_embd_head_k * n_head_kv)Either way the probe reports
[spec] failed to measure draft model memoryand speculative decoding is silently disabled (common_speculative_init: no implementations specified) — making--kv-mean-centerand-mdmutually exclusive in practice.Fix
Clear
kv_mean_center_pathon the copied draft params at both construction sites intools/server/server-context.cpp(the VRAM-measurement probe and the real draft context), following the existing pattern of draft-specific overrides (cache_type_k/cache_type_v). The main context keeps the bias.Coverage note: the only consumption site is
common_context_params_to_llama(common/common.cpp), and these are the only two non-main-context derivations ofcommon_paramsin the server for the-mdpath (the MTP branch shares the target model and is untouched).Verified
RTX A5000, Ternary-Bonsai-27B Q2_0 + dspark drafter +
--kv-mean-center+ q4_0 KV (-ctkd/-ctvd q4_0), based on62061f9:load_kv_mean_centererrorscommon_speculative_impl_draft_dspark: adding speculative implementation 'draft-dspark'— spec active alongside the bias🤖 Generated with Claude Code