A 4B driving vision-language model that runs on a consumer NPU and refuses when it cannot see — plus the controlled experiment showing the expensive part of the recipe did nothing.
Start with HANDOFF.md. It is the entry point: what is verified, what was overturned, what is impossible, and the traps already hit.
Three 4B models, same architecture, same evaluation, differing only in training:
| DriveLM BLEU-4 | CODA corner cases | Held-out refusal | Clean false alarms | |
|---|---|---|---|---|
| Lookup table (no images, no model) | 42.39 | 5.95 | — | — |
| Conventionally fine-tuned | 53.00 | 0.24 | 0.0% | 0.0% |
| Ours (72B-teacher reasoning) | 48.33 | 13.16 | 100% | 0.0% |
| Matched control (same data, no reasoning) | 50.47 | 12.98 | 99.0% | 0.0% |
Read the rows against each other and three things fall out.
The conventionally fine-tuned model wins DriveLM and is useless on corner cases — 0.24, below what a lookup table with no images scores. It learned the benchmark, not the task.
Our model reverses that, and keeps its refusal behaviour on two sensor-failure modes it never saw in training, quantized to INT4 on an Intel NPU, at the same speed as the baseline.
But that refusal is narrower than it sounds, and we tested until it broke. It fires when the whole frame is destroyed (100%, including two unseen modes). It does not fire when only part of the view is lost: cover exactly the object a question asks about — 1.1% of the frame — and the model still answers 97% of the time, confabulating a moped where a bollard is hidden. It learned "this frame is unusable", not "I cannot see what you asked about."
The 72B teacher contributed nothing. The matched control — identical data with the reasoning field emptied — lands within noise on all five measurements, and the sign of the difference alternates (ours ahead on corner cases, refusal and grounding; the control ahead on calibration and content precision). A real effect would push one way. What produced the results is the training data: corner-case examples and corruption augmentation with refusal targets. Nobody needs a 72B teacher for this.
DriveLM is 49.9% reproducible from 79 canned strings. A lookup table of each question's most frequent training answer, with no images and no parameters, reproduces half the test answers verbatim and scores 42.39 BLEU-4 — 80% of what a fine-tuned model gets.
Two widely used metrics inherit that flaw:
- Grounding Score is floored. It is a ratio against normal performance, so its floor is whatever the benchmark gives away free. On DriveLM that is 86.8–91.3%, and every value we measured sat on it. On CODA the floor is 43.4% and our model measures 66.4% — the same metric, finally with range.
- ECE degenerates into accuracy. When confidence concentrates in one bin, ECE reduces to |mean confidence − mean accuracy|. It matched to four decimal places in every run. Re-measured on CODA the ordering reverses: 0.463 for our model against 0.829 for the baseline, which is 83% confident while correct on 0.05% of corner cases.
Runs on the Intel Core Ultra NPU in this workstation: 2.46 s to first token, 6.8 tokens/s, identical to the baseline — the refusal capability is free at inference time.
The CPU is faster than the NPU here (561 ms, 20.0 tok/s) and we make no efficiency claim for the NPU: our NPU loop is an unoptimised reference implementation, and the reasons to use an NPU in a vehicle — power draw, leaving the CPU free — are things we did not measure. The claim is that the model runs there, which the standard export path does not permit, and keeps its refusal behaviour when it does.
openvino_genai.VLMPipeline segfaults on NPU because optimum-cli exports
unbounded dynamic shapes the compiler rejects. edgedrive/deploy/npu_runner.py
pins every shape and drives the five submodels directly. Refusal survives:
100% on both held-out corruption types at INT4, 0% false alarms, confirmed
at n=60. INT8 buys nothing over INT4.
| Path | What |
|---|---|
HANDOFF.md |
Read first. Verified claims, overturned claims, traps |
paper/findings.md |
Every result with the check that established it |
paper/draft.md |
The paper |
paper/tables.md |
Generated — never hand-edited |
edgedrive/eval/blind_template.py |
The zero-parameter baseline |
edgedrive/eval/paper_tables.py |
Regenerates every table from outputs/*.json |
edgedrive/deploy/npu_runner.py |
Static-shape Qwen3-VL on the NPU |
outputs/invalid_*/ |
Results produced by bugs, kept visible rather than deleted |
python -m edgedrive.eval.paper_tables > paper/tables.md # regenerate all tables
grep -c MISSING paper/tables.md # what is still openTrained LoRA adapters are on the Hugging Face Hub (private) — see HANDOFF.md.
Base model: Qwen/Qwen3-VL-4B-Instruct.
Every headline figure was re-derived from raw predictions rather than read out of a metrics file. That found a real error: clean-image false alarms were published as 2.7% when the true value is 0.0%, because refusal was being judged on the whole generation and our model's reasoning says things like "partially obscured by a fence" while correctly answering "Black sedan."
Three other results were produced by bugs and are kept in outputs/invalid_*/
rather than deleted, so the mistakes stay visible next to the corrections.