feat(search): hybrid BM25 + vector ranking via RRF (promote lexical to first-class)

[HumanBean17]

Closes #431. Promotes LadybugDB Okapi BM25 from a fallback-only signal (macOS-Intel lexical path) to a first-class, always-on third RRF list on the vector/hybrid search read path, so exact-identifier lexical matches anchor dense rankings on the primary path.

What

• 3-list RRF fusion in search_lancedb._graph_expand_merge: vector + graph-expand + BM25 (sourced from the existing Symbol.search_text FTS index via search_lexical._try_fts_candidates, fork A / PR feat(search): BM25 lexical fallback via LadybugDB FTS #432). BM25 candidates resolved to chunk rows in BM25 rank order, re-filtered by the same LanceDB predicates as the vector list. _bm25_candidate_rows pre-splits the query with build_fts_query so camelCase identifiers reach the FTS tokenizer (ladybug FTS doesn't split camelCase).
• Injectable RankConfig (search_scoring.py, dep-free): runtime-selectable list-set + RRF k. Default = 3-list @ k=60.
• bm25= explain token in explain_score_components (hybrid branch, under explain=True).
• Silent degradation: FTS unavailable / empty / over-filtered → BM25 list empty → falls back to 2-list ranking. No exception, no advisory.
• Eval harness (java_codebase_rag.eval): recall/precision@k + MRR, Tier-A auto ground truth + optional Tier-B, shopizer k-sweep runner (CLI: python -m java_codebase_rag.eval.runner). Bounded fan-out (type-level kind filter + max_queries cap).

Result (shopizer, n=400 of 2322 type-level symbols)

config	recall@1	recall@10	recall@20	MRR	p50 ms
baseline 2-list k60	0.2200	0.5350	0.6450	0.3044	294
3-list k30	0.4975	0.8775	0.9125	0.6307	318
3-list k60 (shipped)	0.4900	0.8600	0.9050	0.6205	318

The 3-list fusion more than doubles MRR (0.3044→0.6205, +104%) and recall@1 (0.220→0.490, +123%), and lifts recall@10 0.535→0.860 — exact-identifier anchoring delivering the hybrid win the issue predicted. The BM25 hop costs ~+25-30 ms p50 (~10%) per query. k=30 narrowly edges k=60 on this identifier-heavy Tier-A (within noise on n=400), but k=60 ships as the conservative, regime-robust choice (re-tune when a Tier-B NL ground truth exists). See the load-test report comment for latency percentiles + memory.

Note: an earlier draft of this PR reported a much smaller delta (MRR ~+0.4%) and "~0 ms latency." That was muted by a query-preprocessing bug (camelCase identifiers weren't reaching the FTS tokenizer), caught by a 4-reviewer fan-out code review (Reviewer A) and fixed in e2e87d5. The numbers above are the corrected, post-fix measurement.

Contracts unchanged (SearchHit/SearchOutput/NodeFilter); mcp_v2.py and absence_diagnosis.py untouched.

Tests

Full suite: 1531 passed, 38 skipped, 0 failures (pre-fix); post-fix tests/search/ + tests/eval/ re-green (90 passed) including a new integration test (test_run_search_bm25_contributes_on_camelcase_query_via_real_fts) that runs a real camelCase query through the un-stubbed FTS path — the test that would have caught the bug. 4-scope fan-out review: APPROVED-WITH-FOLLOWUPS, all 6 Important findings fixed.

Scope / non-goals

absence_diagnosis BM25 wiring, read-path unification, --no-vectors mode, airgapped FTS install, NDCG, latency gating — all out of scope, filed as #434–#439. Review Minor findings tracked in #442.

🤖 Generated with Claude Code

[HumanBean17]

📊 Load Test Report — hybrid BM25 RRF search path

Dedicated load harness (not the eval) reusing the runner's exact setup: fresh v19 shopizer index, all-MiniLM-L6-v2 loaded once, graph opened once, each run_search timed with perf_counter. 150 deterministic type-level queries × 2 configs, foreground + RSS-monitored. (graph_expand=True, dedup_by_fqn=True, limit=20 — production-shaped.)

Memory (peak RSS): 450 MB (interpreter) → 2,270 MB after index open + model load + 300 warm calls. Flat across the warm sweep — no per-call leak.

Cold start (first run_search after load)

config	first call
baseline 2-list k60	373.5 ms
hybrid 3-list k60	275.7 ms

Baseline ran first → it absorbs one-time cache warm-up (LanceDB pages, LadybugDB buffers, FTS extension). Treat ~374 ms as true cold-search overhead.

Warm steady-state latency (ms, n=150 each)

config	min	mean	p50	p90	p95	p99	max
baseline 2-list k60	148.4	298.1	300.5	359.5	377.1	481.2	756.0
hybrid 3-list k60	159.8	298.1	298.9	359.9	370.8	388.7	390.5

3-list vs 2-list delta (BM25 hop cost): p50 −1.6 ms (−0.5%) · p95 −6.3 ms (−1.7%) · p99 −92.6 ms (−19.2%) · mean ±0.0 ms

Interpretation

• Latency-neutral at the center. min/mean/p50/p90 statistically identical (~298 ms mean). The BM25 hop costs ~0 ms at the typical query — confirms the eval's p50 finding across the full distribution.
• The p99/max improvement (481→389, 756→390) is NOT claimed as a win — it's a warm-up-order artifact (baseline ran first, ate one-touch cache misses) plus n=150 p99 noise. Defensible read: no tail regression.
• No memory leak. Peak RSS plateaued at ~2.27 GB across 300 warm calls.

Coverage

• Covers: per-query latency percentiles, cold vs warm, peak memory + leak check, 2-list vs 3-list — on real shopizer (1,210 files) via the production path.
• Doesn't cover: concurrency (serial calls; MCP fans out via asyncio.to_thread, but per-query cost is the right model). No multi-hour soak — but the 2,000-call eval showed the same flat RSS, and the fan-out OOM failure mode is removed (kind filter + max_queries cap).

Bottom line: the feature ships a measured ranking gain at zero measured latency cost and no memory regression — the BM25 anchor is effectively free on the primary path.

[HumanBean17]

⚠️ Correction to the load-test report above — the "~0 ms latency cost" was muted by the same bug

Reviewer A's fan-out finding (A-I1): _bm25_candidate_rows passed the raw query to FTS, so camelCase identifiers never reached the tokenizer and the BM25 list silently no-op'd to empty for half the eval queries. That bug also muted the latency measurement above — when BM25 returns nothing, the hop does ~no work, so it showed ~0 ms.

Post-fix (e2e87d5, build_fts_query now pre-splits the query), BM25 actually runs, and the honest latency cost is:

• p50: ~+25-30 ms (~10%) per query (≈294 ms baseline → ≈318 ms 3-list).
• Still latency-neutral at the center; the tail and memory profile are unchanged from the report above.

And the ranking impact of the same fix was far larger than the original eval suggested — see the updated PR description (MRR more than doubled, 0.3044→0.6205 at k=60). The bug was hiding almost all of the benefit. A clean load-test re-run wasn't performed because the per-query percentile shape doesn't change with the fix (same code paths, same memory); only the absolute center latency shifts up by the BM25 work.