feat(search): hybrid BM25 + vector ranking via RRF (promote lexical to first-class)

docs/ARCHITECTURE.md

@@ -68,7 +68,7 @@ java_codebase_rag/pipeline.py
 
 ```
 MCP tool call (server.py)  ──asyncio.to_thread──▶  mcp_v2.*
-  ├─ search ─▶ search_lancedb.run_search    (vector / hybrid; optional graph-expand + RRF rank fusion)
+  ├─ search ─▶ search_lancedb.run_search    (vector / hybrid; graph-expand + 3-list RRF: vector + graph + BM25)
   │            └─ lancedb import absent (Intel Mac) → search_lexical (BM25 over Symbol FTS index; heuristic scan fallback)
   ├─ find / describe / neighbors ─▶ ladybug_queries.LadybugGraph   (Cypher)
   └─ resolve ─▶ resolve_service.resolve_v2   (cascade → status one | many | none)
@@ -78,14 +78,16 @@ MCP tool call (server.py)  ──asyncio.to_thread──▶  mcp_v2.*
 
 | Tool | Backing | Notes |
 | --- | --- | --- |
-| `search` | Lance vector/hybrid, or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring` |
+| `search` | Lance vector/hybrid with 3-list RRF (vector + graph + BM25), or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring`; list-set + `k` via injectable `RankConfig` |
 | `find` | Ladybug Cypher | required `NodeFilter`; strict per-kind frame |
 | `describe` | Ladybug Cypher | node record + `edge_summary` (composed/override rollups) |
 | `neighbors` | Ladybug Cypher | one hop; `direction` + `edge_types` required; dot-key composed edges |
 | `resolve` | Ladybug Cypher | per-kind generators exact→fuzzy; cap 10 candidates |
 
 **Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_lexical` (helpers from `search_scoring`). The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs.
 
+**BM25 is also first-class on the primary (vector) path, not only the fallback.** `search_lancedb._graph_expand_merge` fuses **three** RRF lists — vector hits + graph-expand hits + BM25 hits — where the BM25 list is sourced from the same `sym_fts` index (via `search_lexical._try_fts_candidates`), resolved to chunk rows in BM25 rank order and re-filtered by the same LanceDB predicates as the vector list. The list-set and RRF `k` are runtime-injectable via `RankConfig` (`search_scoring.py`; default = 3-list, `k=60`), so the eval can A-B 2-list vs 3-list and sweep `k`. If the FTS extension/index is unavailable, the BM25 list is empty and the fusion degrades silently to the 2-list vector+graph ranking (no exception, no advisory) — so airgapped installs see no regression. Quality is measured by the **eval harness** (`java_codebase_rag.eval`: recall@k / precision@k / MRR over a corpus, with a Tier-A auto ground-truth derived per-Symbol and an optional Tier-B operator-authored file). On shopizer (n=400 of 2322 type-level symbols) the 3-list fusion at `k=60` decisively beats the 2-list baseline on every metric: **MRR 0.3044→0.6205 (+104%)**, **recall@1 0.220→0.490 (+123%)**, recall@10 0.535→0.860 (+61%), recall@20→0.905. The gain is large because Tier-A queries are identifier-derived (BM25's home turf) — exact-identifier matches anchor the dense ranking exactly as the hybrid thesis predicts; a future Tier-B natural-language ground truth is expected to show a smaller-but-positive delta (NL queries favor semantic vectors). The BM25 hop costs **~+25-30 ms p50 (~10%)** per query. `k∈{30,60,90,120}` all beat baseline; `k=30` narrowly edges `k=60` on this identifier-heavy eval (MRR 0.631 vs 0.620, within noise on n=400), but `k=60` ships as the conservative, regime-robust choice — re-tune when Tier-B NL ground truth exists. (An initial eval run reported a much smaller delta; that was muted by a query-preprocessing bug — camelCase identifiers weren't reaching the FTS tokenizer — fixed in `search_lancedb._bm25_candidate_rows` via `search_scoring.build_fts_query`.)
+
 ### Watch path (`jrag watch`) — warm reads + freshness
 
 When a `jrag watch` daemon is running, the **read path gains a warm hop**: the `jrag` read handlers ask the daemon over a Unix socket for the already-built payload instead of cold-loading the model and graph. The daemon reuses the MCP server's warm-cache posture — a process-singleton `_st_model` (SBERT) and a `LadybugGraph` — served to the CLI, so each query skips the per-call torch/model load. Output is byte-identical to the cold path (the same payload cores in `read_payloads.py` run either way). With no daemon running, the client transparently takes the cold path — the daemon is a pure accelerator, never a dependency.

docs/DESIGN.md

@@ -19,7 +19,7 @@ One repo, two stores, two audiences:
 ## Core principles
 
 1. **Deterministic extraction, not LLM extraction.** tree-sitter parses every file; a two-phase build (parse all nodes, then resolve edges against the complete registry) eliminates forward-reference gaps. Reproducible, runs in seconds, no ~30% silent file-skip rate. (DKB = Deterministic Knowledge Base; benchmark + rationale in `docs/paper`.)
-2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal.
+2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. Lexical (BM25) is a **first-class** third signal on the primary path, not just the macOS-Intel fallback: the vector read path fuses vector + graph-expand + BM25 (Okapi BM25 over the `Symbol.search_text` FTS index), because exact-identifier matches (`DistributionChunkService`) are where dense embeddings are weakest and BM25 is strongest — the standard hybrid win for code retrieval. On shopizer (identifier-derived Tier-A ground truth) this **more than doubles** MRR and recall@1 vs the 2-list baseline (exact-identifier anchoring), at a modest ~10% per-query latency cost; the list-set and RRF `k` are runtime-injectable (`RankConfig`) and measured by an in-repo eval harness (recall/precision@k, MRR).
 3. **Walk at read time; don't precompute answers.** `neighbors` is exactly one hop. Multi-hop traces, impact analysis, "explain feature X" are the **agent's** reasoning over repeated one-hop calls. There is deliberately no magic impact/trace tool.
 4. **Static analysis is a lower bound.** `CALLS` excludes reflection, Spring AOP proxies, dynamic dispatch. `resolved=false` means *external* (JDK/Spring), not *missing*. Never present this as proof of a runtime call path.
 5. **Empty results must be honest, not silent.** Every empty hit is classified: `correct_empty` (genuine leaf), `not_in_project`, `external_dependency`, or `refine_query` — with did-you-mean, vocabulary context, and (for hard absence) an auditable proof.