From 3859ed5926ee84ff1dafb68dd72accd8a130ba69 Mon Sep 17 00:00:00 2001
From: Dmitry Teryaev <doudmitry@gmail.com>
Date: Sat, 11 Jul 2026 23:22:34 +0300
Subject: [PATCH] fix(search): lexical BM25 review follow-up (camelCase query +
 fallback + tests + docs)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Follow-up to #432 (BM25 lexical fallback via LadybugDB FTS), applying the
/requesting-code-review findings that came in after #432 merged. Reviewers
found no Critical issues; the one real bug now fixed:

- B1 (recall regression, now in master via #432): pasted camelCase queries
  (e.g. "DistributionChunkService") returned [] — LadybugDB FTS's tokenizer
  does not split camelCase, and the raw query was passed to QUERY_FTS_INDEX
  while Symbol.search_text is built from _split_identifier tokens. The
  heuristic fallback didn't save it (it also doesn't split camelCase). Fixed
  via a shared build_fts_query (search_scoring) that pre-splits the query the
  same way search_text is indexed; added a regression test.

- B2/B3: degenerate queries and empty BM25 results now fall back to the
  heuristic scan (preserves the role-ranked listing for degenerate queries;
  defensive against stemming / selective-filter-thinning cases).

- C1: test_cap_advisory_silent_below_cap now forces the heuristic path so it
  isn't passed vacuously on the BM25 default.
- C2: added a Symbol.search_text schema-anchor test (guards against a silently
  empty search_text column breaking all searches with zero test signal).

- Docs: swept 6 stale ontology-18 references (AGENT-GUIDE, ARCHITECTURE x2,
  DESIGN, CONFIGURATION x2) to 19; fixed the re-rank module attribution
  (search_lexical, not search_scoring) and added `package` to the search_text
  composition.

Review push-back: one reviewer claimed the LadybugDB constraints behind the
defensive build-side code (DROP TABLE refused while an FTS index references
it; DELETE needs LOAD EXTENSION FTS) "don't reproduce" on the shipped version.
Both were re-proved empirically (the DROP refusal reproduced in a fresh probe;
the DELETE-needs-LOAD was the exact failure behind 22 incremental-test fixes
in #432) — the defensive code stays.

Bench (~/jrag-bench/shopizer): query mean ~26ms post-fix (the pre-split adds
~2ms; still ~3.3x faster than the heuristic-only baseline of 82.8ms); build
phases unchanged. 21 lexical tests green.

Co-Authored-By: Claude <noreply@anthropic.com>
---
 docs/AGENT-GUIDE.md                           |  2 +-
 docs/ARCHITECTURE.md                          |  6 +--
 docs/CONFIGURATION.md                         |  6 +--
 docs/DESIGN.md                                |  2 +-
 .../search/search_lexical.py                  | 28 ++++++++---
 .../search/search_scoring.py                  | 26 ++++++++++
 tests/search/test_search_lexical.py           | 49 ++++++++++++++++++-
 7 files changed, 101 insertions(+), 18 deletions(-)

diff --git a/docs/AGENT-GUIDE.md b/docs/AGENT-GUIDE.md
index 860b1be..934bbf2 100644
--- a/docs/AGENT-GUIDE.md
+++ b/docs/AGENT-GUIDE.md
@@ -14,7 +14,7 @@ Copy the block between `<!-- BEGIN` and `<!-- END` into your project's `AGENTS.m
 
 **Indexed content:** Java production sources plus SQL and YAML (use `search` `table`: `java`, `sql`, `yaml`, or `all`).
 
-**Ontology: 18** — if results look structurally wrong or empty across tools, the index may be missing, stale, or built with a different `ontology_version`; you cannot re-index via MCP — ask the operator to rebuild.
+**Ontology: 19** — if results look structurally wrong or empty across tools, the index may be missing, stale, or built with a different `ontology_version`; you cannot re-index via MCP — ask the operator to rebuild.
 
 **Responses:** On success, `search`, `find`, `describe`, `neighbors`, and `resolve` may include two top-level fields: `hints_structured` (≤5 suggested next-tool calls) and `advisories` (≤5 pure informational strings). Each `hints_structured` entry has `tool`, `args`, `actionable`, `label`, and `reason`. `actionable=true` means you can call the tool directly with `args`; `actionable=false` means partial/advisory — fill missing values or use as guidance. `reason` explains why the hint was emitted. `advisories` carry context education (fuzzy strategy warnings, role collision explanations, etc.) with no tool call suggestion. For `search`/`find`, echoed `limit`/`offset`. Hints are advisory; ignore them when `success` is false.
 
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 4468dc9..a6c854d 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -30,7 +30,7 @@ Core library = **top-level `.py` modules** (`py-modules`); the installable **`ja
 | Concern | Modules |
 | --- | --- |
 | Write path | `java_codebase_rag/cli.py`, `java_codebase_rag/pipeline.py`, `java_codebase_rag/lance_optimize.py`, `java_index_flow_lancedb.py`, `build_ast_graph.py` |
-| Parse + ontology | `ast_java.py` (`ONTOLOGY_VERSION=18`), `java_ontology.py` (`EDGE_SCHEMA` + label sets), `graph_enrich.py`, `chunk_heuristics.py` |
+| Parse + ontology | `ast_java.py` (`ONTOLOGY_VERSION=19`), `java_ontology.py` (`EDGE_SCHEMA` + label sets), `graph_enrich.py`, `chunk_heuristics.py` |
 | Read path | `server.py`, `mcp_v2.py`, `ladybug_queries.py`, `search_lancedb.py`, `search_lexical.py`, `search_scoring.py`, `resolve_service.py`, `java_codebase_rag/read_payloads.py` |
 | Hints + absence | `mcp_hints.py`, `graph_types.py`, `absence_types.py`, `absence_vocab.py`, `absence_diagnosis.py` |
 | Config + paths | `java_codebase_rag/config.py`, `path_filtering.py`, `index_common.py`, `brownfield_events.py` |
@@ -59,7 +59,7 @@ java_codebase_rag/pipeline.py
 ```
 
 - **`init`** — refuses a non-empty index dir (exit 2); full vectors + full graph.
-- **`increment`** — CocoIndex `memo=True` catch-up (changed files only) + **incremental graph**. Falls back to **full** rebuild on any of: no graph · `ontology_version < 18` · crash marker (`.graph_increment_in_progress`) · dependent expansion > 50 files.
+- **`increment`** — CocoIndex `memo=True` catch-up (changed files only) + **incremental graph**. Falls back to **full** rebuild on any of: no graph · `ontology_version < 19` · crash marker (`.graph_increment_in_progress`) · dependent expansion > 50 files.
 - **`reprocess`** — default = full vectors + full graph; `--vectors-only` / `--graph-only` selective (mutually exclusive). Exit semantics in `cli._reprocess_exit_code`.
 
 **Phantom nodes:** unresolved callees / supertypes (external libs, `java.lang`) become `Symbol` rows with `resolved=false` and empty filename — so every edge lands on *a* node. Skipped by dependent expansion and scoped deletion.
@@ -84,7 +84,7 @@ MCP tool call (server.py)  ──asyncio.to_thread──▶  mcp_v2.*
 | `neighbors` | Ladybug Cypher | one hop; `direction` + `edge_types` required; dot-key composed edges |
 | `resolve` | Ladybug Cypher | per-kind generators exact→fuzzy; cap 10 candidates |
 
-**Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_scoring`. The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs.
+**Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_lexical` (helpers from `search_scoring`). The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs.
 
 ### Watch path (`jrag watch`) — warm reads + freshness
 
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index 236dafa..1fa137f 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -339,7 +339,7 @@ Deeper documentation for the brownfield blocks (`role_overrides`, `route_overrid
 
 ## 3. Graph layer
 
-A deterministic property graph derived from tree-sitter Java parsing lives next to the LanceDB tables under the index directory (default `${JAVA_CODEBASE_RAG_INDEX_DIR:-./.java-codebase-rag}/code_graph.lbug`). Current ontology version: **18** (see [`EDGE-NAVIGATION.md`](./EDGE-NAVIGATION.md) for MCP-traversable edge shapes).
+A deterministic property graph derived from tree-sitter Java parsing lives next to the LanceDB tables under the index directory (default `${JAVA_CODEBASE_RAG_INDEX_DIR:-./.java-codebase-rag}/code_graph.lbug`). Current ontology version: **19** (see [`EDGE-NAVIGATION.md`](./EDGE-NAVIGATION.md) for MCP-traversable edge shapes).
 
 ### Node kinds
 
@@ -403,7 +403,7 @@ Resolution order for `microservice`:
 
 ### Re-index required when ontology changes
 
-Current ontology version is **18**. Any index built before this version must be rebuilt via `cocoindex update ... --full-reprocess -f` or a full `java-codebase-rag reprocess` (no selective flags) so vectors and graph stay aligned. Until re-indexed, the server defensively JSON-decodes string-form list columns so nothing explodes, but filters like `array_contains` will not work.
+Current ontology version is **19**. Any index built before this version must be rebuilt via `cocoindex update ... --full-reprocess -f` or a full `java-codebase-rag reprocess` (no selective flags) so vectors and graph stay aligned. Until re-indexed, the server defensively JSON-decodes string-form list columns so nothing explodes, but filters like `array_contains` will not work.
 
 Ontology **15** (CALLS-NOISE) adds `CALLS.callee_declaring_role`, `GraphMeta.pass3_unresolved_phantom_receiver` / `pass3_unresolved_chained`, and **supertype-walk dedup** at build time. PR-2 adds `edge_filter` on `neighbors`. **PR-3 (breaking):** receiver-failure sites (`chained_receiver`, unresolved-receiver `phantom`) are no longer `CALLS` rows — they live on `UnresolvedCallSite` + `UNRESOLVED_AT`. Default `neighbors(..., ['CALLS'])` returns fewer rows; use `include_unresolved=True` for a source-ordered interleaved transcript (`row_kind`), `describe(method_id).unresolved_call_sites` (capped), or `java-codebase-rag unresolved-calls list|stats`. Known-receiver-external JDK rows stay on `CALLS` with `resolved=false`.
 
@@ -454,7 +454,7 @@ Combined, these pull `processClientMessage` / `pickEligibleOperator` / `onOperat
 
 #### Graph-only (macOS Intel) lexical ranking
 
-On Intel Mac installs the vector stack is absent (see `README.md`), so `search` runs the **lexical backend** — keyword ranking over the symbol graph instead of embeddings, behind the same tool contract. It is **BM25-first**: at index time every `Symbol` gets a `search_text` column (camelCase-split tokens of name + fqn + signature + annotations + capabilities, tokenized with the same splitter the query path uses) and a LadybugDB FTS index (`sym_fts`, Okapi BM25, porter stemmer) over it. At query time `QUERY_FTS_INDEX` fetches the top-K candidates DB-side, which are then re-ranked in Python by the heuristic below and deduped by FQN.
+On Intel Mac installs the vector stack is absent (see `README.md`), so `search` runs the **lexical backend** — keyword ranking over the symbol graph instead of embeddings, behind the same tool contract. It is **BM25-first**: at index time every `Symbol` gets a `search_text` column (camelCase-split tokens of name + fqn + signature + annotations + capabilities + package, tokenized with the same splitter the query path uses) and a LadybugDB FTS index (`sym_fts`, Okapi BM25, porter stemmer) over it. At query time `QUERY_FTS_INDEX` fetches the top-K candidates DB-side, which are then re-ranked in Python by the heuristic below and deduped by FQN.
 
 The heuristic re-rank decides final order (what `--explain` reports as `relevance=` / `name=` / `type=` / `fqn=`, plus a `bm25=` component for the index score):
 
diff --git a/docs/DESIGN.md b/docs/DESIGN.md
index 08f29a6..50b37b1 100644
--- a/docs/DESIGN.md
+++ b/docs/DESIGN.md
@@ -23,7 +23,7 @@ One repo, two stores, two audiences:
 3. **Walk at read time; don't precompute answers.** `neighbors` is exactly one hop. Multi-hop traces, impact analysis, "explain feature X" are the **agent's** reasoning over repeated one-hop calls. There is deliberately no magic impact/trace tool.
 4. **Static analysis is a lower bound.** `CALLS` excludes reflection, Spring AOP proxies, dynamic dispatch. `resolved=false` means *external* (JDK/Spring), not *missing*. Never present this as proof of a runtime call path.
 5. **Empty results must be honest, not silent.** Every empty hit is classified: `correct_empty` (genuine leaf), `not_in_project`, `external_dependency`, or `refine_query` — with did-you-mean, vocabulary context, and (for hard absence) an auditable proof.
-6. **The ontology version is the contract.** `ONTOLOGY_VERSION` (currently **18**) gates incremental rebuilds, drives a read-time staleness guard, and tells the agent the index shape. A semantic extraction change bumps it; old indexes fall back to full rebuild.
+6. **The ontology version is the contract.** `ONTOLOGY_VERSION` (currently **19**) gates incremental rebuilds, drives a read-time staleness guard, and tells the agent the index shape. A semantic extraction change bumps it; old indexes fall back to full rebuild.
 7. **The file always wins.** When the index disagrees with the open source file, the index is presumed stale or partial. Mismatch is a signal to rebuild, not a fact to report.
 8. **Generated sources are first-class by default.** MapStruct / OpenAPI / protobuf / … are auto-detected **by content** (`@Generated`, header banners), indexed like hand-written code, and filterable (`exclude_generated`) — never silently down-ranked.
 
diff --git a/src/java_codebase_rag/search/search_lexical.py b/src/java_codebase_rag/search/search_lexical.py
index 1bf795b..24cf562 100644
--- a/src/java_codebase_rag/search/search_lexical.py
+++ b/src/java_codebase_rag/search/search_lexical.py
@@ -32,6 +32,7 @@
     _dedup_by_fqn,
     _query_tokens,
     _split_identifier,
+    build_fts_query,
 )
 
 if TYPE_CHECKING:
@@ -272,8 +273,10 @@ def run_lexical_search(
     BM25-first (fork A): when the LadybugDB ``sym_fts`` index exists, candidates are
     fetched DB-side via Okapi BM25 over ``Symbol.search_text`` (killing the bounded
     Python scan that silently missed matches past the cap on large repos) and then
-    re-ranked here by the name/type/fqn/role heuristic. Falls back to that heuristic
-    scan when the FTS index or extension is unavailable (older graph, offline first run).
+    re-ranked here by the name/type/fqn/role heuristic. The query is pre-split with the
+    same tokenizer as ``search_text`` so pasted camelCase identifiers match. Falls back
+    to the heuristic scan when the FTS index or extension is unavailable (older graph,
+    offline first run), or when the query is degenerate / the BM25 result is empty.
 
     Raises ``RuntimeError`` (message contains "lexical search unavailable") if no
     symbol graph exists — the caller maps that to a clean failure envelope. Returns
@@ -292,14 +295,23 @@ def run_lexical_search(
     g = graph or LadybugGraph.get()
 
     # --- candidate fetch: BM25 (FTS) preferred, heuristic scan fallback ---
+    # FTS indexes Symbol.search_text (camelCase-split tokens); pre-split the query the
+    # same way (build_fts_query) so a pasted identifier like "DistributionChunkService"
+    # matches — LadybugDB FTS's own tokenizer does not split camelCase. An empty split
+    # (degenerate / stopword-only query), an unavailable FTS index, OR an empty BM25
+    # result all fall back to the heuristic scan, which yields role-ranked output for
+    # degenerate queries and covers selective filters where BM25's top-K thins to nil.
     bm25_scores: dict[str, float] = {}
     use_fts = False
-    fts = _try_fts_candidates(g, query, filter, path_contains)
-    if fts is not None:
-        rows = fts["rows"]
-        bm25_scores = fts["scores"]
-        use_fts = True
-    else:
+    rows: list[dict] | None = None
+    q_fts = build_fts_query(query)
+    if q_fts:
+        fts = _try_fts_candidates(g, q_fts, filter, path_contains)
+        if fts is not None and fts["rows"]:
+            rows = fts["rows"]
+            bm25_scores = fts["scores"]
+            use_fts = True
+    if rows is None:
         where, params = _lexical_where(filter, path_contains=path_contains)
         # Always exclude structural Symbol nodes. Files and packages are :Symbol-labeled
         # (kind='file'/'package') but aren't searchable code declarations — without this
diff --git a/src/java_codebase_rag/search/search_scoring.py b/src/java_codebase_rag/search/search_scoring.py
index 8d539ec..b94336d 100644
--- a/src/java_codebase_rag/search/search_scoring.py
+++ b/src/java_codebase_rag/search/search_scoring.py
@@ -136,6 +136,32 @@ def _split_identifier(name: str) -> list[str]:
     return [p for p in parts if p]
 
 
+# Alphanumeric-word extractor for FTS query building (mirrors the index side, which
+# runs the same regex over name/fqn/signature/annotations/capabilities/package fields).
+_FTS_WORD_RE = re.compile(r"[A-Za-z][A-Za-z0-9]*")
+
+
+def build_fts_query(text: str) -> str:
+    """Tokenize a search query into the ``Symbol.search_text`` token space (fork A).
+
+    ``search_text`` is indexed from ``_split_identifier`` tokens (camelCase / snake_case
+    split, lowercased). LadybugDB FTS's own tokenizer does NOT split camelCase, so a raw
+    pasted identifier like ``DistributionChunkService`` would match nothing — this
+    extracts alphanumeric words from the query and splits each via ``_split_identifier``
+    so the query lands in the index's token space. Tokens shorter than 2 chars are
+    dropped (mirrors the index side); duplicates collapse. Returns ``""`` for a query
+    with no usable tokens — the caller then falls back to the heuristic / role listing.
+    """
+    out: list[str] = []
+    seen: set[str] = set()
+    for word in _FTS_WORD_RE.findall(text or ""):
+        for tok in _split_identifier(word):
+            if len(tok) >= 2 and tok not in seen:
+                seen.add(tok)
+                out.append(tok)
+    return " ".join(out)
+
+
 def _symbol_bonus(r: dict, query_toks: set[str]) -> float:
     """Symbol-name overlap + action-verb bump for java chunks.
 
diff --git a/tests/search/test_search_lexical.py b/tests/search/test_search_lexical.py
index 8bb5bab..a9f1b5c 100644
--- a/tests/search/test_search_lexical.py
+++ b/tests/search/test_search_lexical.py
@@ -242,10 +242,15 @@ def test_cap_truncation_advisory_fires_at_cap(monkeypatch, tmp_path: Path) -> No
     assert any("repo cap" in a for a in adv), f"cap advisory should fire; got {adv}"
 
 
-def test_cap_advisory_silent_below_cap(tmp_path: Path) -> None:
-    """The cap advisory must NOT fire when the pool fits under the cap (no truncation)."""
+def test_cap_advisory_silent_below_cap(monkeypatch, tmp_path: Path) -> None:
+    """Heuristic-fallback path: the cap advisory must NOT fire when the pool fits under
+    the cap (no truncation). Forces the heuristic scan (the BM25 path has no cap) so the
+    invariant is actually exercised, not passed vacuously (fork A)."""
+    from java_codebase_rag.search import search_lexical
+
     db = _build_corpus(tmp_path)
     g = _graph(db)
+    monkeypatch.setattr(search_lexical, "_try_fts_candidates", lambda *a, **k: None)
     adv: list[str] = []
     run_lexical_search("distribution", limit=5, graph=g, advisories=adv)
     assert adv == [], f"no advisory expected for a small pool; got {adv}"
@@ -390,3 +395,43 @@ def test_bm25_path_respects_role_filter(tmp_path: Path) -> None:
     assert rows, "expected the controller hit through the role filter"
     assert all(r["role"] == "CONTROLLER" for r in rows), [r["role"] for r in rows]
     assert rows[0]["fqn"] == "ctrl.OperatorSessionController"
+
+
+def test_symbol_search_text_populated_for_bm25(tmp_path: Path) -> None:
+    """Fork A schema anchor: resolved Symbols carry a non-empty ``search_text`` whose
+    tokens include the camelCase-split name, so BM25 has something to index. Guards
+    against a regression that silently empties ``search_text`` (a broken
+    ``_compute_symbol_search_text`` or a dropped COPY/SET field would make searches
+    silently return nil with zero test failures)."""
+    import ladybug
+    from java_codebase_rag.search.search_scoring import _split_identifier
+
+    db = _build_corpus(tmp_path)
+    conn = ladybug.Connection(ladybug.Database(str(db), read_only=True))
+    res = conn.execute(
+        "MATCH (s:Symbol) WHERE s.kind <> 'file' AND s.kind <> 'package' "
+        "RETURN s.name AS name, s.search_text AS st"
+    )
+    seen_class = False
+    while res.has_next():
+        name, st = res.get_next()
+        assert st, f"search_text empty for Symbol {name!r}"
+        if name and name[:1].isupper():  # class/type names (not <init>, methods)
+            toks = [t for t in _split_identifier(str(name)) if len(t) >= 2]
+            assert all(t in st.split() for t in toks), (
+                f"{name!r} tokens {toks} missing from search_text {st!r}"
+            )
+            seen_class = True
+    assert seen_class, "no class Symbols found in fixture"
+
+
+def test_bm25_path_handles_pasted_camelcase_identifier(tmp_path: Path) -> None:
+    """Fork A regression guard (review B1): a pasted camelCase identifier like
+    'DistributionChunkService' must match on the BM25 path — LadybugDB FTS does not split
+    camelCase, so the query is pre-split via build_fts_query before QUERY_FTS_INDEX.
+    Without the pre-split this returns []."""
+    db = _build_corpus(tmp_path)
+    rows = run_lexical_search("DistributionChunkService", limit=5, graph=_graph(db))
+    assert rows, "pasted camelCase identifier must match via the pre-split BM25 query"
+    assert rows[0]["fqn"] == "svc.DistributionChunkService"
+    assert "bm25" in rows[0]["_score_components"]