feat(search): BM25 lexical fallback via LadybugDB FTS

[HumanBean17]

Problem

PR #403 shipped a lexical search fallback for macOS Intel (where the vector stack can't install), ranking by a hand-rolled keyword-overlap heuristic capped at a 5000-symbol Python-side scan (search_lexical.py:_CANDIDATE_LIMIT_CAP). On any real enterprise repo that cap is a rounding error — the best match is silently dropped past row 5000. The PR note said "upgradable to SQLite FTS5 later," which pointed at the wrong store.

Change

The graph DB is LadybugDB (the embedded Cypher/GNN store that is kuzu's ancestor, not a kuzu wrapper), and it ships a native FTS extension that scores with Okapi BM25 in the same code_graph.lbug the graph already lives in — no new dependency, no sidecar store. This PR makes the lexical path BM25-first:

Build (build_ast_graph.py)

• New Symbol.search_text STRING column: camelCase-split token soup of name + fqn + signature + annotations + capabilities, built with the same _split_identifier the query path uses (so index- and query-time tokenization agree).
• sym_fts FTS index (Okapi BM25, porter stemmer), ensured at the end of every full build and as a self-heal on incremental. It auto-maintains on COPY/MERGE/DELETE, so increment stays fresh without drop+recreate.
• _drop_all drops the FTS index first (LadybugDB refuses DROP TABLE on a table referenced by an FTS index); incremental_rebuild and _delete_file_scope LOAD EXTENSION FTS so Symbol DML can maintain the index.
• ONTOLOGY_VERSION 18 → 19 (semantic change → one-time reindex).

Query (search_lexical.py)

• QUERY_FTS_INDEX fetches the top-K candidates DB-side, a re-MATCH re-applies the NodeFilter (role / module / path / kind≠file,package) so filter logic stays in one place, then the existing name/type/fqn/role heuristic re-ranks and _dedup_by_fqn runs as before. The bounded Python scan is now the fallback for when the FTS index or extension is unavailable (older graph, or an offline first run where INSTALL FTS can't fetch from extension.ladybugdb.com).
• Per-connection FTS-load cache keyed by the connection object (WeakSet), not id() — id() is reused after GC and broke under test batching.
• No tool-contract change (SearchHit / SearchOutput / NodeFilter unchanged); _score_components gains a bm25 entry.

Tests + docs: BM25-active / heuristic-fallback / role-filter tests; two stale ontology_version == 18 assertions and the bank-chat baseline fixture bumped to 19; ARCHITECTURE / CONFIGURATION / AGENT-GUIDE / README updated (incl. the Kùzu→LadybugDB correction).

Verification

• Affected suites green; full suite green (the only failures during development were 65 tests/package errors from the uv worktree venv lacking pip — environment, not code — and hardcoded ontology == 18 assertions, all bumped).

• Probed the two make-or-break questions directly: the read-only LadybugGraph connection can LOAD EXTENSION FTS + QUERY_FTS_INDEX, and the FTS index auto-maintains on COPY/DELETE.

• Bench on ~/jrag-bench/shopizer (master vs branch, bench_graph.py):

  metric	master (heuristic)	fork A (BM25)	Δ
  init wall	34.65s	30.19s	flat (variance)
  reprocess --graph-only	7.63s	7.21s	flat
  increment (1-file)	18.98s	17.30s	flat
  query mean	82.8ms	23.6ms	−71% (3.5×)
  query max	111.7ms	25.0ms	−78%

  Build phases are single-run variance (the vectors phase dominates init and is untouched; the FTS-index build on shopizer-scale is negligible). The query win is the headline.

Notes

• Offline/airgapped: INSTALL FTS fetches from extension.ladybugdb.com on first use (cached after). Operators on airgapped hosts need a bundled/local-path install — flagged as open question on feat(search): hybrid BM25 + vector ranking via RRF (promote lexical to first-class) #431; the heuristic fallback covers them until then.
• Fork B (fold this BM25 signal into the primary vector/hybrid path as a third RRF list) is tracked in feat(search): hybrid BM25 + vector ranking via RRF (promote lexical to first-class) #431, blocked on this PR.

🤖 Generated with Claude Code