From 8368e711d4a47f0b738f2a512ef0ffad5c169227 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 00:25:47 +0300 Subject: [PATCH 01/15] docs(spec): hybrid BM25 + vector RRF design (issue #431, fork B) Approved design for promoting LadybugDB BM25 to a first-class third RRF list on the vector/hybrid read path, plus a from-scratch recall/precision @k + MRR eval harness over ~/jrag-bench/shopizer. Co-Authored-By: Claude --- .../2026-07-12-hybrid-bm25-rrf-design.md | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md diff --git a/docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md b/docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md new file mode 100644 index 0000000..b14c712 --- /dev/null +++ b/docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md @@ -0,0 +1,119 @@ +# Hybrid BM25 + Vector Ranking via RRF (fork B, issue #431) + +**Status:** Active — design approved 2026-07-12. +**Tracks:** [issue #431](https://github.com/HumanBean17/java-codebase-rag/issues/431). +**Depends on:** Fork A (PR #432, merged) — `Symbol.search_text` column + `sym_fts` LadybugDB FTS index at ontology v19. **Satisfied.** + +## Summary + +Promote LadybugDB Okapi BM25 from a fallback-only signal (macOS-Intel lexical path) to a first-class, always-on **third input** to the vector/hybrid read path's RRF (Reciprocal Rank Fusion). Exact-identifier lexical matches then anchor dense rankings on the primary path. + +``` +final_rank = RRF([vector_hits, graph_expand_hits, bm25_hits], k=tuned) +``` + +## Background & current state + +- The vector path (`search/search_lancedb.py`) fuses **2 lists** — `vector_rows` + `graph_rows` — in `_graph_expand_merge` via `_rrf_merge(k=60)`. +- `_rrf_merge` is already N-list generic; the only "2-ness" lives in (a) the caller and (b) the `_HYBRID_SCORE_MAX` constant in `search/search_scoring.py` (`2.0/61.0` term). +- Fork A shipped `Symbol.search_text` + `sym_fts` (ontology v19) and made BM25 the primary path in `search/search_lexical.py` via `_try_fts_candidates(g, query, filter, path_contains)`, returning `{rows, bm25_scores}`. The hand-rolled token-overlap scan (PR #403) survives only as the lexical fallback. +- `absence_diagnosis` is orthogonal (vocabulary q-gram + difflib did-you-mean); it does not consume BM25 today. +- No eval harness exists in-repo. + +## Goals + +- Add BM25 as an always-on third RRF list on the vector path. +- Make ranking config (`{list-set, k}`) runtime-injectable. +- Build a recall/precision @k + MRR eval harness over `~/jrag-bench/shopizer`. +- Tune RRF `k` for the 3-list fusion on that eval; ship the winner. +- Surface a `bm25` score component on the vector path under `explain=True`. + +## Non-goals + +- No change to `SearchHit` / `SearchOutput` / `NodeFilter` schemas (`score_components` is an open dict). +- No change to `absence_diagnosis` (deferred — see Follow-ups). +- No change to the `sym_fts` index or `search_text` column (fork A). +- No change to the LanceDB-internal `.text(fts_text)` single-table hybrid mode. +- No physical unification of the lexical-only and vector code paths (deferred). +- No `--no-vectors` distribution (deferred). +- No new operator-facing CLI flag for rank config (the injection is internal). + +## Architecture & data flow + +**Current:** +``` +search_v2 → search_lancedb.run_search + → vector query (LanceDB) → vector_rows + → graph-expand (LadybugGraph) → graph_rows + → _rrf_merge([vector_rows, graph_rows], k=60) +``` + +**Proposed:** +``` +search_v2 → search_lancedb.run_search + → vector query (LanceDB) → vector_rows + → graph-expand (LadybugGraph) → graph_rows + → BM25 FTS query (LadybugGraph sym_fts) → bm25_rows [always-on] + → _rrf_merge([vector_rows, graph_rows, bm25_rows], k=tuned) +``` + +- The BM25 fetch reuses `search_lexical._try_fts_candidates` — the same code the lexical backend uses. **No FTS-query, tokenizer, or filter-translation logic is duplicated.** `SYMBOL_FTS_INDEX` (search_scoring.py) remains the single source of truth for the index name. +- **Symbol→chunk resolution:** BM25 candidates are `Symbol` nodes; `_rrf_merge` dedups on `(filename, range_start, range_end)`. BM25 Symbols resolve to chunk(s) via the same expansion `graph_rows` already use. Orphaned Symbols (no chunk) are dropped. A Symbol mapping to multiple chunks expands to all (BM25 is a per-Signal, per-Symbol rank position). +- **FTS unavailable / empty query / over-restrictive filter:** the BM25 list is empty; 3-list `_rrf_merge` with one empty list degrades cleanly to current 2-list behavior. **No exception, no advisory.** This covers the airgapped case silently. +- **macOS-Intel lexical-only path:** behavior unchanged in this PR. Conceptually "hybrid with an empty vector list," but not physically unified (deferred). + +## Components + +| File | Change | +|---|---| +| `search/search_lancedb.py` | `_graph_expand_merge` (or a new sibling helper) calls `search_lexical._try_fts_candidates` → `bm25_rows`; resolves Symbol→chunk; passes `[vector_rows, graph_rows, bm25_rows]` to `_rrf_merge`. Reads `{list-set, k}` from an injected rank config (default = 3-list, tuned-k). Populates `row["_score_components"]["bm25"]`. | +| `search/search_scoring.py` | `_HYBRID_SCORE_MAX` RRF term derived from list count (`num_lists/(k+1)`) instead of hard-coded `2.0/61.0`. `explain_score_components` gains a `bm25=` token in the hybrid branch. | +| `search/search_lexical.py` | No behavioral change. (Possibly expose `_try_fts_candidates` for reuse; minimal.) | +| `eval/` (new package) | New top-level package + CLI runner. Indexes shopizer into a temp dir; builds Tier-A ground truth; runs 2-list baseline, 3-list candidate, and k-sweep; writes Markdown + JSON results. | + +**Score components (vector path, `explain=True`):** adds `bm25` (raw Okapi BM25 score; `0` if the hit did not come from the BM25 list), alongside existing `rrf_raw`, `hybrid_rrf`, `role_weight`, `symbol_bonus`, `import_penalty`. + +**Rank-config contract (internal):** `_graph_expand_merge` accepts a config selecting list-set ∈ `{{vector,graph}, {vector,graph,bm25}}` and integer `k`. Production default = `{vector,graph,bm25}` at the tuned k. The eval and tests inject alternatives. + +## Eval harness + +- **Location:** new `eval/` package (not under `tests/`); invoked via a runner entry point. Not a CI pass/fail gate. +- **Corpus:** `~/jrag-bench/shopizer`, indexed into a temp dir (fresh, never committed), mirroring `tests/conftest.py` hygiene. +- **Ground truth — Tier A (auto-generated, ships with harness):** for each indexed `Symbol`, queries derived from name/FQN tokens (e.g. `"DistributionChunkService"`, `"distribution chunk service"`), `relevant = {that symbol}`. Deterministic, free, no manual authoring. Tests the identifier regime BM25 should dominate. +- **Ground truth — Tier B (user-authored, optional):** `~/jrag-bench/shopizer/ground_truth.{json,yaml}` of natural-language intent queries → relevant symbols. Loaded if present; reported separately. +- **Metrics:** Recall@k (k ∈ {1,5,10,20}); Precision@k; **MRR (primary)**; recall@10 as no-regression guardrail; recall@1 as tiebreak. Binary relevance (graded/NDCG deferred). +- **Comparisons (single pass, one index):** 2-list @ k=60 (baseline); 3-list @ k ∈ {30,60,90,120}. Latency (p50 ms) measured per config. +- **Output:** Markdown table + raw JSON to `~/jrag-bench/shopizer/results//`. +- **Win criterion:** ship the 3-list config at the k maximizing **MRR** with no regression on recall@10 vs baseline. Negative results are reported honestly, not hidden; if no k beats baseline, ship anyway (lexical-first-class is the goal) and open a follow-up. + +## Testing + +- **Unit (CI):** `_rrf_merge` over 3/N lists (dedup, `num_lists/(k+1)` normalization, empty-list degradation); `_HYBRID_SCORE_MAX` derivation for list counts 1/2/3; `bm25` component population + `explain=True` gating; `explain_score_components` `bm25=` token; filter-respect (BM25 Symbol outside `NodeFilter` dropped); rank-config injection honoring injected `{list-set, k}`. +- **Integration (CI):** no-regression on existing `tests/search/` fixtures; FTS-unavailable on vector path → silent 2-list degradation (monkeypatch FTS load, assert no exception/advisory). +- **Eval-harness (CI, lightweight):** Tier-A generator deterministic on a fixture; recall@k & MRR math unit-tested against hand-computed cases; full-harness smoke run on a tiny fixture asserts *output produced* (not specific ranking numbers). +- **Not CI-gated:** the actual shopizer recall/MRR numbers (research artifacts, non-hermetic). +- **Baseline:** develop against the search subset; full suite once at end. + +## Error handling & latency + +- FTS unavailable / degenerate query / over-restrictive filter → empty BM25 list → silent degradation to 2-list. No new exceptions, exit codes, or advisories. +- `QUERY_FTS_INDEX` is a DB-side indexed query returning `top:=200` rows, comparable to the existing graph-expand hop. The eval measures the per-search latency delta (2-list vs 3-list); gating/caching deferred pending that measurement. + +## Deferred / follow-ups (GitHub issues to be opened) + +1. BM25-fed did-you-mean in `absence_diagnosis` (with its own MRR eval). +2. Physically unify the lexical-only and vector read paths ("hybrid with empty vector list"). +3. `--no-vectors` distribution mode. +4. Airgapped/bundled FTS install path (`INSTALL FTS` fetches from `extension.ladybugdb.com` on first use). +5. NDCG@k with graded relevance. +6. Latency gating (query-shape) or BM25-result caching, if the eval shows material cost. + +## References + +- `search/search_lancedb.py:_graph_expand_merge` (RRF caller), `:_rrf_merge` (N-list core). +- `search/search_scoring.py:_HYBRID_SCORE_MAX`, `:SYMBOL_FTS_INDEX`, `:explain_score_components`, `:build_fts_query`. +- `search/search_lexical.py:_try_fts_candidates`, `:_ensure_fts_loaded`, `:_CANDIDATE_LIMIT_CAP`. +- `graph/build_ast_graph.py:_compute_symbol_search_text`, `:_ensure_symbol_fts_index`. +- `ast/ast_java.py:ONTOLOGY_VERSION` (=19). +- `mcp/mcp_v2.py:SearchHit`, `:SearchOutput`, `:NodeFilter`, `:_row_to_search_hit`. +- `absence/absence_diagnosis.py:diagnose`. From 071d8827acd90c65257657826dbcd5117f631e60 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 00:32:00 +0300 Subject: [PATCH 02/15] docs(plan): hybrid BM25 + vector RRF implementation plan (issue #431) 8-task TDD plan: scoring refactor, RankConfig injection, bm25= explain, BM25 candidate fetch + 3-list fusion, and a from-scratch eval harness (metrics, Tier-A/Tier-B ground truth, shopizer k-sweep runner). Co-Authored-By: Claude --- .../active/2026-07-12-hybrid-bm25-rrf.md | 412 ++++++++++++++++++ 1 file changed, 412 insertions(+) create mode 100644 docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md diff --git a/docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md b/docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md new file mode 100644 index 0000000..46bf1e2 --- /dev/null +++ b/docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md @@ -0,0 +1,412 @@ +# Hybrid BM25 + Vector RRF Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Promote LadybugDB Okapi BM25 to a first-class, always-on third RRF list on the vector/hybrid search read path, and build a recall/precision@k + MRR eval harness to measure and tune it. + +**Architecture:** The vector path (`search_lancedb._graph_expand_merge`) currently fuses 2 lists (vector + graph-expand) via the N-list-generic `_rrf_merge`. We add a third list — BM25 candidates from the existing `search_lexical._try_fts_candidates` helper (fork A, merged) — resolved to chunk rows in BM25 rank order, filter the result through the same LanceDB predicates as the vector path, and fuse with RRF. A new dep-free `RankConfig` makes `{lists, rrf_k}` runtime-injectable so the eval can A-B 2-list vs 3-list and sweep `k`. The `score_components["bm25"]` entry surfaces under `explain=True`. + +**Tech Stack:** Python 3.11 (`.venv/bin/python`, editable install), LadybugDB 0.17.1 FTS (Okapi BM25) via the `sym_fts` index, LanceDB 0.34, pydantic v2, pytest. + +**Spec:** `docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md` (commit `8368e71`). **Issue:** #431. **Deferred follow-ups:** #434–#439. + +## Global Constraints + +- Use `.venv/bin/python` and `.venv/bin/pip` only — never system `python`/`pip`. Editable install only; if pytest complains about a stale install, run `.venv/bin/pip install -e ".[dev]"`. +- `search/search_scoring.py` MUST NOT import lancedb / torch / sentence_transformers / cocoindex — it is imported on graph-only (macOS Intel) installs where those are absent. `RankConfig` and any new metric helpers that the eval imports without the vector stack MUST live there (or another dep-free module). +- `SYMBOL_FTS_INDEX = "sym_fts"` (`search_scoring.py:20`) is the single source of truth for the FTS index name — never inline the string. +- Erase stale manual indexes before running tests: `rm -rf tests/*/.java-codebase-rag tests/*/.java-codebase-rag.{yml,hosts}`. Tests build their own index in a temp dir; never commit one under `tests/`. +- Develop against the search subset (`-k "lexical or search or hybrid or rrf"`); run the full suite once at the end. +- `SearchHit` / `SearchOutput` / `NodeFilter` schemas (`mcp/mcp_v2.py`) are UNCHANGED — `score_components` is an open dict. +- FTS unavailable must degrade SILENTLY to 2-list ranking on the vector path: no exception, no advisory, no exit-code change. + +--- + +## File Structure + +| Path | Responsibility | Status | +|---|---|---| +| `src/java_codebase_rag/search/search_scoring.py` | Dep-free scoring/dedup primitives. Add `RankConfig` dataclass; derive `_HYBRID_SCORE_MAX` RRF term from list count; add `bm25=` token to `explain_score_components`. | Modify | +| `src/java_codebase_rag/search/search_lancedb.py` | Vector/hybrid read path. Add `_bm25_candidate_rows` helper; thread `rank_config` through `run_search` → `_graph_expand_merge`; pass BM25 list to `_rrf_merge`. | Modify | +| `src/java_codebase_rag/search/search_lexical.py` | Lexical backend. Possibly widen visibility of `_try_fts_candidates` for cross-module reuse. | Modify (minimal) | +| `src/java_codebase_rag/eval/__init__.py` | Eval package marker. | Create | +| `src/java_codebase_rag/eval/metrics.py` | Dep-free IR metrics: recall@k, precision@k, MRR. | Create | +| `src/java_codebase_rag/eval/ground_truth.py` | Tier-A auto ground-truth generator (Symbol→query). Tier-B YAML/JSON loader. | Create | +| `src/java_codebase_rag/eval/runner.py` | Build shopizer index, run 2-list baseline + 3-list k-sweep via injected `RankConfig`, write Markdown + JSON results. | Create | +| `tests/search/test_search_scoring.py` | Unit tests for `RankConfig`, `_HYBRID_SCORE_MAX` derivation, `bm25=` token. | Modify | +| `tests/search/test_search_lancedb.py` | Unit + integration tests for `_bm25_candidate_rows`, rank-config injection, FTS-unavailable degradation. | Modify | +| `tests/eval/test_metrics.py`, `tests/eval/test_ground_truth.py`, `tests/eval/test_runner.py` | Unit tests for eval harness machinery (metric math, generator determinism, runner smoke). | Create | + +--- + +## Task 1: Derive `_HYBRID_SCORE_MAX` from RRF list count + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py:87-93` +- Test: `tests/search/test_search_scoring.py` + +**Interfaces:** +- Produces: a new module-level function `_rrf_max(num_lists: int, k: int = 60) -> float` returning `num_lists / (k + 1)`. `_HYBRID_SCORE_MAX` is redefined to use `_rrf_max(2)` (preserving today's exact `2.0/61.0` value) PLUS the unchanged bonus terms (`max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS`). The numeric value of `_HYBRID_SCORE_MAX` MUST stay byte-identical to today (it still describes the shipped 2-list hybrid display path until Task 4 flips production to 3-list). +- Consumes: nothing new. + +- [ ] **Step 1: Write failing tests** + +In `tests/search/test_search_scoring.py` add: + +(a) `test_rrf_max_formula`: assert `_rrf_max(2, 60)` equals `2.0 / 61.0` (within 1e-12); `_rrf_max(3, 60)` equals `3.0 / 61.0`; `_rrf_max(3, 30)` equals `3.0 / 31.0`. + +(b) `test_hybrid_score_max_unchanged`: assert the module attribute `_HYBRID_SCORE_MAX` equals the literal `(2.0/61.0) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS` computed in the test from the same imported constants — i.e. the refactor introduces no numeric drift. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py::test_rrf_max_formula -v` (and the second test). +Expected: FAIL — `_rrf_max` not defined (ImportError/AttributeError). + +- [ ] **Step 3: Implement** + +Add `_rrf_max(num_lists, k=60)` returning `num_lists / (k + 1)`. Redefine `_HYBRID_SCORE_MAX` to call `_rrf_max(2)` for the RRF term; leave the bonus terms and the surrounding comment unchanged except to note the RRF term is now derived. Do not change any other constant. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py -v` +Expected: PASS (both new tests + all existing scoring tests). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py tests/search/test_search_scoring.py` +Run: `git commit -m "refactor(scoring): derive _HYBRID_SCORE_MAX RRF term from list count"` + +--- + +## Task 2: Add `RankConfig` and thread it through the read path + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py` (add `RankConfig`) +- Modify: `src/java_codebase_rag/search/search_lancedb.py:635-646` (`_graph_expand_merge` signature), `:887-897` (`run_search` call site), and `run_search` signature (line ~764) +- Test: `tests/search/test_search_scoring.py`, `tests/search/test_search_lancedb.py` + +**Interfaces:** +- Produces (in `search_scoring.py`, dep-free): + - `@dataclass(frozen=True) class RankConfig:` with fields: + - `lists: frozenset[str]` — subset of `{"vector", "graph", "bm25"}`. Must contain `"vector"`. Validation in `__post_init__`: raise `ValueError` if `"vector"` not in `lists`, or if any element is outside the allowed set, or if `lists` is empty. + - `rrf_k: int = 60` — must be `>= 1` (else `ValueError`). + - `DEFAULT_RANK_CONFIG = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60)` — production default. (3-list is the shipped behavior after Task 4; until Task 4 lands the BM25 list, the `"bm25"` element is honored but yields an empty list, so behavior is effectively 2-list. This is intentional — Task 4 only adds the non-empty BM25 list.) + - `BASELINE_2LIST_CONFIG = RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=60)` — eval convenience. +- Produces (in `search_lancedb.py`): + - `_graph_expand_merge` gains keyword-only param `rank_config: RankConfig = DEFAULT_RANK_CONFIG`. It passes `k=rank_config.rrf_k` into `_rrf_merge`. It fuses exactly the lists named in `rank_config.lists` (vector always present; graph present unless omitted; bm25 handled in Task 4, empty until then). + - `run_search` gains keyword-only param `rank_config: RankConfig = DEFAULT_RANK_CONFIG`, forwarded to `_graph_expand_merge`. +- Consumes: Task 1's `_rrf_max`. + +- [ ] **Step 1: Write failing tests** + +(a) In `test_search_scoring.py`: +- `test_rank_config_defaults`: `DEFAULT_RANK_CONFIG.lists == frozenset({"vector","graph","bm25"})`, `.rrf_k == 60`. +- `test_rank_config_validation`: constructing `RankConfig(lists=frozenset({"graph"}))` raises `ValueError` (no vector); `RankConfig(lists=frozenset({"vector","nope"}))` raises `ValueError` (unknown list); `RankConfig(lists=frozenset({"vector"}), rrf_k=0)` raises `ValueError`. +- `test_rank_config_frozen`: mutating `DEFAULT_RANK_CONFIG.rrf_k = 5` raises `FrozenInstanceError`. + +(b) In `test_search_lancedb.py`: +- `test_graph_expand_merge_honors_injected_k` (monkeypatch-based): construct a tiny scenario where `_graph_expand_merge` is called with `rank_config=RankConfig(lists=frozenset({"vector","graph"}), rrf_k=30)`; assert the fused rows' `_score_components["rrf_raw"]` normalization reflects `k=30` (i.e. max = `2/31`), proving `k` is injected. Use the existing fixture/monkeypatch pattern already present in this test file for `_graph_expand_merge`; if none exists, build the minimal stub `LadybugGraph` + `_search_one_table` doubles the file already uses elsewhere. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py::test_rank_config_defaults tests/search/test_search_lancedb.py::test_graph_expand_merge_honors_injected_k -v` +Expected: FAIL — `RankConfig` not defined / `rank_config` param not accepted. + +- [ ] **Step 3: Implement** + +Add the `RankConfig` dataclass + the two module constants in `search_scoring.py` (with `from dataclasses import dataclass`). In `search_lancedb.py`: add the `rank_config` keyword-only param to `_graph_expand_merge` and `run_search`; pass `k=rank_config.rrf_k` to `_rrf_merge` in `_graph_expand_merge`; forward the param at the call site (line ~888). Do NOT yet add the BM25 list (Task 4) — this task only plumbs injection and keeps behavior identical to today because the default omits no behavior the current code has (graph still fuses; bm25 not yet fetched). + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/ -k "scoring or lancedb or hybrid" -v` +Expected: PASS — including the new tests, with zero regressions (behavior unchanged). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py src/java_codebase_rag/search/search_lancedb.py tests/search/test_search_scoring.py tests/search/test_search_lancedb.py` +Run: `git commit -m "feat(search): injectable RankConfig for RRF list-set and k"` + +--- + +## Task 3: Add `bm25=` token to `explain_score_components` + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py:342-393` +- Test: `tests/search/test_search_scoring.py` + +**Interfaces:** +- Produces: `explain_score_components` emits a `bm25=` token when called with `hybrid=True` and the component dict contains a truthy `"bm25"` key. Format: `bm25={float(value):.3f}`. It is appended after the existing `rrf=` token (within the `elif hybrid:` branch) and before the shared role/symbol/import tokens. When `"bm25"` is absent or zero, no token is emitted (consistent with how `symbol_bonus`/`role_weight` are conditionally shown). +- Consumes: nothing new. + +- [ ] **Step 1: Write failing tests** + +In `test_search_scoring.py`: +- `test_explain_bm25_token_present`: `explain_score_components({"rrf_raw": 0.03, "bm25": 12.5}, hybrid=True)` returns a string containing `"rrf=0.030"` AND `"bm25=12.500"` (order: rrf before bm25). +- `test_explain_bm25_token_absent_when_zero_or_missing`: `explain_score_components({"rrf_raw": 0.03}, hybrid=True)` does NOT contain `"bm25="`; same for `{"rrf_raw": 0.03, "bm25": 0.0}`. +- `test_explain_bm25_only_in_hybrid`: `explain_score_components({"bm25": 12.5}, lexical=True)` must NOT emit a `bm25=` token (the lexical branch keeps its existing `relevance=`/`name=` tokens — `bm25=` is hybrid-only). + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py::test_explain_bm25_token_present -v` +Expected: FAIL — no `bm25=` token produced. + +- [ ] **Step 3: Implement** + +In the `elif hybrid:` branch of `explain_score_components`, after appending the `rrf=` token, read `comps.get("bm25")` and, when truthy, append `f"bm25={float(bm25):.3f}"`. No other branch changes. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py -v` +Expected: PASS (new + existing explain tests). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py tests/search/test_search_scoring.py` +Run: `git commit -m "feat(scoring): surface bm25= token in hybrid explain output"` + +--- + +## Task 4: BM25 candidate fetch + wire as third RRF list + +**Files:** +- Modify: `src/java_codebase_rag/search/search_lancedb.py` (new `_bm25_candidate_rows` near `_graph_expand_merge` line 635; edits inside `_graph_expand_merge` at 709-716; default config already 3-list from Task 2) +- Modify: `src/java_codebase_rag/search/search_lexical.py:210` (widen visibility of `_try_fts_candidates` if needed for cross-module import — e.g. expose a thin public alias, do not change behavior) +- Test: `tests/search/test_search_lancedb.py` + +**Interfaces:** + +- Produces (in `search_lancedb.py`): + - `def _bm25_candidate_rows(*, g: "LadybugGraph", query: str, uri: str, db: object, extra_predicates: list[str], columns: list[str]) -> list[dict]:` + - Behavior: + 1. Call `search_lexical._try_fts_candidates(g, query, filter=None, path_contains=None)`. If it returns `None` (FTS extension/index unavailable) OR its `"rows"` is empty, return `[]`. + 2. From the result: `"scores"` is `{symbol_node_id: bm25_float}`, `"rows"` is a list of Symbol dicts each carrying at least `id` and a fully-qualified name field (the Symbol FQN — use the same key the lexical backend exposes for the FQN; confirm against `_SYMBOL_RETURN` in `search_lexical.py`). Build `ordered_fqns`: the distinct FQNs of the returned symbols, sorted by their BM25 score descending (ties broken by FQN ascending for determinism). Build `fqn_to_bm25: dict[str,float]` mapping each FQN to its max BM25 score among its symbols. + 3. Fetch chunk rows from LanceDB for exactly those FQNs, filter-respecting: build `preds = list(extra_predicates) + _build_extra_predicates(columns=columns, fqn_in=ordered_fqns)` and query the `java` table WITHOUT a vector ranking (filter-only `.where()` query through the same table-access pattern `_search_one_table` uses to open the table — but omitting the vector search so the rows are not re-ranked by similarity). Return columns must include at least `filename`, `range_start`, `range_end`, `primary_type_fqn`, plus whatever `_apply_chunk_hints` / `_refine_java_start_lines` need (mirror the column set `_search_one_table` returns for `java`). + 4. Group fetched chunks by `primary_type_fqn`. Emit chunk rows ordered by the BM25 rank of their owning FQN: iterate `ordered_fqns`; for each FQN that has chunks, emit its chunks in their natural table order. Each emitted chunk row is the same dict shape as a `vector_row`/`graph_row` and additionally has `_score_components["bm25"] = round(float(fqn_to_bm25[fqn]), 4)`. Chunks whose FQN was filtered out by `extra_predicates` are never fetched, so filter parity with the vector path is automatic. + 5. Return the ordered chunk-row list (may be empty). + - Error handling: any exception from the FTS call or the LanceDB fetch is caught and the function returns `[]` (silent degradation — matches the spec's "FTS failure is a silent no-op on the vector path"). Log via `_debug_ctx` (the existing helper in this module) at debug level. + + - `_graph_expand_merge` change: when `"bm25"` is in `rank_config.lists`, after computing `graph_rows`, call `_bm25_candidate_rows(g=g, query=query, uri=uri, db=db, extra_predicates=extra_predicates, columns=_table_columns(uri, TABLES["java"], db))`. Then, when building the lists for `_rrf_merge`, pass `[vector_rows, graph_rows, bm25_rows]` with `k=rank_config.rrf_k`. The `query` string must be threaded into `_graph_expand_merge` as a new keyword-only param (it is not currently a param — add `query: str` to the signature and forward it from `run_search`, which already has `query`). When `"bm25"` is NOT in `rank_config.lists`, omit the BM25 list entirely (2-list behavior, used by the eval baseline). + - Note on `graph_rows` presence: if `"graph"` is in `rank_config.lists` the graph list is fetched as today; if omitted, only vector + (optionally) bm25 are fused. The `vector` list is always present (validated by `RankConfig`). + +- Consumes: Task 2's `RankConfig` / `DEFAULT_RANK_CONFIG`; `search_lexical._try_fts_candidates`; `search_scoring.SYMBOL_FTS_INDEX`; `_build_extra_predicates`, `_table_columns`, `_debug_ctx`, `TABLES`. + +- [ ] **Step 1: Write failing tests** + +In `test_search_lancedb.py` (monkeypatch `LadybugGraph`, `_search_one_table`/table access, and `search_lexical._try_fts_candidates` — use the doubling patterns already in this file): + +(a) `test_bm25_candidate_rows_orders_by_bm25_score`: stub `_try_fts_candidates` to return symbols for FQNs `["B", "A", "C"]` with BM25 scores `{B: 30, A: 20, C: 10}`; stub the LanceDB chunk fetch to return one chunk per FQN. Assert the returned list is ordered `B, A, C` (BM25 desc) and each row's `_score_components["bm25"]` equals the owning FQN's score (B→30.0, A→20.0, C→10.0). + +(b) `test_bm25_candidate_rows_fts_unavailable_returns_empty`: stub `_try_fts_candidates` to return `None`; assert `_bm25_candidate_rows(...)` returns `[]` and no LanceDB fetch is attempted (assert the table-access double is never called). + +(c) `test_bm25_candidate_rows_respects_filter`: stub `_try_fts_candidates` to return FQNs `["A","B"]`; pass `extra_predicates=["primary_type_fqn <> 'B'"]` (or the project's equivalent SQL form) and assert only FQN `A`'s chunks are emitted (the predicate is applied at the chunk fetch, so B is filtered). + +(d) `test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order`: stub `_try_fts_candidates` returning FQN `A` (score 20) and FQN `B` (score 10); stub the chunk fetch to return 2 chunks for A and 1 for B. Assert output order is `[A_chunk1, A_chunk2, B_chunk1]` and all three carry the correct `bm25` component. + +(e) `test_graph_expand_merge_includes_bm25_list`: with `rank_config=DEFAULT_RANK_CONFIG` (3-list), monkeypatch `_bm25_candidate_rows` to return a known non-empty chunk list, and assert the fused result contains rows whose `_score_components` includes contributions from the BM25 list (a row appearing ONLY in the BM25 list is present in the fused output and has `_score_components["bm25"]` set). + +(f) `test_graph_expand_merge_omits_bm25_when_excluded`: with `rank_config=BASELINE_2LIST_CONFIG`, assert `_bm25_candidate_rows` is never called and the fused result matches today's 2-list behavior. + +(g) Integration: `test_run_search_bm25_degrades_silently_when_fts_missing` — run `run_search` against a fixture index whose LadybugDB graph has NO `sym_fts` index (or monkeypatch `_ensure_fts_loaded` → False); assert it returns results with no exception, no row has a `bm25` component, and ranking matches the pre-change 2-list vector path (compare against a baseline snapshot or a `BASELINE_2LIST_CONFIG` run — they must be equal). + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_lancedb.py -k "bm25_candidate or graph_expand_merge or run_search_bm25" -v` +Expected: FAIL — `_bm25_candidate_rows` undefined; `query` param not threaded. + +- [ ] **Step 3: Implement** + +Add `_bm25_candidate_rows` per the Produces contract above. Thread `query` into `_graph_expand_merge` and forward from `run_search`. Modify `_graph_expand_merge` to conditionally fetch + fuse the BM25 list based on `rank_config.lists`. If `_try_fts_candidates` is not importable from `search_lexical` due to its leading underscore, add a thin non-underscore alias in `search_lexical.py` (e.g. `fetch_fts_candidates = _try_fts_candidates`) without changing `_try_fts_candidates` behavior, and import the alias. Apply `_apply_chunk_hints` and `_refine_java_start_lines` to BM25 rows before returning (consistency with graph_rows handling at lines 701-702). + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/ -k "lexical or search or hybrid or rrf or bm25" -v` +Expected: PASS — all new tests green, no regressions in the existing search subset (104+ passed baseline from worktree setup). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_lancedb.py src/java_codebase_rag/search/search_lexical.py tests/search/test_search_lancedb.py` +Run: `git commit -m "feat(search): fuse LadybugDB BM25 as third RRF list on vector path"` + +--- + +## Task 5: Eval metrics module (recall@k, precision@k, MRR) + +**Files:** +- Create: `src/java_codebase_rag/eval/__init__.py`, `src/java_codebase_rag/eval/metrics.py` +- Test: `tests/eval/__init__.py`, `tests/eval/test_metrics.py` + +**Interfaces:** +- Produces (`eval/metrics.py`, dep-free, pure functions): + - `def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:` — fraction of `relevant` appearing in `retrieved[:k]`. Returns `0.0` if `relevant` is empty. Value in `[0.0, 1.0]`. + - `def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:` — `|retrieved[:k] ∩ relevant| / k`. Returns `0.0` if `k == 0`. Value in `[0.0, 1.0]`. + - `def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:` — `1.0 / rank` of the first `retrieved` item that is in `relevant` (1-indexed); `0.0` if none. + - `def mean(values: list[float]) -> float:` — arithmetic mean, `0.0` for empty. + - `def aggregate(per_query: list[dict]) -> dict[str, float]:` — given a list of per-query dicts each containing keys `recall@1, recall@5, recall@10, recall@20, precision@5, mrr`, return the mean of each across all queries, keyed identically (e.g. `{"recall@10": 0.42, "mrr": 0.55, ...}`). +- Consumes: nothing. + +- [ ] **Step 1: Write failing tests** + +In `tests/eval/test_metrics.py`, hand-computed cases: +- `test_recall_at_k`: `retrieved=["a","b","c"], relevant={"b","d"}, k=3` → `0.5` (b found, d not); `k=1` → `0.0`; `relevant=set()` → `0.0`; `k=10` (longer than retrieved) → `0.5`. +- `test_precision_at_k`: `retrieved=["a","b","c"], relevant={"b"}, k=2` → `0.5`; `k=3` → `0.333...`; `k=0` → `0.0`. +- `test_reciprocal_rank`: `retrieved=["a","b","c"], relevant={"b"}` → `0.5`; `relevant={"z"}` → `0.0`; first-position match `relevant={"a"}` → `1.0`. +- `test_mean`: `[1.0, 0.0, 0.5]` → `0.5`; `[]` → `0.0`. +- `test_aggregate`: two per-query dicts `{"recall@10":1.0,"mrr":1.0, ...}` and `{"recall@10":0.0,"mrr":0.0, ...}` aggregate to `recall@10==0.5, mrr==0.5`. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/eval/test_metrics.py -v` +Expected: FAIL — module/import not found. + +- [ ] **Step 3: Implement** + +Create `eval/__init__.py` (empty) and `eval/metrics.py` with the five functions per contract. Pure stdlib only. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/eval/test_metrics.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/eval/__init__.py src/java_codebase_rag/eval/metrics.py tests/eval/__init__.py tests/eval/test_metrics.py` +Run: `git commit -m "feat(eval): IR metrics — recall@k, precision@k, reciprocal rank, aggregate"` + +--- + +## Task 6: Eval ground-truth — Tier-A generator + Tier-B loader + +**Files:** +- Create: `src/java_codebase_rag/eval/ground_truth.py` +- Test: `tests/eval/test_ground_truth.py` + +**Interfaces:** +- Produces (`eval/ground_truth.py`): + - `@dataclass(frozen=True) class LabeledQuery:` fields `query: str`, `relevant: frozenset[str]` (set of relevant Symbol FQNs), `tier: str` ("A" or "B"). + - `def build_tier_a(symbols: Iterable[SymbolLike]) -> list[LabeledQuery]:` — for each symbol produce queries from its simple name via `search_scoring._split_identifier`: one camelCase-joined identifier form and one space-joined lowercase token form (e.g. for `DistributionChunkService` produce `"DistributionChunkService"` and `"distribution chunk service"`). `relevant` = `frozenset({symbol.fqn})`. Skip symbols whose simple name splits to fewer than 2 tokens or is shorter than 3 chars (noise). `tier="A"`. Deterministic: output sorted by `(query, fqn)`. + - `def load_tier_b(path: str | Path) -> list[LabeledQuery]:` — parse a YAML (`.yaml`/`.yml`) or JSON (`.json`) file whose schema is a list of `{query: str, relevant: [str, ...]}` objects; return `LabeledQuery(query=..., relevant=frozenset(...), tier="B")`. Raise `FileNotFoundError` if missing (the runner treats absence as "Tier-B disabled", so the runner checks existence before calling). `SymbolLike` is a small structural type (duck-typed) exposing `.fqn: str` and `.name: str` (the simple name). +- Consumes: `search_scoring._split_identifier` (dep-free). YAML via `yaml.safe_load` if PyYAML is already a dependency (check `pyproject.toml`); if not, support JSON only and document that Tier-B YAML requires the existing YAML dep or add it. + +- [ ] **Step 1: Write failing tests** + +In `tests/eval/test_ground_truth.py`: +- `test_build_tier_a_deterministic`: feed a fixed list of `SymbolLike` (use simple objects/namedtuples with `.fqn`/`.name`) including `name="DistributionChunkService"`; assert the output contains `LabeledQuery("DistributionChunkService", frozenset({fqn}), "A")` and `LabeledQuery("distribution chunk service", frozenset({fqn}), "A")`; assert running twice yields identical lists; assert output is sorted by `(query, fqn)`. +- `test_build_tier_a_skips_noise`: a symbol with `name="A"` (1 char) and one with `name="Do"` (single token after split) produce no queries. +- `test_load_tier_b_yaml` (if YAML available) / `test_load_tier_b_json`: write a temp file with two entries; assert parsed into two `LabeledQuery` with `tier="B"` and correct `relevant` frozensets. Assert `load_tier_b` on a non-existent path raises `FileNotFoundError`. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/eval/test_ground_truth.py -v` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +Create `eval/ground_truth.py` per contract. Reuse `search_scoring._split_identifier` for query derivation so tokenization parity with the FTS index holds (the index is built from the same splitter). Check `pyproject.toml` for PyYAML before using `yaml`. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/eval/test_ground_truth.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/eval/ground_truth.py tests/eval/test_ground_truth.py` +Run: `git commit -m "feat(eval): Tier-A auto ground-truth generator + Tier-B loader"` + +--- + +## Task 7: Eval runner — index shopizer, sweep configs, emit results + +**Files:** +- Create: `src/java_codebase_rag/eval/runner.py` +- Test: `tests/eval/test_runner.py` + +**Interfaces:** +- Produces (`eval/runner.py`): + - `@dataclass(frozen=True) class EvalConfig:` fields: `corpus_dir: str` (default `~/jrag-bench/shopizer`), `index_dir: str` (temp dir; created by the runner), `results_dir: str` (default `~/jrag-bench/shopizer/results`), `tier_b_path: str | None = None`, `ks: tuple[int, ...] = (30, 60, 90, 120)`, `top_k_metrics: tuple[int, ...] = (1, 5, 10, 20)`, `model_name: str = ` (read from the same default `run_search` consumers use — confirm against `search_lancedb.run_search`'s `model_name` default). + - `def run_eval(cfg: EvalConfig) -> EvalReport:` orchestration: + 1. Build a fresh shopizer index into `cfg.index_dir` using the project's existing index entry point (the `jrag`/`java-codebase-rag` CLI index command — invoke programmatically via the function the CLI wraps, not a subprocess, so errors surface). Locate the indexer function by searching the `cli`/pipeline module for the index entry the console script calls. + 2. Open the index for query: obtain the LanceDB `uri`/`db` handles and the LadybugDB graph path the same way `run_search`'s callers do (mirror the MCP `search_v2` wiring in `mcp/mcp_v2.py`). Load a `SentenceTransformer` model once (reuse `search_lancedb`'s model-loading helper). + 3. Build ground truth: enumerate `Symbol` nodes from the graph (mirror `search_lexical`'s symbol enumeration), `build_tier_a(...)`; if `cfg.tier_b_path` exists, `load_tier_b(...)` and concatenate. + 4. For each config to evaluate — `BASELINE_2LIST_CONFIG` (k=60) and `DEFAULT_RANK_CONFIG`-shape at each `k` in `cfg.ks` (i.e. `RankConfig(lists=frozenset({"vector","graph","bm25"}), rrf_k=k)`) — for each `LabeledQuery`: call `search_lancedb.run_search(query=..., uri=uri, table_keys=["java"], limit=max(cfg.top_k_metrics), path_substring=None, model_name=cfg.model_name, model=, rank_config=)`, measure wall-clock per query, map returned hits' `primary_type_fqn` (or `fqn`) to retrieved FQN list, compute per-query recall@k/precision@k/mrr, aggregate. + 5. Return an `EvalReport` dataclass with per-config aggregated metrics + per-config p50 latency in ms. + 6. Persist: write `results//report.md` (a Markdown table: rows = configs, columns = recall@1, recall@5, recall@10, recall@20, precision@5, mrr, p50_latency_ms) and `report.json` (raw `EvalReport` as JSON). +- Consumes: Tasks 2/4 (`RankConfig`, `BASELINE_2LIST_CONFIG`, `DEFAULT_RANK_CONFIG`, `run_search` with `rank_config`), Task 5 metrics, Task 6 ground truth, the existing indexer + index-opening + model-loading helpers. + +- [ ] **Step 1: Write failing tests** + +In `tests/eval/test_runner.py` (do NOT depend on the real shopizer corpus — use a tiny in-repo Java fixture corpus the test suite already indexes, or the smallest existing fixture): +- `test_eval_report_shape`: run `run_eval` against a tiny fixture corpus (point `corpus_dir` at a small fixture under `tests/`, a fresh temp `index_dir`); assert the returned `EvalReport` has one entry per config evaluated (`1 + len(ks)` entries), each entry contains all metric keys, and the latency field is a non-negative float. +- `test_eval_report_persists_files`: after `run_eval`, assert `/report.md` and `/report.json` exist and the Markdown contains a header row with all metric column names and one data row per config. +- `test_eval_tier_b_optional`: with `tier_b_path=None`, the run completes using only Tier-A ground truth (no exception). With `tier_b_path` pointing to a temp file with one entry, the report still produces all configs (Tier-B queries are included but the test only asserts the run completes and shape is correct — not specific numbers). +- A smoke marker test `test_runner_smoke_exists` is NOT needed if the above cover it; numbers are research outputs, never asserted. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/eval/test_runner.py -v` +Expected: FAIL — `runner` module not found. + +- [ ] **Step 3: Implement** + +Create `eval/runner.py` per contract. Reuse `run_search` directly (not the MCP layer) so `rank_config` is injectable. Keep the indexer/model/index-open wiring thin by delegating to existing helpers; if a helper doesn't exist for programmatic index-open, mirror the minimal calls `mcp_v2.search_v2` makes. Guard the whole run so a missing corpus raises a clear `FileNotFoundError` with the path (operators pointed at `~/jrag-bench/shopizer`). + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/eval/ -v` +Expected: PASS. (These tests build a real small index and may take longer than unit tests — that's acceptable; they are not part of the fast search subset.) + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/eval/runner.py tests/eval/test_runner.py` +Run: `git commit -m "feat(eval): shopizer recall/precision@k + MRR runner with k-sweep"` + +--- + +## Task 8: Run eval on shopizer, tune k, set default, docs, full suite + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py` (`DEFAULT_RANK_CONFIG.rrf_k` → eval winner) +- Modify: `docs/ARCHITECTURE.md` (HOW: vector read path now 3-list; rank-config injection point) +- Modify: `docs/DESIGN.md` (WHAT/WHY: BM25 promoted to first-class; reference the eval) +- Possibly `docs/CONFIGURATION.md` (only if any operator-visible surface changed — none should; if not, skip) +- Test: full suite + +**Interfaces:** none new. + +- [ ] **Step 1: Run the eval on the real corpus** + +Precondition: `~/jrag-bench/shopizer` exists with the shopizer source. If absent, STOP and report — the operator must clone shopizer there before this task can complete. +Run: `.venv/bin/python -m java_codebase_rag.eval.runner` (or the chosen invocation). Inspect `~/jrag-bench/shopizer/results//report.md`. + +- [ ] **Step 2: Pick the winning k** + +Per the spec win criterion: choose the `k` (among 30/60/90/120) maximizing **MRR** with **no regression on recall@10** vs the 2-list baseline; recall@1 is the tiebreak. If no 3-list config beats baseline on MRR, keep `DEFAULT_RANK_CONFIG` at 3-list @ k=60 anyway (lexical-first-class is the goal) and record the result honestly in the docs + the PR description. Open a follow-up issue if negative. + +- [ ] **Step 3: Set the default k** + +Update `DEFAULT_RANK_CONFIG` in `search_scoring.py` so `rrf_k` equals the winning k (if unchanged from 60, note that explicitly). Commit. + +- [ ] **Step 4: Update docs** + +Update `docs/ARCHITECTURE.md` read-path description to the 3-list RRF and mention `RankConfig` as the injection point. Update `docs/DESIGN.md` to reflect BM25 as first-class on the primary path, referencing the eval result (winning k + headline metric). Do NOT change operator docs unless an operator-facing surface changed (none did). + +- [ ] **Step 5: Run the full suite once** + +Erase stale indexes: `rm -rf tests/*/.java-codebase-rag tests/*/.java-codebase-rag.{yml,hosts}` +Run: `.venv/bin/python -m pytest tests/ -q` +Expected: PASS (0 failures). Report any failures before proceeding. + +- [ ] **Step 6: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py docs/ARCHITECTURE.md docs/DESIGN.md` +Run: `git commit -m "feat(search): promote BM25 to first-class hybrid RRF list (issue #431)"` + +--- + +## Self-Review Notes (resolved during authoring) + +- **Code leakage:** Steps describe behavior, data shapes, signatures, and exact expected test results — no method bodies or test code. +- **Self-containment:** Every task carries full Consumes/Produces contracts; Tasks 5–7 do not require reading the spec to implement. +- **Spec coverage:** 3-list fusion (T2/T4), `_HYBRID_SCORE_MAX` derivation (T1), `bm25=` explain (T3), filter-respect + FTS-unavailable degradation (T4g), rank-config injection (T2), eval harness with Tier-A/Tier-B + recall/precision/MRR + k-sweep + latency + win criterion (T5–T8) — all spec sections mapped. +- **Type consistency:** `RankConfig.lists`/`rrf_k`, `DEFAULT_RANK_CONFIG`, `BASELINE_2LIST_CONFIG`, `_bm25_candidate_rows`, `EvalConfig`, `EvalReport`, `LabeledQuery` — names used consistently across tasks. +- **Deferred (issues #434–#439):** absence unification, did-you-mean, no-vectors, airgapped FTS, NDCG, latency gating — all explicitly out of scope and filed. From a2947891325b2860107cdec7fe3e9c47657d353b Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 00:50:10 +0300 Subject: [PATCH 03/15] refactor(scoring): derive _HYBRID_SCORE_MAX RRF term from list count --- .../search/search_scoring.py | 22 +++++++++++++-- tests/search/test_search_scoring.py | 27 +++++++++++++++++++ 2 files changed, 47 insertions(+), 2 deletions(-) diff --git a/src/java_codebase_rag/search/search_scoring.py b/src/java_codebase_rag/search/search_scoring.py index b94336d..72e5f27 100644 --- a/src/java_codebase_rag/search/search_scoring.py +++ b/src/java_codebase_rag/search/search_scoring.py @@ -84,13 +84,31 @@ "DTO": -0.08, } + +def _rrf_max(num_lists: int, k: int = 60) -> float: + """Return the theoretical maximum RRF score for N-list fusion. + + Reciprocal Rank Fusion (RRF) bounds each contribution to ≤ 1/(rank + k). + For N fused lists, the maximum possible sum is N/(k + 1) (achieved when + an item ranks #1 across all lists). + + Args: + num_lists: Number of ranked lists being fused (e.g., 2 for vector+lexical). + k: The RRF constant (default 60 per the original paper). + + Returns: + The maximum RRF contribution: num_lists / (k + 1). + """ + return num_lists / (k + 1) + + # Theoretical maximum for hybrid composite score (used for display normalization). # Hybrid sort metric: raw_rrf * (import_factor if import_heavy else 1) # + role_weight + symbol_bonus # where raw_rrf ≤ 2/(k+1) for 2-list RRF, role_weight ≤ max(_ROLE_SCORE_WEIGHTS), # and symbol_bonus ≤ _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS. -# The import factor is ≤ 1, so we use the raw max (2/61). -_HYBRID_SCORE_MAX = (2.0 / 61.0) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS +# The import factor is ≤ 1, so we use the raw max (derived via _rrf_max(2)). +_HYBRID_SCORE_MAX = _rrf_max(2) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS def _query_tokens(query: str) -> set[str]: diff --git a/tests/search/test_search_scoring.py b/tests/search/test_search_scoring.py index 9d865d2..c19f61c 100644 --- a/tests/search/test_search_scoring.py +++ b/tests/search/test_search_scoring.py @@ -10,6 +10,11 @@ import pytest from java_codebase_rag.search.search_scoring import ( + _ACTION_VERB_BONUS, + _HYBRID_SCORE_MAX, + _ROLE_SCORE_WEIGHTS, + _SYMBOL_MATCH_BONUS_CAP, + _TYPE_MATCH_BONUS_CAP, declaration_line_number, vector_display_score, ) @@ -83,3 +88,25 @@ def test_declaration_line_number_method_only_chunk_keeps_anchor() -> None: # Empty / missing inputs are safe. assert declaration_line_number(None, 5) == 5 assert declaration_line_number("public class X {", None) is None + + +# ---------- _rrf_max (Task 1) ---------- + + +def test_rrf_max_formula() -> None: + """Test the RRF max formula: num_lists / (k + 1).""" + from java_codebase_rag.search.search_scoring import _rrf_max + + # Test the exact formula with different inputs + assert _rrf_max(2, 60) == pytest.approx(2.0 / 61.0, abs=1e-12) + assert _rrf_max(3, 60) == pytest.approx(3.0 / 61.0, abs=1e-12) + assert _rrf_max(3, 30) == pytest.approx(3.0 / 31.0, abs=1e-12) + + +def test_hybrid_score_max_unchanged() -> None: + """Verify _HYBRID_SCORE_MAX preserves its exact numeric value after refactor.""" + # Compute the expected value from the same constants used in the definition + expected = (2.0 / 61.0) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS + + # The refactor must not introduce any numeric drift + assert _HYBRID_SCORE_MAX == pytest.approx(expected, abs=1e-12) From 54695ae56cf9a08e21d3cbbde75d1961eaf0c3fd Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 00:57:47 +0300 Subject: [PATCH 04/15] feat(search): injectable RankConfig for RRF list-set and k Adds a dep-free RankConfig dataclass in search_scoring.py describing which RRF lists to fuse (vector required; graph/bm25 optional) and the RRF k, then threads it as a keyword-only param through run_search -> _graph_expand_merge -> _rrf_merge. Behavior is unchanged: _graph_expand_merge still fuses [vector_rows, graph_rows] and the DEFAULT_RANK_CONFIG ships rrf_k=60. The "bm25" element of the default config is inert until Task 4 wires the BM25 candidate fetch; this task only makes k injectable (testable via the new test_graph_expand_merge_honors_injected_k). Co-Authored-By: Claude --- .../search/search_lancedb.py | 7 ++ .../search/search_scoring.py | 55 +++++++++++++++ tests/search/test_search_lancedb.py | 69 +++++++++++++++++++ tests/search/test_search_scoring.py | 50 ++++++++++++++ 4 files changed, 181 insertions(+) diff --git a/src/java_codebase_rag/search/search_lancedb.py b/src/java_codebase_rag/search/search_lancedb.py index a4958b6..e73efe5 100644 --- a/src/java_codebase_rag/search/search_lancedb.py +++ b/src/java_codebase_rag/search/search_lancedb.py @@ -27,7 +27,10 @@ # graph-only (macOS Intel) installs where this module is unimportable. Re-exported # here for backward compatibility (`from search_lancedb import _clamp01`, etc.). from java_codebase_rag.search.search_scoring import ( # noqa: F401 + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, DEDUP_OVERFETCH, + RankConfig, _ACTION_VERB_BONUS, _ACTION_VERB_PREFIXES, _HYBRID_SCORE_MAX, @@ -642,6 +645,7 @@ def _graph_expand_merge( extra_predicates: list[str], expand_depth: int, ladybug_path: str | None, + rank_config: RankConfig = DEFAULT_RANK_CONFIG, ) -> list[dict]: """Expand vector top-k through the LadybugDB graph and fuse (RRF) with the original list.""" # Lazy import so the module works without ladybug installed when graph_expand=False. @@ -708,6 +712,7 @@ def _graph_expand_merge( ) fused = _rrf_merge( [vector_rows, graph_rows], + k=rank_config.rrf_k, row_weight_for_list_index=[ None, lambda row: float(row.get("_graph_expand_weight", 1.0)), @@ -790,6 +795,7 @@ def run_search( generated_only: bool = False, exclude_generated: bool = False, dedup_by_fqn: bool = False, + rank_config: RankConfig = DEFAULT_RANK_CONFIG, ) -> list[dict]: effective_hybrid = hybrid effective_fts = fts_text @@ -894,6 +900,7 @@ def run_search( extra_predicates=extra_java, expand_depth=expand_depth, ladybug_path=ladybug_path, + rank_config=rank_config, ) # Dedup by primary_type_fqn after all sorting/merging, before windowing diff --git a/src/java_codebase_rag/search/search_scoring.py b/src/java_codebase_rag/search/search_scoring.py index 72e5f27..ca4dcb6 100644 --- a/src/java_codebase_rag/search/search_scoring.py +++ b/src/java_codebase_rag/search/search_scoring.py @@ -13,6 +13,7 @@ import json import re +from dataclasses import dataclass # Name of the LadybugDB FTS (Okapi BM25) index over Symbol.search_text (fork A). # Shared by the build path (build_ast_graph._ensure_symbol_fts_index) and the @@ -102,6 +103,60 @@ def _rrf_max(num_lists: int, k: int = 60) -> float: return num_lists / (k + 1) +# Allowed list names in a RankConfig: vector is always required (it is the +# backbone retrieval signal); graph and bm25 are optional fusion participants. +# NOTE: "bm25" is honored as a config element from Task 2, but the BM25 candidate +# fetch is wired only in Task 4 — until then a configured "bm25" list yields no +# candidates and behaves as a 2-list (vector+graph) fusion. +_RANK_LIST_NAMES: frozenset[str] = frozenset({"vector", "graph", "bm25"}) + + +@dataclass(frozen=True) +class RankConfig: + """Which ranked lists to fuse and the RRF constant to fuse them with. + + This is a dep-free value object (no lancedb/torch) so it can be constructed + on every install flavor, including graph-only (macOS Intel). It is plumbed + through ``run_search`` → ``_graph_expand_merge`` to control (a) which lists + contribute to the final RRF fusion and (b) the ``k`` constant passed into + ``_rrf_merge``. + + Attributes: + lists: Subset of ``{"vector", "graph", "bm25"}``. Must contain + ``"vector"`` (the backbone retrieval signal) and be non-empty. + rrf_k: The RRF constant (default 60 per the original paper). Must be ≥ 1. + """ + + lists: frozenset[str] + rrf_k: int = 60 + + def __post_init__(self) -> None: + if not isinstance(self.lists, frozenset) or not self.lists: + raise ValueError("RankConfig.lists must be a non-empty frozenset") + if "vector" not in self.lists: + raise ValueError( + "RankConfig.lists must contain 'vector' (the backbone signal)" + ) + unknown = self.lists - _RANK_LIST_NAMES + if unknown: + raise ValueError( + f"RankConfig.lists has unknown names {sorted(unknown)!r}; " + f"allowed: {sorted(_RANK_LIST_NAMES)!r}" + ) + if not isinstance(self.rrf_k, int) or self.rrf_k < 1: + raise ValueError(f"RankConfig.rrf_k must be an int >= 1, got {self.rrf_k!r}") + + +# Production default: ship the 3-list config (vector+graph+bm25). The "bm25" +# element is inert until Task 4 wires the BM25 candidate fetch; until then this +# is effectively a 2-list (vector+graph) fusion, identical to pre-Task-2 behavior. +DEFAULT_RANK_CONFIG = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60) + +# Eval convenience: the historical 2-list (vector+graph) fusion, used by +# evaluation harnesses that do not want the bm25 list even after Task 4. +BASELINE_2LIST_CONFIG = RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=60) + + # Theoretical maximum for hybrid composite score (used for display normalization). # Hybrid sort metric: raw_rrf * (import_factor if import_heavy else 1) # + role_weight + symbol_bonus diff --git a/tests/search/test_search_lancedb.py b/tests/search/test_search_lancedb.py index cf889cd..d30d493 100644 --- a/tests/search/test_search_lancedb.py +++ b/tests/search/test_search_lancedb.py @@ -67,6 +67,75 @@ def test_java_enriched_columns_include_symbol_identity_fields() -> None: assert "metadata" in JAVA_ENRICHED_COLUMNS +def test_graph_expand_merge_honors_injected_k(monkeypatch) -> None: + """RankConfig.rrf_k flows through _graph_expand_merge into _rrf_merge. + + With k=30 injected, a row reinforced at rank 0 across the vector and graph + lists has ``rrf_raw = 2/(k+1) = 2/31``. Under the old default k=60 it would + be 2/61 — strictly smaller — so observing 2/31 proves the injected k wins. + """ + import sys + import types + + from java_codebase_rag.search.search_scoring import RankConfig + + # Stub the lazily-imported LadybugGraph so the function reaches the merge. + class _FakeGraph: + @staticmethod + def exists(path): + return True + + @staticmethod + def get(path): + class _G: + def expand_fqns(self, fqns, depth): + # One novel FQN so the function proceeds to graph fetch + fuse. + return ["com.example.Other"] + + def expand_methods(self, fqns, depth, exclude_external=False): + return [] + return _G() + + fake_mod = types.ModuleType("java_codebase_rag.graph.ladybug_queries") + fake_mod.LadybugGraph = _FakeGraph + monkeypatch.setitem(sys.modules, "java_codebase_rag.graph.ladybug_queries", fake_mod) + + # Same (filename, range_start, range_end) key in both lists → rank-0 in both, + # so raw RRF = 1/(k+1) + 1/(k+1) = 2/(k+1). + vector_rows = [ + {"filename": "a.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.example.Foo"}, + ] + graph_rows = [ + {"filename": "a.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.example.Other"}, + ] + + monkeypatch.setattr(search_lancedb, "_search_one_table", lambda *a, **kw: graph_rows) + monkeypatch.setattr(search_lancedb, "_table_columns", lambda *a, **kw: set()) + monkeypatch.setattr(search_lancedb, "_build_extra_predicates", lambda **kw: []) + monkeypatch.setattr(search_lancedb, "_apply_chunk_hints", lambda rows: None) + monkeypatch.setattr(search_lancedb, "_refine_java_start_lines", lambda rows: None) + monkeypatch.setattr(search_lancedb, "_vector_sort_key", lambda r: 0.0) + + result = search_lancedb._graph_expand_merge( + vector_rows, + query_vec=np.zeros(3), + db=object(), + uri="mem://", + limit=10, + extra_predicates=[], + expand_depth=1, + ladybug_path=None, + rank_config=RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=30), + ) + + top = result[0] + # k=30 → raw = 2/31; would be 2/61 if the default leaked through. + assert top["_score_components"]["rrf_raw"] == pytest.approx(2.0 / 31.0, abs=1e-12) + assert top["_score_components"]["rrf_raw"] > 2.0 / 61.0 + + def test_search_one_table_selects_symbol_identity_columns_when_schema_has_them(monkeypatch) -> None: selected: list[str] = [] diff --git a/tests/search/test_search_scoring.py b/tests/search/test_search_scoring.py index c19f61c..0d81b04 100644 --- a/tests/search/test_search_scoring.py +++ b/tests/search/test_search_scoring.py @@ -110,3 +110,53 @@ def test_hybrid_score_max_unchanged() -> None: # The refactor must not introduce any numeric drift assert _HYBRID_SCORE_MAX == pytest.approx(expected, abs=1e-12) + + +# ---------- RankConfig (Task 2) ---------- + + +def test_rank_config_defaults() -> None: + """DEFAULT_RANK_CONFIG ships the 3-list set (bm25 inert until Task 4) + and the canonical RRF k=60 from the original paper.""" + from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + ) + + assert DEFAULT_RANK_CONFIG.lists == frozenset({"vector", "graph", "bm25"}) + assert DEFAULT_RANK_CONFIG.rrf_k == 60 + # Eval convenience: omits bm25. + assert BASELINE_2LIST_CONFIG.lists == frozenset({"vector", "graph"}) + assert BASELINE_2LIST_CONFIG.rrf_k == 60 + + +def test_rank_config_validation() -> None: + """RankConfig validates its lists set and rrf_k range at construction.""" + from java_codebase_rag.search.search_scoring import RankConfig + + # Missing required "vector" element. + with pytest.raises(ValueError): + RankConfig(lists=frozenset({"graph"})) + # Unknown list name. + with pytest.raises(ValueError): + RankConfig(lists=frozenset({"vector", "nope"})) + # Empty set. + with pytest.raises(ValueError): + RankConfig(lists=frozenset()) + # rrf_k below 1. + with pytest.raises(ValueError): + RankConfig(lists=frozenset({"vector"}), rrf_k=0) + # Sanity: a minimal valid config constructs cleanly. + ok = RankConfig(lists=frozenset({"vector"}), rrf_k=1) + assert ok.lists == frozenset({"vector"}) + assert ok.rrf_k == 1 + + +def test_rank_config_frozen() -> None: + """RankConfig is frozen so configs can be safely shared as defaults.""" + from dataclasses import FrozenInstanceError + + from java_codebase_rag.search.search_scoring import DEFAULT_RANK_CONFIG + + with pytest.raises(FrozenInstanceError): + DEFAULT_RANK_CONFIG.rrf_k = 5 # type: ignore[misc] From 2f6d9de69e6a697bd8a58fe7d985ae032b661b18 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 01:02:19 +0300 Subject: [PATCH 05/15] feat(scoring): surface bm25= token in hybrid explain output --- .../search/search_scoring.py | 3 ++ tests/search/test_search_scoring.py | 32 +++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/src/java_codebase_rag/search/search_scoring.py b/src/java_codebase_rag/search/search_scoring.py index ca4dcb6..9868e40 100644 --- a/src/java_codebase_rag/search/search_scoring.py +++ b/src/java_codebase_rag/search/search_scoring.py @@ -447,6 +447,9 @@ def explain_score_components( rrf = comps.get("rrf_raw") or comps.get("hybrid_rrf") if rrf is not None: parts.append(f"rrf={float(rrf):.3f}") + bm25 = comps.get("bm25") + if bm25: + parts.append(f"bm25={float(bm25):.3f}") else: d = comps.get("distance") if d is not None: diff --git a/tests/search/test_search_scoring.py b/tests/search/test_search_scoring.py index 0d81b04..94cbcb7 100644 --- a/tests/search/test_search_scoring.py +++ b/tests/search/test_search_scoring.py @@ -16,6 +16,7 @@ _SYMBOL_MATCH_BONUS_CAP, _TYPE_MATCH_BONUS_CAP, declaration_line_number, + explain_score_components, vector_display_score, ) @@ -160,3 +161,34 @@ def test_rank_config_frozen() -> None: with pytest.raises(FrozenInstanceError): DEFAULT_RANK_CONFIG.rrf_k = 5 # type: ignore[misc] + + +# ---------- explain_score_components (Task 3) ---------- + + +def test_explain_bm25_token_present() -> None: + """When hybrid=True and bm25 is truthy, output contains both rrf= and bm25= tokens.""" + result = explain_score_components({"rrf_raw": 0.03, "bm25": 12.5}, hybrid=True) + assert "rrf=0.030" in result + assert "bm25=12.500" in result + # Verify order: rrf appears before bm25 + assert result.index("rrf=0.030") < result.index("bm25=12.500") + + +def test_explain_bm25_token_absent_when_zero_or_missing() -> None: + """When bm25 is missing or zero, no bm25= token is emitted.""" + # Missing bm25 + result1 = explain_score_components({"rrf_raw": 0.03}, hybrid=True) + assert "bm25=" not in result1 + # Zero bm25 + result2 = explain_score_components({"rrf_raw": 0.03, "bm25": 0.0}, hybrid=True) + assert "bm25=" not in result2 + + +def test_explain_bm25_only_in_hybrid() -> None: + """bm25= token is only emitted in hybrid mode, not lexical mode.""" + result = explain_score_components({"bm25": 12.5}, lexical=True) + assert "bm25=" not in result + # Lexical mode should still emit its standard tokens + assert "relevance=" in result or "name=" in result or not result # May be empty if no lexical comps + From c2c0d17c14ac7920f13fe8e5ddb790478847532d Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 01:31:51 +0300 Subject: [PATCH 06/15] feat(search): fuse LadybugDB BM25 as third RRF list on vector path Wire the FTS-based BM25 candidate fetch into the vector path's RRF fusion. A new _bm25_candidate_rows helper fetches BM25-ranked Symbols via the LadybugDB FTS index, maps each to its enclosing type fqn (deduping by max BM25 score), and resolves them to LanceDB chunk rows in BM25 rank order using a filter-only (non-vector) query so BM25 ordering is preserved. The helper degrades silently to [] on any failure (FTS unavailable / empty / filtered out), leaving the vector path unaffected. _graph_expand_merge now honors rank_config.lists: "graph" and "bm25" are each fetched only when configured, and the lists are fused with the configured rrf_k. The `query` string is threaded in as a new keyword-only param (forwarded from run_search) to drive the FTS candidate fetch. search_lexical exposes non-underscore aliases (fetch_fts_candidates, enclosing_type_fqn) for cross-module use without behavior change. Co-Authored-By: Claude --- .../search/search_lancedb.py | 291 ++++++++++--- .../search/search_lexical.py | 10 + tests/search/test_search_lancedb.py | 385 +++++++++++++++++- 3 files changed, 628 insertions(+), 58 deletions(-) diff --git a/src/java_codebase_rag/search/search_lancedb.py b/src/java_codebase_rag/search/search_lancedb.py index e73efe5..fd095ed 100644 --- a/src/java_codebase_rag/search/search_lancedb.py +++ b/src/java_codebase_rag/search/search_lancedb.py @@ -20,6 +20,7 @@ from java_codebase_rag.ast.chunk_heuristics import analyze_chunk, looks_like_code_identifier from java_codebase_rag.search.index_common import SBERT_MODEL +from java_codebase_rag.search import search_lexical from java_codebase_rag.config import maybe_expand_embedding_model_path, resolved_sbert_model_for_process_env # Scoring & dedup primitives live in `search_scoring` (dependency-free — no @@ -635,9 +636,140 @@ def _attach_neighbor_context( _debug_ctx(f"attached context to {attached}/{len(java_rows)} java rows") +def _bm25_candidate_rows( + *, + g: object, + query: str, + uri: str, + db: object, + extra_predicates: list[str], + columns: set[str], + limit: int = 100, +) -> list[dict]: + """Fetch BM25-ranked Symbol candidates from the FTS index and resolve them to + chunk rows in BM25 rank order. Returns ``[]`` on any failure (silent degradation). + + Pipeline: + 1. ``search_lexical.fetch_fts_candidates(g, query)`` → BM25-ranked Symbols + + a ``{symbol_node_id: bm25_score}`` map. ``None`` / empty → return ``[]``. + 2. Map each Symbol fqn to its enclosing TYPE fqn (``primary_type_fqn`` has no + ``#``; a member ``Type#method`` maps to ``Type``). Dedupe by type fqn, + keeping the MAX BM25 score among same-type symbols. + 3. Order type fqns by BM25 desc (fqn asc tiebreak — deterministic). + 4. Fetch chunk rows from LanceDB with a FILTER-ONLY query (no vector ranking, + so BM25 order is preserved). Predicates = caller's ``extra_predicates`` + + the ``primary_type_fqn IN (...)`` built from the ordered types — preserving + filter parity with the vector path. + 5. Group fetched chunks by ``primary_type_fqn``; emit in BM25 rank order, + each chunk carrying ``_score_components["bm25"]``. + 6. Apply ``_apply_chunk_hints`` + ``_refine_java_start_lines`` for consistency + with graph_rows handling. + + Any exception (FTS or LanceDB) → ``_debug_ctx`` log + return ``[]`` (silent + degradation; the vector path is unaffected). + """ + # 1. BM25 candidate fetch via the FTS index. + try: + fts = search_lexical.fetch_fts_candidates(g, query, filter=None, path_contains=None) + except Exception as exc: # noqa: BLE001 — silent degradation + _debug_ctx(f"bm25 FTS fetch raised: {exc!r}") + return [] + if not fts or not fts.get("rows"): + return [] + sym_rows = fts["rows"] + scores = fts.get("scores") or {} + + # 2. Map symbol fqns → enclosing type fqns; keep MAX bm25 per type. + type_fqn_to_bm25: dict[str, float] = {} + for r in sym_rows: + fqn = r.get("fqn") + if not fqn: + continue + type_fqn = search_lexical.enclosing_type_fqn(str(fqn)) + if not type_fqn: + continue + score = float(scores.get(r.get("id"), 0.0)) + prev = type_fqn_to_bm25.get(type_fqn) + if prev is None or score > prev: + type_fqn_to_bm25[type_fqn] = score + if not type_fqn_to_bm25: + return [] + + # 3. Deterministic ordering: BM25 desc, fqn asc. + ordered_types = sorted( + type_fqn_to_bm25.keys(), + key=lambda f: (-type_fqn_to_bm25[f], f), + ) + + # 4. Filter-only chunk fetch (NO vector ranking → BM25 order preserved). The + # ``primary_type_fqn IN (...)`` predicate must be buildable; if the index is so + # old that the column is absent, we can't restrict the fetch and degrade to []. + if "primary_type_fqn" not in columns: + _debug_ctx("bm25 fetch skipped: primary_type_fqn column absent from schema") + return [] + preds = list(extra_predicates) + _build_extra_predicates( + columns=columns, + role=None, module=None, microservice=None, + package_prefix=None, fqn_in=ordered_types, + ) + combined_pred = _combine_predicates(preds) + base_cols = ["filename", "text", "start", "end"] + for col in ("range_start", "range_end"): + if col in columns: + base_cols.append(col) + java_extra = [c for c in JAVA_ENRICHED_COLUMNS if c in columns] + select_cols = [*base_cols, "language", *java_extra] + + try: + tbl = db.open_table(TABLES["java"]) + # LanceDB 0.34 filter-only path: search() with no vector arg issues a + # non-vector scan; .where/.select/.limit/.to_list returns rows in table + # order without re-ranking by similarity. (tbl.query() is NOT available in + # 0.34; to_lance().scanner() requires pylance, which isn't installed on the + # PEP 508 graph-only profile — search() with no vector is the supported API.) + q = tbl.search().select(select_cols).limit( + max(limit, len(ordered_types) * 4) + ) + if combined_pred: + q = q.where(combined_pred, prefilter=True) + with _silence_lance_autoproj_warnings(): + fetched = q.to_list() + except Exception as exc: # noqa: BLE001 — silent degradation + _debug_ctx(f"bm25 chunk fetch failed: {exc!r}") + return [] + + # 5. Group by primary_type_fqn; emit in BM25 rank order. + by_type: dict[str, list[dict]] = {} + for ch in fetched: + tf = ch.get("primary_type_fqn") + if tf is None: + continue + by_type.setdefault(str(tf), []).append(ch) + + out: list[dict] = [] + for type_fqn in ordered_types: + chunks = by_type.get(type_fqn) + if not chunks: + continue # filtered out by extra_predicates / absent from index + bm25_val = round(float(type_fqn_to_bm25[type_fqn]), 4) + for ch in chunks: + ch["_kind"] = "java" + ch["_hybrid"] = False + ch.setdefault("_score_components", {})["bm25"] = bm25_val + ch["start"] = coerce_position_field(ch.get("start")) + ch["end"] = coerce_position_field(ch.get("end")) + out.append(ch) + + # 6. Consistency with graph_rows handling. + _apply_chunk_hints(out) + _refine_java_start_lines(out) + return out + + def _graph_expand_merge( vector_rows: list[dict], *, + query: str, query_vec: np.ndarray, db: object, uri: str, @@ -647,7 +779,22 @@ def _graph_expand_merge( ladybug_path: str | None, rank_config: RankConfig = DEFAULT_RANK_CONFIG, ) -> list[dict]: - """Expand vector top-k through the LadybugDB graph and fuse (RRF) with the original list.""" + """Expand vector top-k through the graph and/or fuse BM25, then RRF-merge. + + Which lists contribute is controlled by ``rank_config.lists``: + - ``"vector"`` — always present (the backbone; validated by RankConfig). + - ``"graph"`` — graph expand + fetch (skipped entirely when absent). + - ``"bm25"`` — LadybugDB FTS candidate fetch fused as a third list. + + Silent degradation: any failure in the graph or BM25 path drops just that list; + the vector list is never lost. Returns ``vector_rows`` unchanged when no + auxiliary list yields rows. + """ + want_graph = "graph" in rank_config.lists + want_bm25 = "bm25" in rank_config.lists + if not want_graph and not want_bm25: + return vector_rows + # Lazy import so the module works without ladybug installed when graph_expand=False. try: from java_codebase_rag.graph.ladybug_queries import LadybugGraph @@ -657,68 +804,97 @@ def _graph_expand_merge( if not LadybugGraph.exists(ladybug_path): return vector_rows - seed_fqns = sorted({r.get("primary_type_fqn") for r in vector_rows if r.get("primary_type_fqn")}) - if not seed_fqns: - return vector_rows + java_cols = _table_columns(uri, TABLES["java"], db) - try: - graph = LadybugGraph.get(ladybug_path) - structural = graph.expand_fqns(seed_fqns, depth=expand_depth) - method_pairs = graph.expand_methods( - seed_fqns, depth=expand_depth, exclude_external=True, - ) - expand_weight_by_fqn: dict[str, float] = {} - for f in structural: - if f: - expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), 1.0) - for f, conf in method_pairs: - if f: - expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), conf) - neighbor_fqns = list(dict.fromkeys( - list(structural) + [f for f, _ in method_pairs], - )) - except Exception: - return vector_rows + # --- graph list --- + graph_rows: list[dict] = [] + expand_weight_by_fqn: dict[str, float] = {} + if want_graph: + seed_fqns = sorted({r.get("primary_type_fqn") for r in vector_rows if r.get("primary_type_fqn")}) + neighbor_fqns: list[str] = [] + if seed_fqns: + try: + graph_obj = LadybugGraph.get(ladybug_path) + structural = graph_obj.expand_fqns(seed_fqns, depth=expand_depth) + method_pairs = graph_obj.expand_methods( + seed_fqns, depth=expand_depth, exclude_external=True, + ) + for f in structural: + if f: + expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), 1.0) + for f, conf in method_pairs: + if f: + expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), conf) + neighbor_fqns = list(dict.fromkeys( + list(structural) + [f for f, _ in method_pairs], + )) + except Exception: + neighbor_fqns = [] + + novel = [fqn for fqn in neighbor_fqns if fqn and fqn not in set(seed_fqns)] + if novel: + extra = list(extra_predicates) + extra.extend(_build_extra_predicates( + columns=java_cols, + role=None, module=None, microservice=None, + package_prefix=None, fqn_in=novel, + )) + try: + graph_rows = _search_one_table( + TABLES["java"], + uri=uri, db=db, query_vec=query_vec, + limit=max(limit, 20), + path_predicate=None, kind="java", + hybrid=False, fts_text=None, + extra_predicates=extra, + ) + except Exception: + graph_rows = [] + _apply_chunk_hints(graph_rows) + _refine_java_start_lines(graph_rows) + graph_rows.sort(key=_vector_sort_key) + for r in graph_rows: + r["_graph_expanded"] = True + r["_graph_expand_weight"] = expand_weight_by_fqn.get( + r.get("primary_type_fqn"), 1.0, + ) + + # --- bm25 list --- + bm25_rows: list[dict] = [] + if want_bm25: + try: + graph_obj = LadybugGraph.get(ladybug_path) + except Exception: + graph_obj = None + if graph_obj is not None: + bm25_rows = _bm25_candidate_rows( + g=graph_obj, + query=query, + uri=uri, + db=db, + extra_predicates=extra_predicates, + columns=java_cols, + limit=limit, + ) - novel = [fqn for fqn in neighbor_fqns if fqn and fqn not in set(seed_fqns)] - if not novel: + # --- RRF fusion (only lists that yielded rows beyond vector) --- + lists: list[list[dict]] = [vector_rows] + row_weights: list[Callable[[dict], float] | None] = [None] + if want_graph and graph_rows: + lists.append(graph_rows) + row_weights.append(lambda row: float(row.get("_graph_expand_weight", 1.0))) + if want_bm25 and bm25_rows: + lists.append(bm25_rows) + row_weights.append(None) + + if len(lists) == 1: return vector_rows - extra = list(extra_predicates) - extra.extend(_build_extra_predicates( - columns=_table_columns(uri, TABLES["java"], db), - role=None, module=None, microservice=None, - package_prefix=None, fqn_in=novel, - )) - - try: - graph_rows = _search_one_table( - TABLES["java"], - uri=uri, db=db, query_vec=query_vec, - limit=max(limit, 20), - path_predicate=None, kind="java", - hybrid=False, fts_text=None, - extra_predicates=extra, - ) - except Exception: - return vector_rows - _apply_chunk_hints(graph_rows) - _refine_java_start_lines(graph_rows) - graph_rows.sort(key=_vector_sort_key) - for r in graph_rows: - r["_graph_expanded"] = True - r["_graph_expand_weight"] = expand_weight_by_fqn.get( - r.get("primary_type_fqn"), 1.0, - ) - fused = _rrf_merge( - [vector_rows, graph_rows], + return _rrf_merge( + lists, k=rank_config.rrf_k, - row_weight_for_list_index=[ - None, - lambda row: float(row.get("_graph_expand_weight", 1.0)), - ], + row_weight_for_list_index=row_weights, ) - return fused def _rrf_merge( @@ -893,6 +1069,7 @@ def run_search( if graph_expand and key == "java" and expand_depth > 0: rows = _graph_expand_merge( rows, + query=query, query_vec=query_vec, db=db, uri=uri, diff --git a/src/java_codebase_rag/search/search_lexical.py b/src/java_codebase_rag/search/search_lexical.py index 24cf562..964ace5 100644 --- a/src/java_codebase_rag/search/search_lexical.py +++ b/src/java_codebase_rag/search/search_lexical.py @@ -127,6 +127,11 @@ def _enclosing_type_fqn(fqn: str) -> str: return fqn.split("#", 1)[0] if fqn else fqn +# Non-underscore aliases for cross-module callers (search_lancedb's BM25 fusion). +# Behavior is identical; the leading-underscore originals stay module-private. +enclosing_type_fqn = _enclosing_type_fqn + + def _resolve_source_root(graph: LadybugGraph) -> str: """Authoritative source root is the one cached on the graph at index time.""" try: @@ -255,6 +260,11 @@ def _try_fts_candidates( return {"rows": rows, "scores": scores} +# Non-underscore alias for cross-module callers (search_lancedb's BM25 fusion on the +# vector path). Behavior is identical to the leading-underscore original. +fetch_fts_candidates = _try_fts_candidates + + def run_lexical_search( query: str, *, diff --git a/tests/search/test_search_lancedb.py b/tests/search/test_search_lancedb.py index d30d493..aeb362e 100644 --- a/tests/search/test_search_lancedb.py +++ b/tests/search/test_search_lancedb.py @@ -11,7 +11,7 @@ pytest.importorskip("sentence_transformers") from java_codebase_rag.search import search_lancedb -from java_codebase_rag.search.search_lancedb import JAVA_ENRICHED_COLUMNS, _rrf_merge +from java_codebase_rag.search.search_lancedb import JAVA_ENRICHED_COLUMNS, _rrf_merge, run_search def test_rrf_merge_weights_second_list_by_row() -> None: @@ -120,6 +120,7 @@ def expand_methods(self, fqns, depth, exclude_external=False): result = search_lancedb._graph_expand_merge( vector_rows, + query="query", query_vec=np.zeros(3), db=object(), uri="mem://", @@ -723,3 +724,385 @@ def test_refine_java_start_lines_skips_nonjava_and_method_chunks() -> None: assert rows[0]["start"]["line"] == 7 assert rows[1]["start"]["line"] == 30 assert "start" not in rows[2] + + +# ---------- Task 4: BM25 candidate fetch + third RRF list ---------- + + +def _eval_pred(rows: list[dict], pred: str | None) -> list[dict]: + """Tiny SQL-ish predicate evaluator for test fakes (AND / IN / <> / =).""" + if not pred: + return list(rows) + out: list[dict] = [] + clauses = [c.strip().strip("()") for c in pred.replace("(", " ").replace(")", " ").split("AND")] + for r in rows: + keep = True + for clause in clauses: + clause = clause.strip() + if not clause: + continue + if " IN (" in clause: + col = clause.split(" IN ", 1)[0].strip() + vals_part = clause[clause.index("(") + 1 : clause.rindex(")")] + vals = [v.strip().strip("'") for v in vals_part.split(",")] + if str(r.get(col)) not in vals: + keep = False + break + elif "<>" in clause: + col, val = [p.strip() for p in clause.split("<>", 1)] + val = val.strip("'") + if str(r.get(col)) == val: + keep = False + break + elif "=" in clause: + col, val = [p.strip() for p in clause.split("=", 1)] + val = val.strip("'") + if str(r.get(col)) != val: + keep = False + break + if keep: + out.append(r) + return out + + +class _FilterQuery: + """Fake LanceDB filter-only query: search().where().select().limit().to_list().""" + + def __init__(self, rows: list[dict]) -> None: + self._rows = rows + self._pred: str | None = None + self._limit: int | None = None + + def where(self, pred: str | None, prefilter: bool = False) -> "_FilterQuery": + self._pred = pred + return self + + def select(self, _cols: list[str]) -> "_FilterQuery": + return self + + def limit(self, n: int) -> "_FilterQuery": + self._limit = n + return self + + def to_list(self) -> list[dict]: + filtered = _eval_pred(self._rows, self._pred) + if self._limit is not None: + filtered = filtered[: self._limit] + return list(filtered) + + +class _FilterTable: + """Fake LanceDB table: open_table(...).search() with no vector → filter-only.""" + + def __init__(self, rows: list[dict]) -> None: + self._rows = rows + + def search(self, *args, **kwargs) -> _FilterQuery: + # Filter-only when no positional vector arg is passed. + return _FilterQuery(self._rows) + + +class _RecordingDb: + """Fake DB recording open_table calls; returns a per-table _FilterTable.""" + + def __init__(self) -> None: + self.tables: dict[str, _FilterTable] = {} + self.opened: list[str] = [] + + def add(self, name: str, rows: list[dict]) -> None: + self.tables[name] = _FilterTable(rows) + + def open_table(self, name: str) -> _FilterTable: + self.opened.append(name) + return self.tables.get(name) or _FilterTable([]) + + +def _bm25_chunk(filename: str, fqn: str, rs: int, re_: int) -> dict: + return { + "filename": filename, + "range_start": rs, + "range_end": re_, + "primary_type_fqn": fqn, + "text": f"body of {fqn}", + "language": "java", + "start": {"line": rs, "byte_offset": 0}, + "end": {"line": re_, "byte_offset": 100}, + } + + +def _patch_bm25_environment(monkeypatch, *, fts_result, chunk_rows) -> _RecordingDb: + """Wire the common monkeypatches for _bm25_candidate_rows unit tests.""" + from java_codebase_rag.search import search_lexical + + monkeypatch.setattr(search_lexical, "fetch_fts_candidates", lambda *a, **kw: fts_result) + monkeypatch.setattr(search_lexical, "enclosing_type_fqn", lambda fqn: fqn.split("#", 1)[0]) + monkeypatch.setattr(search_lancedb, "_apply_chunk_hints", lambda rows: None) + monkeypatch.setattr(search_lancedb, "_refine_java_start_lines", lambda rows: None) + # Provide a schema that includes the enriched java columns the helper selects. + monkeypatch.setattr( + search_lancedb, "_table_columns", + lambda *a, **kw: {"filename", "text", "start", "end", "language", "range_start", + "range_end", "primary_type_fqn", "role", "package"}, + ) + db = _RecordingDb() + db.add(search_lancedb.TABLES["java"], list(chunk_rows)) + return db + + +def test_bm25_candidate_rows_orders_by_bm25_score(monkeypatch) -> None: + """(a) BM25 candidates are emitted in BM25-desc order with the owning score attached.""" + fts_result = { + "rows": [ + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sc", "fqn": "com.x.C", "kind": "class", "name": "C"}, + ], + "scores": {"sb": 30.0, "sa": 20.0, "sc": 10.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("b.java", "com.x.B", 1, 10), + _bm25_chunk("c.java", "com.x.C", 1, 10), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="q", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + fqns = [r["primary_type_fqn"] for r in out] + assert fqns == ["com.x.B", "com.x.A", "com.x.C"] + scores = {r["primary_type_fqn"]: r["_score_components"]["bm25"] for r in out} + assert scores == {"com.x.B": 30.0, "com.x.A": 20.0, "com.x.C": 10.0} + + +def test_bm25_candidate_rows_fts_unavailable_returns_empty(monkeypatch) -> None: + """(b) FTS returns None → [] and no LanceDB fetch is attempted.""" + db = _patch_bm25_environment(monkeypatch, fts_result=None, chunk_rows=[]) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="q", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + assert out == [] + assert db.opened == [] # no chunk fetch attempted + + +def test_bm25_candidate_rows_respects_filter(monkeypatch) -> None: + """(c) extra_predicates flow into the chunk fetch and filter out types.""" + fts_result = { + "rows": [ + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, + ], + "scores": {"sa": 20.0, "sb": 10.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("b.java", "com.x.B", 1, 10), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="q", uri="mem://", db=db, + extra_predicates=["primary_type_fqn <> 'com.x.B'"], + columns={"primary_type_fqn"}, + ) + fqns = [r["primary_type_fqn"] for r in out] + assert fqns == ["com.x.A"] + + +def test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order(monkeypatch) -> None: + """(d) Multiple chunks per type stay grouped, in table order, ordered by BM25 rank.""" + fts_result = { + "rows": [ + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, + ], + "scores": {"sa": 20.0, "sb": 10.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("a.java", "com.x.A", 11, 20), + _bm25_chunk("b.java", "com.x.B", 1, 10), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="q", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + keys = [(r["primary_type_fqn"], r["range_start"]) for r in out] + assert keys == [("com.x.A", 1), ("com.x.A", 11), ("com.x.B", 1)] + for r in out: + if r["primary_type_fqn"] == "com.x.A": + assert r["_score_components"]["bm25"] == 20.0 + else: + assert r["_score_components"]["bm25"] == 10.0 + + +def _stub_ladybug_graph(monkeypatch) -> None: + """Install a LadybugGraph stub that exists but expands to nothing novel.""" + import sys + import types + + class _FakeGraph: + @staticmethod + def exists(_path): + return True + + @staticmethod + def get(_path): + class _G: + def expand_fqns(self, fqns, depth): + return [] + + def expand_methods(self, fqns, depth, exclude_external=False): + return [] + + return _G() + + fake_mod = types.ModuleType("java_codebase_rag.graph.ladybug_queries") + fake_mod.LadybugGraph = _FakeGraph + monkeypatch.setitem(sys.modules, "java_codebase_rag.graph.ladybug_queries", fake_mod) + + +def test_graph_expand_merge_includes_bm25_list(monkeypatch) -> None: + """(e) With DEFAULT_RANK_CONFIG (3-list), a BM25-only row surfaces in the fusion.""" + from java_codebase_rag.search.search_scoring import RankConfig + + _stub_ladybug_graph(monkeypatch) + rc = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60) + + vector_rows = [ + {"filename": "v.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.V"}, + ] + bm25_rows = [ + {"filename": "b.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.B", "_kind": "java", + "_score_components": {"bm25": 0.42}}, + ] + monkeypatch.setattr(search_lancedb, "_bm25_candidate_rows", lambda **kw: list(bm25_rows)) + monkeypatch.setattr(search_lancedb, "_table_columns", lambda *a, **kw: set()) + monkeypatch.setattr(search_lancedb, "_build_extra_predicates", lambda **kw: []) + # Force the graph path to produce nothing (no novel fqns via stub) → graph_rows = []. + + result = search_lancedb._graph_expand_merge( + vector_rows, + query="something", + query_vec=np.zeros(3), + db=object(), + uri="mem://", + limit=10, + extra_predicates=[], + expand_depth=1, + ladybug_path=None, + rank_config=rc, + ) + files = [r["filename"] for r in result] + assert "b.java" in files + bm25_row = next(r for r in result if r["filename"] == "b.java") + assert bm25_row["_score_components"]["bm25"] == 0.42 + + +def test_graph_expand_merge_omits_bm25_when_excluded(monkeypatch) -> None: + """(f) With BASELINE_2LIST_CONFIG, _bm25_candidate_rows is never called.""" + from java_codebase_rag.search.search_scoring import BASELINE_2LIST_CONFIG + + _stub_ladybug_graph(monkeypatch) + calls: list[dict] = [] + monkeypatch.setattr(search_lancedb, "_bm25_candidate_rows", + lambda **kw: calls.append(kw) or []) + monkeypatch.setattr(search_lancedb, "_table_columns", lambda *a, **kw: set()) + monkeypatch.setattr(search_lancedb, "_build_extra_predicates", lambda **kw: []) + + vector_rows = [ + {"filename": "v.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.V"}, + ] + result = search_lancedb._graph_expand_merge( + vector_rows, + query="something", + query_vec=np.zeros(3), + db=object(), + uri="mem://", + limit=10, + extra_predicates=[], + expand_depth=1, + ladybug_path=None, + rank_config=BASELINE_2LIST_CONFIG, + ) + assert calls == [] + # Result is the vector rows (graph produced nothing novel via stub). + assert result == vector_rows + + +def test_run_search_bm25_degrades_silently_when_fts_missing(monkeypatch, tmp_path) -> None: + """(g) When FTS is unavailable, the 3-list config degrades to the 2-list baseline. + + Builds a real LanceDB index with one java row, stubs LadybugGraph to exist (so + _graph_expand_merge runs) but expand to nothing, and forces _ensure_fts_loaded→False + so the BM25 fetch returns None. Asserts: no exception, no row carries a `bm25` + score component, and the result equals the BASELINE_2LIST_CONFIG run. + """ + import uuid + + import lancedb + from sentence_transformers import SentenceTransformer + + from java_codebase_rag.ast.ast_java import ONTOLOGY_VERSION + from java_codebase_rag.search import search_lexical + from java_codebase_rag.search.index_common import SBERT_MODEL + from java_codebase_rag.search.search_lancedb import TABLES, _query_vector + from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + ) + + _stub_ladybug_graph(monkeypatch) + # Force FTS unavailable → _try_fts_candidates returns None. + monkeypatch.setattr(search_lexical, "_ensure_fts_loaded", lambda g: False) + + uri = str(tmp_path / "ldb") + model = SentenceTransformer(SBERT_MODEL, device="cpu", trust_remote_code=True) + text = "service that processes inbound Kafka records on the listener endpoint" + emb = _query_vector(model, text) + row = { + "id": str(uuid.uuid4()), + "filename": "smoke/p/Svc.java", + "text": text, + "language": "java", + "range_start": 0, + "range_end": 500, + "start": {"line": 1, "byte_offset": 0}, + "end": {"line": 20, "byte_offset": 400}, + "embedding": emb, + "package": "p", + "module": "smoke", + "microservice": "smoke", + "primary_type_fqn": "p.Svc", + "primary_type_kind": "class", + "role": "SERVICE", + "annotations_on_type": [], + "symbols": ["process"], + "ontology_version": ONTOLOGY_VERSION, + "capabilities": [], + } + db = lancedb.connect(uri) + db.create_table(TABLES["java"], [row], mode="create") + + common = dict( + uri=uri, table_keys=["java"], limit=5, path_substring=None, + model_name=SBERT_MODEL, device="cpu", model=model, + graph_expand=True, expand_depth=1, + ) + three = run_search(text, rank_config=DEFAULT_RANK_CONFIG, **common) + two = run_search(text, rank_config=BASELINE_2LIST_CONFIG, **common) + + # No bm25 component anywhere on either run. + for rows in (three, two): + assert all("bm25" not in (r.get("_score_components") or {}) for r in rows) + # Degradation: 3-list with FTS-missing == 2-list baseline. + assert [r["filename"] for r in three] == [r["filename"] for r in two] + assert len(three) == len(two) and len(two) >= 1 From 18675aac086d30f1ac737d1041095aa9ec192e74 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 01:39:06 +0300 Subject: [PATCH 07/15] test(search): cover member-fqn dedup-by-max in _bm25_candidate_rows Co-Authored-By: Claude --- tests/search/test_search_lancedb.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tests/search/test_search_lancedb.py b/tests/search/test_search_lancedb.py index aeb362e..a844c11 100644 --- a/tests/search/test_search_lancedb.py +++ b/tests/search/test_search_lancedb.py @@ -941,6 +941,35 @@ def test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order(monkeypat assert r["_score_components"]["bm25"] == 10.0 +def test_bm25_candidate_rows_dedup_members_of_same_type_keep_max(monkeypatch) -> None: + """(d2) Two member symbols (#-fqns) of the SAME type dedup to one type entry, + emitted ONCE at the MAX BM25 score among the members.""" + fts_result = { + "rows": [ + {"id": "sm1", "fqn": "com.x.A#method1()", "kind": "method", "name": "method1"}, + {"id": "sm2", "fqn": "com.x.A#method2()", "kind": "method", "name": "method2"}, + ], + "scores": {"sm1": 30.0, "sm2": 20.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("a.java", "com.x.A", 11, 20), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="q", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + # The type appears exactly once (both member chunks present, but only one type rank). + type_fqns = [r["primary_type_fqn"] for r in out] + assert type_fqns.count("com.x.A") == 2 # 2 chunks of the same single type + assert set(type_fqns) == {"com.x.A"} # no other type leaked; deduped to one entry + # Keep-max: every emitted chunk carries the MAX member score (30.0), not 20.0. + for r in out: + assert r["_score_components"]["bm25"] == 30.0 + + def _stub_ladybug_graph(monkeypatch) -> None: """Install a LadybugGraph stub that exists but expands to nothing novel.""" import sys From 0d5ff028ada4491ba2bbd3c0731777fa68f48959 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 01:41:09 +0300 Subject: [PATCH 08/15] =?UTF-8?q?feat(eval):=20IR=20metrics=20=E2=80=94=20?= =?UTF-8?q?recall@k,=20precision@k,=20reciprocal=20rank,=20aggregate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/java_codebase_rag/eval/__init__.py | 1 + src/java_codebase_rag/eval/metrics.py | 107 +++++++++++++++++++++++++ tests/eval/__init__.py | 1 + tests/eval/test_metrics.py | 100 +++++++++++++++++++++++ 4 files changed, 209 insertions(+) create mode 100644 src/java_codebase_rag/eval/__init__.py create mode 100644 src/java_codebase_rag/eval/metrics.py create mode 100644 tests/eval/__init__.py create mode 100644 tests/eval/test_metrics.py diff --git a/src/java_codebase_rag/eval/__init__.py b/src/java_codebase_rag/eval/__init__.py new file mode 100644 index 0000000..b8ae542 --- /dev/null +++ b/src/java_codebase_rag/eval/__init__.py @@ -0,0 +1 @@ +# Eval package for IR metrics diff --git a/src/java_codebase_rag/eval/metrics.py b/src/java_codebase_rag/eval/metrics.py new file mode 100644 index 0000000..487ad9d --- /dev/null +++ b/src/java_codebase_rag/eval/metrics.py @@ -0,0 +1,107 @@ +"""IR evaluation metrics — pure functions, stdlib only. + +Functions take `retrieved: list[str]` (ordered list of retrieved FQN ids) +and `relevant: set[str]` (ground-truth relevant set). +""" + +from __future__ import annotations + + +def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float: + """Fraction of relevant documents appearing in retrieved[:k]. + + Args: + retrieved: Ordered list of retrieved document IDs. + relevant: Set of ground-truth relevant document IDs. + k: Cut-off rank (1-indexed). + + Returns: + Recall@k in [0.0, 1.0]. Returns 0.0 if relevant is empty. + """ + if not relevant: + return 0.0 + + retrieved_at_k = set(retrieved[:k]) + relevant_retrieved = retrieved_at_k.intersection(relevant) + + return len(relevant_retrieved) / len(relevant) + + +def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float: + """Precision at cut-off k: |retrieved[:k] ∩ relevant| / k. + + Args: + retrieved: Ordered list of retrieved document IDs. + relevant: Set of ground-truth relevant document IDs. + k: Cut-off rank (1-indexed). + + Returns: + Precision@k in [0.0, 1.0]. Returns 0.0 if k == 0. + """ + if k == 0: + return 0.0 + + retrieved_at_k = set(retrieved[:k]) + relevant_retrieved = retrieved_at_k.intersection(relevant) + + return len(relevant_retrieved) / k + + +def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float: + """Reciprocal rank: 1.0 / rank of first retrieved relevant document. + + Args: + retrieved: Ordered list of retrieved document IDs. + relevant: Set of ground-truth relevant document IDs. + + Returns: + Reciprocal rank in [0.0, 1.0]. Returns 0.0 if no relevant document + is retrieved. Ranks are 1-indexed (first hit = 1.0). + """ + for rank, doc_id in enumerate(retrieved, start=1): + if doc_id in relevant: + return 1.0 / rank + + return 0.0 + + +def mean(values: list[float]) -> float: + """Arithmetic mean of a list of floats. + + Args: + values: List of float values. + + Returns: + Arithmetic mean. Returns 0.0 for empty list. + """ + if not values: + return 0.0 + + return sum(values) / len(values) + + +def aggregate(per_query: list[dict]) -> dict[str, float]: + """Aggregate per-query metrics into means across all queries. + + Takes a list of per-query metric dicts and computes the mean of each + metric across all queries. All dicts must have the same keys. + + Args: + per_query: List of dicts, each containing metric names as keys + and float values (e.g., {"recall@10": 1.0, "mrr": 0.5, ...}). + + Returns: + Dict with same keys as input, containing mean of each metric + across all queries (e.g., {"recall@10": 0.42, "mrr": 0.55, ...}). + """ + if not per_query: + return {} + + metric_names = list(per_query[0].keys()) + result: dict[str, float] = {} + + for metric in metric_names: + values = [query_metrics[metric] for query_metrics in per_query] + result[metric] = mean(values) + + return result diff --git a/tests/eval/__init__.py b/tests/eval/__init__.py new file mode 100644 index 0000000..3f8d105 --- /dev/null +++ b/tests/eval/__init__.py @@ -0,0 +1 @@ +# Test package marker diff --git a/tests/eval/test_metrics.py b/tests/eval/test_metrics.py new file mode 100644 index 0000000..0ca089b --- /dev/null +++ b/tests/eval/test_metrics.py @@ -0,0 +1,100 @@ +"""Tests for eval.metrics — hand-computed cases to verify metric math.""" + +import pytest +from java_codebase_rag.eval.metrics import ( + recall_at_k, + precision_at_k, + reciprocal_rank, + mean, + aggregate, +) + + +class TestRecallAtK: + def test_recall_at_k_basic(self): + retrieved = ["a", "b", "c"] + relevant = {"b", "d"} + k = 3 + assert recall_at_k(retrieved, relevant, k) == 0.5 # b found, d not + + def test_recall_at_k_small_k(self): + retrieved = ["a", "b", "c"] + relevant = {"b", "d"} + k = 1 + assert recall_at_k(retrieved, relevant, k) == 0.0 # b not in first position + + def test_recall_at_k_empty_relevant(self): + retrieved = ["a", "b", "c"] + relevant = set() + k = 3 + assert recall_at_k(retrieved, relevant, k) == 0.0 # empty relevant set + + def test_recall_at_k_k_larger_than_retrieved(self): + retrieved = ["a", "b", "c"] + relevant = {"b", "d"} + k = 10 # longer than retrieved + assert recall_at_k(retrieved, relevant, k) == 0.5 # only b found + + +class TestPrecisionAtK: + def test_precision_at_k_basic(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + k = 2 + assert precision_at_k(retrieved, relevant, k) == 0.5 # 1 out of 2 + + def test_precision_at_k_full_retrieved(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + k = 3 + assert precision_at_k(retrieved, relevant, k) == 1.0 / 3.0 # 1 out of 3 + + def test_precision_at_k_zero_k(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + k = 0 + assert precision_at_k(retrieved, relevant, k) == 0.0 # k == 0 + + +class TestReciprocalRank: + def test_reciprocal_rank_second_position(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + assert reciprocal_rank(retrieved, relevant) == 0.5 # 1/2 + + def test_reciprocal_rank_no_match(self): + retrieved = ["a", "b", "c"] + relevant = {"z"} + assert reciprocal_rank(retrieved, relevant) == 0.0 # no match + + def test_reciprocal_rank_first_position(self): + retrieved = ["a", "b", "c"] + relevant = {"a"} + assert reciprocal_rank(retrieved, relevant) == 1.0 # 1/1 + + +class TestMean: + def test_mean_basic(self): + values = [1.0, 0.0, 0.5] + assert mean(values) == 0.5 # (1.0 + 0.0 + 0.5) / 3 + + def test_mean_empty(self): + values = [] + assert mean(values) == 0.0 # empty list + + +class TestAggregate: + def test_aggregate_basic(self): + per_query = [ + {"recall@1": 1.0, "recall@5": 1.0, "recall@10": 1.0, "recall@20": 1.0, "precision@5": 1.0, "mrr": 1.0}, + {"recall@1": 0.0, "recall@5": 0.0, "recall@10": 0.0, "recall@20": 0.0, "precision@5": 0.0, "mrr": 0.0}, + ] + result = aggregate(per_query) + assert result == { + "recall@1": 0.5, + "recall@5": 0.5, + "recall@10": 0.5, + "recall@20": 0.5, + "precision@5": 0.5, + "mrr": 0.5, + } From 61a96d2313764ace19d735ac18af2ad8d1ab465b Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 01:45:16 +0300 Subject: [PATCH 09/15] feat(eval): Tier-A auto ground-truth generator + Tier-B loader Co-Authored-By: Claude --- src/java_codebase_rag/eval/ground_truth.py | 100 ++++++++++++++++++ tests/eval/test_ground_truth.py | 115 +++++++++++++++++++++ 2 files changed, 215 insertions(+) create mode 100644 src/java_codebase_rag/eval/ground_truth.py create mode 100644 tests/eval/test_ground_truth.py diff --git a/src/java_codebase_rag/eval/ground_truth.py b/src/java_codebase_rag/eval/ground_truth.py new file mode 100644 index 0000000..3087ffd --- /dev/null +++ b/src/java_codebase_rag/eval/ground_truth.py @@ -0,0 +1,100 @@ +"""Eval ground-truth — Tier-A auto generator + Tier-B file loader. + +Tier-A derives labeled queries deterministically from indexed symbols (no +manual labeling). Tier-B loads hand-curated labeled queries from YAML/JSON. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable, Protocol + +import yaml + +from java_codebase_rag.search.search_scoring import _split_identifier + + +class SymbolLike(Protocol): + """Structural type for symbols — duck-typed .fqn / .name.""" + + fqn: str + name: str + + +@dataclass(frozen=True) +class LabeledQuery: + """A labeled retrieval query and its set of relevant Symbol FQNs.""" + + query: str + relevant: frozenset[str] + tier: str + + +def build_tier_a(symbols: Iterable[SymbolLike]) -> list[LabeledQuery]: + """Auto-generate labeled queries from each symbol's simple name. + + For each symbol two query strings are derived from its simple name: + 1. The original simple name verbatim (e.g. ``"DistributionChunkService"``) + — matches identifier-joined index text. + 2. A space-joined lowercase token form (e.g. ``"distribution chunk service"``) + produced via ``search_scoring._split_identifier`` so tokenization parity + with the FTS index holds. + + Symbols whose simple name splits to fewer than 2 tokens or is shorter than + 3 characters are skipped (noise). Output is deterministic, sorted by + ``(query, fqn)``; all entries carry ``tier="A"``. + """ + out: list[LabeledQuery] = [] + for sym in symbols: + name: str = sym.name + if len(name) < 3: + continue + tokens = _split_identifier(name) + if len(tokens) < 2: + continue + fqn: str = sym.fqn + relevant = frozenset({fqn}) + # 1. identifier-joined form = ORIGINAL simple name (preserve case). + out.append(LabeledQuery(name, relevant, "A")) + # 2. space-joined lowercase token form. + out.append(LabeledQuery(" ".join(tokens), relevant, "A")) + out.sort(key=lambda q: (q.query, next(iter(q.relevant)))) + return out + + +def load_tier_b(path: str | Path) -> list[LabeledQuery]: + """Load hand-curated Tier-B labeled queries from a YAML (``.yaml``/``.yml``) + or JSON (``.json``) file. + + Schema: a list of ``{query: str, relevant: [str, ...]}`` objects. + + Raises: + FileNotFoundError: if the path does not exist (the runner checks + existence before calling, treating absence as "Tier-B disabled"). + """ + p = Path(path) + if not p.exists(): + raise FileNotFoundError(f"Tier-B ground-truth file not found: {p}") + + suffix = p.suffix.lower() + raw = p.read_text() + if suffix in (".yaml", ".yml"): + data = yaml.safe_load(raw) + elif suffix == ".json": + data = json.loads(raw) + else: + # Fall back to YAML (superset of JSON) for unknown extensions. + data = yaml.safe_load(raw) + + out: list[LabeledQuery] = [] + for entry in data or []: + out.append( + LabeledQuery( + query=str(entry["query"]), + relevant=frozenset(entry.get("relevant", []) or []), + tier="B", + ) + ) + return out diff --git a/tests/eval/test_ground_truth.py b/tests/eval/test_ground_truth.py new file mode 100644 index 0000000..d1804f0 --- /dev/null +++ b/tests/eval/test_ground_truth.py @@ -0,0 +1,115 @@ +"""Tests for eval.ground_truth — Tier-A generator + Tier-B loader.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from java_codebase_rag.eval.ground_truth import ( + LabeledQuery, + build_tier_a, + load_tier_b, +) + + +class _Sym: + """Minimal structural symbol stand-in (duck-typed .fqn / .name).""" + + def __init__(self, fqn: str, name: str) -> None: + self.fqn = fqn + self.name = name + + +class TestBuildTierA: + def test_build_tier_a_deterministic(self) -> None: + syms = [ + _Sym("com.example.DistributionChunkService", "DistributionChunkService"), + _Sym("com.example.Other", "Other"), + ] + out = build_tier_a(syms) + + fqn = "com.example.DistributionChunkService" + assert LabeledQuery("DistributionChunkService", frozenset({fqn}), "A") in out + assert LabeledQuery( + "distribution chunk service", frozenset({fqn}), "A" + ) in out + + # Deterministic: same input -> identical output + assert build_tier_a(syms) == out + + # Sorted by (query, fqn) + keys = [(q.query, next(iter(q.relevant))) for q in out] + assert keys == sorted(keys) + + def test_build_tier_a_skips_noise(self) -> None: + syms = [ + _Sym("com.example.A", "A"), # <3 chars + _Sym("com.example.Do", "Do"), # splits to single token + ] + assert build_tier_a(syms) == [] + + +class TestLoadTierB: + def test_load_tier_b_yaml(self, tmp_path: Path) -> None: + path = tmp_path / "tier_b.yaml" + path.write_text( + "- query: distribution chunk service\n" + " relevant:\n" + " - com.example.DistributionChunkService\n" + "- query: user service\n" + " relevant:\n" + " - com.example.UserService\n" + " - com.other.UserService\n" + ) + out = load_tier_b(path) + assert out == [ + LabeledQuery( + "distribution chunk service", + frozenset({"com.example.DistributionChunkService"}), + "B", + ), + LabeledQuery( + "user service", + frozenset({"com.example.UserService", "com.other.UserService"}), + "B", + ), + ] + + def test_load_tier_b_json(self, tmp_path: Path) -> None: + path = tmp_path / "tier_b.json" + path.write_text( + json.dumps( + [ + { + "query": "distribution chunk service", + "relevant": ["com.example.DistributionChunkService"], + }, + { + "query": "user service", + "relevant": [ + "com.example.UserService", + "com.other.UserService", + ], + }, + ] + ) + ) + out = load_tier_b(path) + assert out == [ + LabeledQuery( + "distribution chunk service", + frozenset({"com.example.DistributionChunkService"}), + "B", + ), + LabeledQuery( + "user service", + frozenset({"com.example.UserService", "com.other.UserService"}), + "B", + ), + ] + + def test_load_tier_b_missing_raises(self, tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError): + load_tier_b(tmp_path / "does_not_exist.yaml") From 01bbb328d9745beecf7fe84ccbc293800e64e7a4 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 01:59:47 +0300 Subject: [PATCH 10/15] feat(eval): shopizer recall/precision@k + MRR runner with k-sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 7 of the hybrid-BM25 RRF plan. Adds eval/runner.py: EvalConfig + run_eval orchestrator that builds a fresh index via the operator CLI subprocess, enumerates Symbol nodes for Tier-A ground truth (+ optional Tier-B), sweeps BASELINE_2LIST_CONFIG + a 3-list RankConfig at each k in cfg.ks, computes recall@k/precision@5/MRR via eval.metrics, and persists report.md + report.json under cfg.results_dir. Metric-granularity resolution: normalizes member FQNs (com.x.A#foo()) to enclosing-type FQNs via search_lexical._enclosing_type_fqn on the relevant set, matching the type-level primary_type_fqn carried by run_search rows — so recall for member symbols is meaningful. Indexer choice: subprocess over `java-codebase-rag init` (the stable operator surface) rather than reaching into cocoindex/pipeline internals. Tests assert SHAPE + file persistence + tier-B optionality against a tiny fixture (cross_service_smoke); skip cleanly when the vector stack or cocoindex CLI is absent. Co-Authored-By: Claude --- src/java_codebase_rag/eval/runner.py | 431 +++++++++++++++++++++++++++ tests/eval/test_runner.py | 123 ++++++++ 2 files changed, 554 insertions(+) create mode 100644 src/java_codebase_rag/eval/runner.py create mode 100644 tests/eval/test_runner.py diff --git a/src/java_codebase_rag/eval/runner.py b/src/java_codebase_rag/eval/runner.py new file mode 100644 index 0000000..dbeb3a5 --- /dev/null +++ b/src/java_codebase_rag/eval/runner.py @@ -0,0 +1,431 @@ +"""Eval runner — index a corpus, sweep RankConfigs, emit recall/precision/MRR. + +This is the integration layer of the eval harness (Task 7 of the hybrid-BM25 +RRF plan). Unlike ``eval/metrics.py`` and ``eval/ground_truth.py`` (pure +stdlib), the runner MAY import the full vector stack (torch / lancedb / +sentence_transformers) and invokes the operator CLI to build a real index. + +Pipeline (``run_eval``): + +1. Build a fresh index into ``cfg.index_dir`` via the operator CLI + (``java-codebase-rag init``) as a **subprocess** — the stable operator + surface. Reaching into cocoindex/pipeline internals is fragile (process env, + progress renderers), so the subprocess wins on robustness. A non-zero exit + surfaces stdout/stderr in the raised ``RuntimeError``. +2. Open the index for query: set ``JAVA_CODEBASE_RAG_INDEX_DIR`` so + ``resolve_ladybug_path`` + ``run_search``'s URI resolve to our temp index; + load ``SentenceTransformer`` once and reuse. +3. Enumerate ``Symbol`` nodes from the LadybugDB graph (mirrors + ``search_lexical``) and build Tier-A ground truth; optionally concat Tier-B. +4. For each ``RankConfig`` (``BASELINE_2LIST_CONFIG`` + a 3-list config per + swept ``k``), run ``run_search`` per query, map rows to ``primary_type_fqn`` + and compute per-query recall@k / precision@k / MRR via ``eval.metrics``. +5. Aggregate, persist Markdown + JSON under ``/report.{md,json}``. + +Granularity note (metric mapping) +--------------------------------- +Tier-A ``build_tier_a`` sets ``relevant = {symbol.fqn}`` where the fqn may be a +MEMBER fqn (``com.x.A#processClientMessage()``). ``run_search`` rows carry +``primary_type_fqn`` = the enclosing TYPE fqn (``com.x.A``, no ``#``). To make +both sides type-level, the runner normalizes member→type via +``search_lexical._enclosing_type_fqn`` BEFORE scoring. This keeps Task 6's +``build_tier_a`` untouched. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import time +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +from java_codebase_rag.eval.ground_truth import LabeledQuery, build_tier_a, load_tier_b +from java_codebase_rag.eval import metrics as M +from java_codebase_rag.graph.ladybug_queries import LadybugGraph, resolve_ladybug_path +from java_codebase_rag.search.index_common import SBERT_MODEL +from java_codebase_rag.search.search_lexical import _enclosing_type_fqn, _SYMBOL_RETURN +from java_codebase_rag.search.search_lancedb import run_search +from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + RankConfig, +) + +# Markdown table columns — order-stable, mirrored by the test suite. +METRIC_COLUMNS: tuple[str, ...] = ( + "recall@1", + "recall@5", + "recall@10", + "recall@20", + "precision@5", + "mrr", + "p50_latency_ms", +) + + +@dataclass(frozen=True) +class EvalConfig: + """Configuration for a single eval run. + + ``index_dir`` may be empty — the runner creates a temp dir in that case + (and writes it back into the returned ``EvalReport``). + """ + + corpus_dir: str = field( + default_factory=lambda: str(Path.home() / "jrag-bench" / "shopizer") + ) + index_dir: str = "" + results_dir: str = field( + default_factory=lambda: str(Path.home() / "jrag-bench" / "shopizer" / "results") + ) + tier_b_path: str | None = None + ks: tuple[int, ...] = (30, 60, 90, 120) + top_k_metrics: tuple[int, ...] = (1, 5, 10, 20) + model_name: str = SBERT_MODEL + device: str | None = field( + default_factory=lambda: os.environ.get("SBERT_DEVICE") or None + ) + + +@dataclass(frozen=True) +class ConfigMetrics: + """Aggregated metrics for one RankConfig under test.""" + + config_name: str + metrics: dict[str, float] + num_queries: int + rrf_k: int + lists: tuple[str, ...] + + +@dataclass(frozen=True) +class EvalReport: + """Result of a full eval sweep.""" + + configs: list[ConfigMetrics] + timestamp: str + num_queries: int + corpus_dir: str + index_dir: str + + def to_json(self) -> str: + return json.dumps(asdict(self), indent=2, sort_keys=True) + + +# --------------------------------------------------------------------------- +# Index build (subprocess over the operator CLI — the stable surface) +# --------------------------------------------------------------------------- + + +def _build_index_subprocess(*, corpus_dir: str, index_dir: str) -> None: + """Build a fresh index via ``java-codebase-rag init``. + + Raises FileNotFoundError if the corpus is missing, RuntimeError on a + non-zero CLI exit (surfacing clipped stdout/stderr). + """ + if not Path(corpus_dir).is_dir(): + raise FileNotFoundError( + f"eval corpus_dir does not exist: {corpus_dir} " + "(point EvalConfig.corpus_dir at a checked-out Java repo)" + ) + + Path(index_dir).mkdir(parents=True, exist_ok=True) + env = { + **os.environ, + "JAVA_CODEBASE_RAG_INDEX_DIR": str(Path(index_dir).resolve()), + "JAVA_CODEBASE_RAG_SOURCE_ROOT": str(Path(corpus_dir).resolve()), + } + cmd = [ + sys.executable, + "-m", + "java_codebase_rag.cli", + "init", + "--source-root", + str(Path(corpus_dir).resolve()), + "--index-dir", + str(Path(index_dir).resolve()), + "--quiet", + ] + proc = subprocess.run( + cmd, + capture_output=True, + text=True, + env=env, + timeout=int(os.environ.get("JAVA_CODEBASE_RAG_EVAL_INDEX_TIMEOUT", "1800")), + ) + if proc.returncode != 0: + raise RuntimeError( + f"java-codebase-rag init exited {proc.returncode} for corpus {corpus_dir}.\n" + f"--- stdout (clipped 8000) ---\n{proc.stdout[-8000:]}\n" + f"--- stderr (clipped 8000) ---\n{proc.stderr[-8000:]}" + ) + + +# --------------------------------------------------------------------------- +# Symbol enumeration (mirror search_lexical) +# --------------------------------------------------------------------------- + + +class _SymbolRow: + """Attribute-view over a Cypher row dict (build_tier_a ducks on .fqn/.name).""" + + __slots__ = ("fqn", "name", "kind") + + def __init__(self, row: dict[str, Any]) -> None: + self.fqn = str(row.get("fqn") or "") + self.name = str(row.get("name") or "") + self.kind = str(row.get("kind") or "") + + +def _enumerate_symbols(graph: LadybugGraph) -> list[_SymbolRow]: + """Return Symbol rows as duck-typed objects (.fqn / .name) for build_tier_a. + + ``LadybugGraph._rows`` returns dicts; ``build_tier_a``'s ``SymbolLike`` + protocol ducks on attributes, so we wrap each row. + """ + rows = graph._rows(f"MATCH (s:Symbol) RETURN {_SYMBOL_RETURN}") # noqa: SLF001 + return [_SymbolRow(r) for r in rows] + + +# --------------------------------------------------------------------------- +# Metric computation +# --------------------------------------------------------------------------- + + +def _retrieved_fqns(rows: list[dict]) -> list[str]: + """Map search rows to ordered, deduped type-level FQNs.""" + out: list[str] = [] + seen: set[str] = set() + for r in rows: + fqn = r.get("primary_type_fqn") or r.get("fqn") or "" + if not fqn: + continue + if fqn in seen: + continue + seen.add(fqn) + out.append(fqn) + return out + + +def _relevant_type_fqns(labeled: LabeledQuery) -> set[str]: + """Normalize member FQNs to enclosing-type FQNs so both sides are type-level.""" + return {_enclosing_type_fqn(fqn) for fqn in labeled.relevant if fqn} + + +def _p50(values: list[float]) -> float: + if not values: + return 0.0 + s = sorted(values) + mid = len(s) // 2 + if len(s) % 2: + return float(s[mid]) + return float((s[mid - 1] + s[mid]) / 2.0) + + +def _eval_one_config( + *, + config_name: str, + rank_config: RankConfig, + queries: list[LabeledQuery], + uri: str, + ladybug_path: str, + model, + model_name: str, + device: str | None, + top_k_metrics: tuple[int, ...], + limit: int, +) -> ConfigMetrics: + """Run one RankConfig over all queries; return aggregated ConfigMetrics.""" + limit_k = max(top_k_metrics) + per_query: list[dict[str, float]] = [] + latencies_ms: list[float] = [] + + for q in queries: + t0 = time.perf_counter() + rows = run_search( + q.query, + uri=uri, + table_keys=["java"], + limit=limit, + offset=0, + path_substring=None, + model_name=model_name, + device=device, + model=model, + rank_config=rank_config, + graph_expand=True, + expand_depth=1, + ladybug_path=ladybug_path, + dedup_by_fqn=True, + ) + elapsed_ms = (time.perf_counter() - t0) * 1000.0 + latencies_ms.append(elapsed_ms) + + retrieved = _retrieved_fqns(rows) + relevant = _relevant_type_fqns(q) + if not relevant: + # No relevant set (e.g. Tier-B with empty relevant) — skip from scoring. + continue + + qm: dict[str, float] = {} + for k in top_k_metrics: + qm[f"recall@{k}"] = M.recall_at_k(retrieved, relevant, k) + qm["precision@5"] = M.precision_at_k(retrieved, relevant, 5) + qm["mrr"] = M.reciprocal_rank(retrieved, relevant) + per_query.append(qm) + + agg = M.aggregate(per_query) + metrics: dict[str, float] = {} + for k in top_k_metrics: + metrics[f"recall@{k}"] = float(agg.get(f"recall@{k}", 0.0)) + metrics["precision@5"] = float(agg.get("precision@5", 0.0)) + metrics["mrr"] = float(agg.get("mrr", 0.0)) + metrics["p50_latency_ms"] = _p50(latencies_ms) + + return ConfigMetrics( + config_name=config_name, + metrics=metrics, + num_queries=len(per_query), + rrf_k=rank_config.rrf_k, + lists=tuple(sorted(rank_config.lists)), + ) + + +# --------------------------------------------------------------------------- +# Markdown rendering +# --------------------------------------------------------------------------- + + +def _render_markdown(report: EvalReport) -> str: + lines: list[str] = [] + lines.append(f"# Eval Report — {report.timestamp}") + lines.append("") + lines.append( + f"Corpus: `{report.corpus_dir}` | Index: `{report.index_dir}` " + f"| Queries scored: {report.num_queries}" + ) + lines.append("") + header = "| config | " + " | ".join(METRIC_COLUMNS) + " |" + sep = "| --- " * (len(METRIC_COLUMNS) + 1) + "|" + lines.append(header) + lines.append(sep) + for entry in report.configs: + cells = [entry.config_name] + for col in METRIC_COLUMNS: + v = entry.metrics.get(col, 0.0) + if col == "p50_latency_ms": + cells.append(f"{v:.1f}") + else: + cells.append(f"{v:.4f}") + lines.append("| " + " | ".join(cells) + " |") + lines.append("") + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Orchestration +# --------------------------------------------------------------------------- + + +def run_eval(cfg: EvalConfig) -> EvalReport: + """Build a fresh index, sweep RankConfigs, return an EvalReport. + + See module docstring for the pipeline and the metric-granularity note. + """ + # Late import — torch/lancedb only needed for the run, not for module import. + from sentence_transformers import SentenceTransformer + + # Resolve index_dir (temp dir if blank). + if not cfg.index_dir: + index_dir = tempfile.mkdtemp(prefix="jrag-eval-") + else: + index_dir = cfg.index_dir + Path(index_dir).mkdir(parents=True, exist_ok=True) + + # 1. Build the index (subprocess). + _build_index_subprocess(corpus_dir=cfg.corpus_dir, index_dir=index_dir) + + # 2. Wire the process env so resolve_ladybug_path + run_search's URI hit our index. + os.environ["JAVA_CODEBASE_RAG_INDEX_DIR"] = str(Path(index_dir).resolve()) + os.environ.setdefault( + "JAVA_CODEBASE_RAG_SOURCE_ROOT", str(Path(cfg.corpus_dir).resolve()) + ) + uri = str(Path(index_dir).resolve()) + ladybug_path = resolve_ladybug_path(None) + + # Load the model ONCE — pass into every run_search call. + model = SentenceTransformer( + cfg.model_name, device=cfg.device, trust_remote_code=True + ) + + # 3. Open graph + build ground truth. + # Reset the LadybugGraph singleton — a prior test/process may have cached + # a different path. We force-bind to our index's graph. + LadybugGraph._instance = None # noqa: SLF001 + LadybugGraph._instance_path = None # noqa: SLF001 + graph = LadybugGraph.get(ladybug_path) + + symbols = _enumerate_symbols(graph) + queries = list(build_tier_a(symbols)) + if cfg.tier_b_path: + queries.extend(load_tier_b(cfg.tier_b_path)) + + # 4. Enumerate configs: BASELINE_2LIST_CONFIG (k=60) + 3-list at each swept k. + limit = max(cfg.top_k_metrics) + configs: list[tuple[str, RankConfig]] = [ + ("baseline_2list_k60", BASELINE_2LIST_CONFIG), + ] + for k in cfg.ks: + configs.append( + ( + f"hybrid_3list_k{k}", + RankConfig( + lists=frozenset({"vector", "graph", "bm25"}), + rrf_k=k, + ), + ) + ) + + # 5. Run + aggregate. + results: list[ConfigMetrics] = [] + for name, rc in configs: + results.append( + _eval_one_config( + config_name=name, + rank_config=rc, + queries=queries, + uri=uri, + ladybug_path=ladybug_path, + model=model, + model_name=cfg.model_name, + device=cfg.device, + top_k_metrics=cfg.top_k_metrics, + limit=limit, + ) + ) + + timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + report = EvalReport( + configs=results, + timestamp=timestamp, + num_queries=len(queries), + corpus_dir=cfg.corpus_dir, + index_dir=index_dir, + ) + + # 6. Persist. + _persist(report, cfg.results_dir) + return report + + +def _persist(report: EvalReport, results_dir: str) -> None: + out = Path(results_dir) + out.mkdir(parents=True, exist_ok=True) + (out / "report.md").write_text(_render_markdown(report)) + (out / "report.json").write_text(report.to_json()) diff --git a/tests/eval/test_runner.py b/tests/eval/test_runner.py new file mode 100644 index 0000000..30a3dfe --- /dev/null +++ b/tests/eval/test_runner.py @@ -0,0 +1,123 @@ +"""Tests for eval.runner — shopizer-style eval sweep over a tiny fixture corpus. + +These tests build a REAL (tiny) index via the operator CLI subprocess and run +``run_search`` under multiple ``RankConfig``s. They assert SHAPE and file +persistence only — never specific ranking numbers (those are research outputs). + +Skipped cleanly when the vector stack (torch / lancedb / sentence_transformers) +or the cocoindex CLI is unavailable (graph-only envs). +""" +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path + +import pytest + +# Skip the whole file when the vector stack is missing — runner needs it. +pytest.importorskip("lancedb") +pytest.importorskip("sentence_transformers") +pytest.importorskip("torch") + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent +TINY_CORPUS = REPO_ROOT / "tests" / "fixtures" / "cross_service_smoke" + +# Expected metric keys (table columns) — order-stable. +METRIC_KEYS = ( + "recall@1", + "recall@5", + "recall@10", + "recall@20", + "precision@5", + "mrr", + "p50_latency_ms", +) + + +def _cocoindex_available() -> bool: + """True when the cocoindex CLI sits next to the pytest interpreter.""" + return (Path(sys.executable).parent / "cocoindex").is_file() + + +pytestmark = pytest.mark.skipif( + not _cocoindex_available(), + reason="cocoindex CLI not installed in this venv; runner integration test needs the full stack", +) + + +def _cfg(tmp_path: Path, *, tier_b_path: str | None = None, tag: str = "run"): + from java_codebase_rag.eval.runner import EvalConfig + + # Fresh index dir per tag — `init` refuses an occupied index_dir, and some + # tests invoke run_eval twice into the same tmp_path. + return EvalConfig( + corpus_dir=str(TINY_CORPUS), + index_dir=str(tmp_path / f"index_{tag}"), + results_dir=str(tmp_path / f"results_{tag}"), + tier_b_path=tier_b_path, + ks=(60,), # single k keeps the smoke fast; shape is what we assert + top_k_metrics=(1, 5, 10, 20), + ) + + +def test_eval_report_shape(tmp_path): + from java_codebase_rag.eval.runner import run_eval + + report = run_eval(_cfg(tmp_path)) + + # 1 baseline + 1 per swept k. + assert len(report.configs) == 1 + 1 + + for entry in report.configs: + assert entry.config_name + assert entry.num_queries >= 0 + for key in METRIC_KEYS: + assert key in entry.metrics, f"missing metric {key} in {entry.config_name}" + assert isinstance(entry.metrics[key], float) + assert entry.metrics["p50_latency_ms"] >= 0.0 + + +def test_eval_report_persists_files(tmp_path): + from java_codebase_rag.eval.runner import run_eval + + cfg = _cfg(tmp_path) + report = run_eval(cfg) + + md_path = Path(cfg.results_dir) / "report.md" + json_path = Path(cfg.results_dir) / "report.json" + assert md_path.is_file(), f"missing report.md at {md_path}" + assert json_path.is_file(), f"missing report.json at {json_path}" + + md = md_path.read_text() + # Header row mentions every metric column. + for key in METRIC_KEYS: + assert key in md, f"report.md header missing column {key}" + # One data row per config (count pipe-led rows under the header). + data_rows = [ln for ln in md.splitlines() if ln.startswith("| ") and "-" not in ln[:3]] + assert len(data_rows) >= len(report.configs) + + payload = json.loads(json_path.read_text()) + assert "configs" in payload + assert len(payload["configs"]) == len(report.configs) + + +def test_eval_tier_b_optional(tmp_path): + """tier_b_path=None completes on Tier-A only; a one-entry file still works.""" + from java_codebase_rag.eval.runner import run_eval + + # None — no exception. + report = run_eval(_cfg(tmp_path, tier_b_path=None, tag="a")) + assert len(report.configs) == 1 + 1 + + # With a single Tier-B entry in a temp file. + tier_b = tmp_path / "tier_b.json" + tier_b.write_text( + json.dumps([{"query": "OrderService", "relevant": ["com.example.OrderService"]}]) + ) + report_b = run_eval(_cfg(tmp_path, tier_b_path=str(tier_b), tag="b")) + assert len(report_b.configs) == 1 + 1 + for entry in report_b.configs: + for key in METRIC_KEYS: + assert key in entry.metrics From a040c45140294b21be82dd833e3bab60507b156a Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 02:08:25 +0300 Subject: [PATCH 11/15] fix(eval): timestamped results subdir + cleanups (dead code, public reset API) Co-Authored-By: Claude --- src/java_codebase_rag/eval/runner.py | 18 ++++++++++++------ tests/eval/test_runner.py | 8 ++++++-- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/src/java_codebase_rag/eval/runner.py b/src/java_codebase_rag/eval/runner.py index dbeb3a5..18741fb 100644 --- a/src/java_codebase_rag/eval/runner.py +++ b/src/java_codebase_rag/eval/runner.py @@ -20,7 +20,9 @@ 4. For each ``RankConfig`` (``BASELINE_2LIST_CONFIG`` + a 3-list config per swept ``k``), run ``run_search`` per query, map rows to ``primary_type_fqn`` and compute per-query recall@k / precision@k / MRR via ``eval.metrics``. -5. Aggregate, persist Markdown + JSON under ``/report.{md,json}``. +5. Aggregate, persist Markdown + JSON under + ``//report.{md,json}`` (timestamped subdir so + successive sweeps don't clobber each other). Granularity note (metric mapping) --------------------------------- @@ -113,6 +115,8 @@ class EvalReport: num_queries: int corpus_dir: str index_dir: str + # Absolute path to the timestamped output dir holding report.md / report.json. + out_dir: str = "" def to_json(self) -> str: return json.dumps(asdict(self), indent=2, sort_keys=True) @@ -242,7 +246,6 @@ def _eval_one_config( limit: int, ) -> ConfigMetrics: """Run one RankConfig over all queries; return aggregated ConfigMetrics.""" - limit_k = max(top_k_metrics) per_query: list[dict[str, float]] = [] latencies_ms: list[float] = [] @@ -367,8 +370,7 @@ def run_eval(cfg: EvalConfig) -> EvalReport: # 3. Open graph + build ground truth. # Reset the LadybugGraph singleton — a prior test/process may have cached # a different path. We force-bind to our index's graph. - LadybugGraph._instance = None # noqa: SLF001 - LadybugGraph._instance_path = None # noqa: SLF001 + LadybugGraph.reset_for_path(None) graph = LadybugGraph.get(ladybug_path) symbols = _enumerate_symbols(graph) @@ -411,21 +413,25 @@ def run_eval(cfg: EvalConfig) -> EvalReport: ) timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + # Namespace outputs under the timestamp so successive sweeps don't clobber. + out_dir = str(Path(cfg.results_dir) / timestamp) report = EvalReport( configs=results, timestamp=timestamp, num_queries=len(queries), corpus_dir=cfg.corpus_dir, index_dir=index_dir, + out_dir=out_dir, ) - # 6. Persist. + # 6. Persist into //report.{md,json}. _persist(report, cfg.results_dir) return report def _persist(report: EvalReport, results_dir: str) -> None: - out = Path(results_dir) + # Write into a timestamped subdir so successive sweeps don't clobber. + out = Path(results_dir) / report.timestamp out.mkdir(parents=True, exist_ok=True) (out / "report.md").write_text(_render_markdown(report)) (out / "report.json").write_text(report.to_json()) diff --git a/tests/eval/test_runner.py b/tests/eval/test_runner.py index 30a3dfe..3f2ffe0 100644 --- a/tests/eval/test_runner.py +++ b/tests/eval/test_runner.py @@ -85,10 +85,14 @@ def test_eval_report_persists_files(tmp_path): cfg = _cfg(tmp_path) report = run_eval(cfg) - md_path = Path(cfg.results_dir) / "report.md" - json_path = Path(cfg.results_dir) / "report.json" + # Outputs are namespaced under a timestamped subdir (no flat-path clobbering). + out_dir = Path(cfg.results_dir) / report.timestamp + md_path = out_dir / "report.md" + json_path = out_dir / "report.json" assert md_path.is_file(), f"missing report.md at {md_path}" assert json_path.is_file(), f"missing report.json at {json_path}" + # EvalReport also exposes the resolved out_dir. + assert report.out_dir == str(out_dir) md = md_path.read_text() # Header row mentions every metric column. From 7c4d1aac93b7a97c7de8c9d92f7754f636b96b8c Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 10:52:52 +0300 Subject: [PATCH 12/15] fix(eval): bound Tier-A query fan-out (kind filter + max_queries cap) Co-Authored-By: Claude --- src/java_codebase_rag/eval/runner.py | 64 ++++++++++++++++++++++++++-- tests/eval/test_runner.py | 45 ++++++++++++++++++- 2 files changed, 103 insertions(+), 6 deletions(-) diff --git a/src/java_codebase_rag/eval/runner.py b/src/java_codebase_rag/eval/runner.py index 18741fb..6a098d6 100644 --- a/src/java_codebase_rag/eval/runner.py +++ b/src/java_codebase_rag/eval/runner.py @@ -71,6 +71,20 @@ ) +# Type-level Symbol kinds emitted by the graph (ast_java._TYPE_KINDS + +# build_ast_graph._TYPE_KINDS). Confirmed literals — note it's "annotation", +# NOT "annotation_type". Tier-A recall is measured at type level (chunks carry +# primary_type_fqn), so member-symbol queries are redundant; defaulting +# symbol_kinds to these type kinds both bounds fan-out and is semantically right. +_TYPE_SYMBOL_KINDS: tuple[str, ...] = ( + "class", + "interface", + "enum", + "annotation", + "record", +) + + @dataclass(frozen=True) class EvalConfig: """Configuration for a single eval run. @@ -93,6 +107,19 @@ class EvalConfig: device: str | None = field( default_factory=lambda: os.environ.get("SBERT_DEVICE") or None ) + # When set, _enumerate_symbols filters Symbol nodes to these kinds (type + # level by default). None = all kinds (escape hatch). Bounds Tier-A fan-out + # AND matches the type-level recall granularity. + symbol_kinds: tuple[str, ...] | None = _TYPE_SYMBOL_KINDS + # Deterministic cap on Tier-A LabeledQuery items produced (after the kind + # filter). Tier-B queries are NOT capped (operator-curated). See run_eval. + max_queries: int = 400 + + def __post_init__(self) -> None: + if self.max_queries < 1: + raise ValueError( + f"EvalConfig.max_queries must be >= 1, got {self.max_queries}" + ) @dataclass(frozen=True) @@ -115,6 +142,9 @@ class EvalReport: num_queries: int corpus_dir: str index_dir: str + # Tier-A queries available BEFORE the max_queries cap was applied (Tier-B + # excluded from this count). Recorded so capped runs stay interpretable. + num_queries_available: int = 0 # Absolute path to the timestamped output dir holding report.md / report.json. out_dir: str = "" @@ -187,13 +217,25 @@ def __init__(self, row: dict[str, Any]) -> None: self.kind = str(row.get("kind") or "") -def _enumerate_symbols(graph: LadybugGraph) -> list[_SymbolRow]: +def _enumerate_symbols( + graph: LadybugGraph, *, symbol_kinds: tuple[str, ...] | None +) -> list[_SymbolRow]: """Return Symbol rows as duck-typed objects (.fqn / .name) for build_tier_a. ``LadybugGraph._rows`` returns dicts; ``build_tier_a``'s ``SymbolLike`` protocol ducks on attributes, so we wrap each row. + + When ``symbol_kinds`` is not None, filters to those kinds via a + parameterized ``WHERE s.kind IN $kinds`` predicate — type-level only by + default, which both bounds fan-out and matches the type-level recall + granularity (chunks carry ``primary_type_fqn``). """ - rows = graph._rows(f"MATCH (s:Symbol) RETURN {_SYMBOL_RETURN}") # noqa: SLF001 + if symbol_kinds is None: + cypher = f"MATCH (s:Symbol) RETURN {_SYMBOL_RETURN}" + rows = graph._rows(cypher) # noqa: SLF001 + else: + cypher = f"MATCH (s:Symbol) WHERE s.kind IN $kinds RETURN {_SYMBOL_RETURN}" + rows = graph._rows(cypher, {"kinds": list(symbol_kinds)}) # noqa: SLF001 return [_SymbolRow(r) for r in rows] @@ -313,6 +355,11 @@ def _render_markdown(report: EvalReport) -> str: f"Corpus: `{report.corpus_dir}` | Index: `{report.index_dir}` " f"| Queries scored: {report.num_queries}" ) + if report.num_queries_available: + lines.append( + f"Tier-A available (pre-cap): {report.num_queries_available} | " + f"Total scored (Tier-A kept + Tier-B): {report.num_queries}" + ) lines.append("") header = "| config | " + " | ".join(METRIC_COLUMNS) + " |" sep = "| --- " * (len(METRIC_COLUMNS) + 1) + "|" @@ -373,8 +420,16 @@ def run_eval(cfg: EvalConfig) -> EvalReport: LadybugGraph.reset_for_path(None) graph = LadybugGraph.get(ladybug_path) - symbols = _enumerate_symbols(graph) - queries = list(build_tier_a(symbols)) + symbols = _enumerate_symbols(graph, symbol_kinds=cfg.symbol_kinds) + tier_a = list(build_tier_a(symbols)) + num_queries_available = len(tier_a) + # Deterministic cap on Tier-A: sort by (query, fqn) and keep the first + # max_queries. fqn lives as the sole element of each LabeledQuery.relevant + # (build_tier_a sets relevant = {symbol.fqn}). Tier-B is operator-curated + # and NOT capped. + tier_a_sorted = sorted(tier_a, key=lambda q: (q.query, next(iter(q.relevant)))) + tier_a_kept = tier_a_sorted[: cfg.max_queries] + queries = list(tier_a_kept) if cfg.tier_b_path: queries.extend(load_tier_b(cfg.tier_b_path)) @@ -421,6 +476,7 @@ def run_eval(cfg: EvalConfig) -> EvalReport: num_queries=len(queries), corpus_dir=cfg.corpus_dir, index_dir=index_dir, + num_queries_available=num_queries_available, out_dir=out_dir, ) diff --git a/tests/eval/test_runner.py b/tests/eval/test_runner.py index 3f2ffe0..5d1eca2 100644 --- a/tests/eval/test_runner.py +++ b/tests/eval/test_runner.py @@ -47,12 +47,18 @@ def _cocoindex_available() -> bool: ) -def _cfg(tmp_path: Path, *, tier_b_path: str | None = None, tag: str = "run"): +def _cfg( + tmp_path: Path, + *, + tier_b_path: str | None = None, + tag: str = "run", + max_queries: int | None = None, +): from java_codebase_rag.eval.runner import EvalConfig # Fresh index dir per tag — `init` refuses an occupied index_dir, and some # tests invoke run_eval twice into the same tmp_path. - return EvalConfig( + kwargs = dict( corpus_dir=str(TINY_CORPUS), index_dir=str(tmp_path / f"index_{tag}"), results_dir=str(tmp_path / f"results_{tag}"), @@ -60,6 +66,9 @@ def _cfg(tmp_path: Path, *, tier_b_path: str | None = None, tag: str = "run"): ks=(60,), # single k keeps the smoke fast; shape is what we assert top_k_metrics=(1, 5, 10, 20), ) + if max_queries is not None: + kwargs["max_queries"] = max_queries + return EvalConfig(**kwargs) def test_eval_report_shape(tmp_path): @@ -125,3 +134,35 @@ def test_eval_tier_b_optional(tmp_path): for entry in report_b.configs: for key in METRIC_KEYS: assert key in entry.metrics + + +def test_eval_max_queries_caps_tier_a(tmp_path): + """EvalConfig.max_queries deterministically caps the Tier-A query set. + + On the tiny fixture the default kind filter yields a handful of type-level + symbols, so a max_queries=2 cap must keep num_queries_available >= 2 while + the actually-scored Tier-A count is exactly 2 (no Tier-B here). Also + confirms the report records the pre-cap availability. + """ + from java_codebase_rag.eval.runner import run_eval + + report = run_eval(_cfg(tmp_path, max_queries=2, tag="cap")) + + # Pre-cap Tier-A availability is recorded and at least 2 (the fixture has + # multiple type-level symbols; the cap must have had something to bite on). + assert report.num_queries_available >= 2 + # With no Tier-B, num_queries == Tier-A kept == max_queries(2). + assert report.num_queries == 2 + # Every config scored exactly the capped query count. + for entry in report.configs: + assert entry.num_queries <= 2 + + +def test_eval_max_queries_validation(): + """max_queries < 1 is rejected with ValueError at construction time.""" + from java_codebase_rag.eval.runner import EvalConfig + + with pytest.raises(ValueError): + EvalConfig(max_queries=0) + # Boundary: 1 is accepted. + EvalConfig(max_queries=1) From e43884816327dfe46897b101feb63b5f2a2cff07 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 11:22:10 +0300 Subject: [PATCH 13/15] docs(search): BM25 first-class on primary path + eval result (issue #431) Update read-path docs: vector search now fuses 3 RRF lists (vector + graph + BM25), list-set/k injectable via RankConfig. DESIGN principle #2 notes BM25 is first-class, not just the macOS-Intel fallback. Records the shopizer eval result (3-list beats 2-list on all metrics at ~0 latency). Co-Authored-By: Claude --- docs/ARCHITECTURE.md | 6 ++++-- docs/DESIGN.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a6c854d..e2ebf90 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -68,7 +68,7 @@ java_codebase_rag/pipeline.py ``` MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* - ├─ search ─▶ search_lancedb.run_search (vector / hybrid; optional graph-expand + RRF rank fusion) + ├─ search ─▶ search_lancedb.run_search (vector / hybrid; graph-expand + 3-list RRF: vector + graph + BM25) │ └─ lancedb import absent (Intel Mac) → search_lexical (BM25 over Symbol FTS index; heuristic scan fallback) ├─ find / describe / neighbors ─▶ ladybug_queries.LadybugGraph (Cypher) └─ resolve ─▶ resolve_service.resolve_v2 (cascade → status one | many | none) @@ -78,7 +78,7 @@ MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* | Tool | Backing | Notes | | --- | --- | --- | -| `search` | Lance vector/hybrid, or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring` | +| `search` | Lance vector/hybrid with 3-list RRF (vector + graph + BM25), or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring`; list-set + `k` via injectable `RankConfig` | | `find` | Ladybug Cypher | required `NodeFilter`; strict per-kind frame | | `describe` | Ladybug Cypher | node record + `edge_summary` (composed/override rollups) | | `neighbors` | Ladybug Cypher | one hop; `direction` + `edge_types` required; dot-key composed edges | @@ -86,6 +86,8 @@ MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* **Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_lexical` (helpers from `search_scoring`). The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs. +**BM25 is also first-class on the primary (vector) path, not only the fallback.** `search_lancedb._graph_expand_merge` fuses **three** RRF lists — vector hits + graph-expand hits + BM25 hits — where the BM25 list is sourced from the same `sym_fts` index (via `search_lexical._try_fts_candidates`), resolved to chunk rows in BM25 rank order and re-filtered by the same LanceDB predicates as the vector list. The list-set and RRF `k` are runtime-injectable via `RankConfig` (`search_scoring.py`; default = 3-list, `k=60`), so the eval can A-B 2-list vs 3-list and sweep `k`. If the FTS extension/index is unavailable, the BM25 list is empty and the fusion degrades silently to the 2-list vector+graph ranking (no exception, no advisory) — so airgapped installs see no regression. Quality is measured by the **eval harness** (`java_codebase_rag.eval`: recall@k / precision@k / MRR over a corpus, with a Tier-A auto ground-truth derived per-Symbol and an optional Tier-B operator-authored file). On shopizer (n=400 of 2322 type-level symbols) the 3-list fusion at `k∈[60,120]` beats the 2-list baseline on every metric (MRR 0.3043→0.3082, recall@1 +0.005, recall@10 +0.0025) with **negligible latency cost** (~0 ms for the BM25 hop); `k=30` regresses recall@10 and is not used. The improvement is modest because Tier-A queries are identifier-derived (BM25's home turf) — a future Tier-B natural-language ground truth is expected to show a larger delta. + ### Watch path (`jrag watch`) — warm reads + freshness When a `jrag watch` daemon is running, the **read path gains a warm hop**: the `jrag` read handlers ask the daemon over a Unix socket for the already-built payload instead of cold-loading the model and graph. The daemon reuses the MCP server's warm-cache posture — a process-singleton `_st_model` (SBERT) and a `LadybugGraph` — served to the CLI, so each query skips the per-call torch/model load. Output is byte-identical to the cold path (the same payload cores in `read_payloads.py` run either way). With no daemon running, the client transparently takes the cold path — the daemon is a pure accelerator, never a dependency. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 50b37b1..966ac65 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -19,7 +19,7 @@ One repo, two stores, two audiences: ## Core principles 1. **Deterministic extraction, not LLM extraction.** tree-sitter parses every file; a two-phase build (parse all nodes, then resolve edges against the complete registry) eliminates forward-reference gaps. Reproducible, runs in seconds, no ~30% silent file-skip rate. (DKB = Deterministic Knowledge Base; benchmark + rationale in `docs/paper`.) -2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. +2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. Lexical (BM25) is a **first-class** third signal on the primary path, not just the macOS-Intel fallback: the vector read path fuses vector + graph-expand + BM25 (Okapi BM25 over the `Symbol.search_text` FTS index), because exact-identifier matches (`DistributionChunkService`) are where dense embeddings are weakest and BM25 is strongest — the standard hybrid win for code retrieval. On shopizer this beats the 2-list baseline on every ranking metric at ~0 latency cost; the list-set and RRF `k` are runtime-injectable (`RankConfig`) and measured by an in-repo eval harness (recall/precision@k, MRR). 3. **Walk at read time; don't precompute answers.** `neighbors` is exactly one hop. Multi-hop traces, impact analysis, "explain feature X" are the **agent's** reasoning over repeated one-hop calls. There is deliberately no magic impact/trace tool. 4. **Static analysis is a lower bound.** `CALLS` excludes reflection, Spring AOP proxies, dynamic dispatch. `resolved=false` means *external* (JDK/Spring), not *missing*. Never present this as proof of a runtime call path. 5. **Empty results must be honest, not silent.** Every empty hit is classified: `correct_empty` (genuine leaf), `not_in_project`, `external_dependency`, or `refine_query` — with did-you-mean, vocabulary context, and (for hard absence) an auditable proof. From e2e87d5bf7e682d0c0a2ccf786b62b1daaacdd99 Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 14:05:55 +0300 Subject: [PATCH 14/15] fix(search): review follow-ups (A-I1 build_fts_query + eval operability + test rigor) PR #443 hybrid BM25+RRF review fixes: - A-I1 (Important): _bm25_candidate_rows passed the RAW query to fetch_fts_candidates; LadybugDB FTS does not split camelCase, so a query like DistributionChunkService matched nothing and BM25 silently no-op'd on the feature's target queries. Pre-split with search_scoring.build_fts_query (mirrors the lexical backend) and return [] early on a degenerate split. Added an integration test that runs a camelCase query through the UN-stubbed FTS path (real LadybugDB sym_fts index) and asserts BM25 contributed. - C-I1 (Important): eval/runner.py had no __main__, so `python -m java_codebase_rag.eval.runner` was a silent no-op. Added a minimal argparse CLI (corpus_dir, --index-dir, --results-dir, --max-queries, --ks, --tier-b, --device) that builds an EvalConfig, calls run_eval, prints out_dir; exit 1 + traceback on failure. - C-I2 (Important): run_eval called load_tier_b unconditionally on a truthy tier_b_path, raising FileNotFoundError despite the docstring's "missing = Tier-B disabled" contract. Now guards on Path.exists(); load_tier_b still raises when called directly. - D-I1 (Important): the _RecordingDb fake's _eval_pred destroyed parens before the "IN (" check, so primary_type_fqn IN (...) never evaluated. Rewrote to split on AND without paren-destruction and strip one outer paren layer. Added a test where ordered_types excludes a fetched row's FQN and asserts it is NOT emitted. - D-I2 (Important): test_bm25_candidate_rows_orders_by_bm25_score supplied rows already in score-desc order (tautological). Scrambled to [C(10),A(20),B(30)] so only a real score-sort yields the asserted [B,A,C]. - D-I3 (Important): dedup-keep-max test listed the max-score member first, so a first-wins bug also yielded 30. Reversed to [sm2(20),sm1(30)] so the ==30.0 assertion discriminates keep-max from first-wins. - B-M1 (Minor): rewrote stale "inert until Task 4" comments in search_scoring to reflect that the bm25 list is now wired in _graph_expand_merge. Existing _bm25_candidate_rows unit tests updated from the degenerate placeholder query "q" to "query" (build_fts_query now drops single-char tokens, returning early). Removed a pre-existing unused DEFAULT_RANK_CONFIG import from runner.py. Co-Authored-By: Claude --- src/java_codebase_rag/eval/runner.py | 67 ++++- .../search/search_lancedb.py | 13 +- .../search/search_scoring.py | 15 +- tests/search/test_search_lancedb.py | 284 +++++++++++++++++- 4 files changed, 354 insertions(+), 25 deletions(-) diff --git a/src/java_codebase_rag/eval/runner.py b/src/java_codebase_rag/eval/runner.py index 6a098d6..87a71c2 100644 --- a/src/java_codebase_rag/eval/runner.py +++ b/src/java_codebase_rag/eval/runner.py @@ -36,12 +36,14 @@ from __future__ import annotations +import argparse import json import os import subprocess import sys import tempfile import time +import traceback from dataclasses import asdict, dataclass, field from datetime import datetime, timezone from pathlib import Path @@ -55,7 +57,6 @@ from java_codebase_rag.search.search_lancedb import run_search from java_codebase_rag.search.search_scoring import ( BASELINE_2LIST_CONFIG, - DEFAULT_RANK_CONFIG, RankConfig, ) @@ -430,7 +431,10 @@ def run_eval(cfg: EvalConfig) -> EvalReport: tier_a_sorted = sorted(tier_a, key=lambda q: (q.query, next(iter(q.relevant)))) tier_a_kept = tier_a_sorted[: cfg.max_queries] queries = list(tier_a_kept) - if cfg.tier_b_path: + # Tier-B is optional: a configured but missing path means "Tier-B disabled" + # (matches load_tier_b's docstring). load_tier_b itself still raises + # FileNotFoundError when called directly on a genuinely missing path. + if cfg.tier_b_path and Path(cfg.tier_b_path).exists(): queries.extend(load_tier_b(cfg.tier_b_path)) # 4. Enumerate configs: BASELINE_2LIST_CONFIG (k=60) + 3-list at each swept k. @@ -491,3 +495,62 @@ def _persist(report: EvalReport, results_dir: str) -> None: out.mkdir(parents=True, exist_ok=True) (out / "report.md").write_text(_render_markdown(report)) (out / "report.json").write_text(report.to_json()) + + +def _build_eval_config_from_args(argv: list[str] | None = None) -> EvalConfig: + """Parse CLI args into an EvalConfig. Defaults mirror EvalConfig fields.""" + # Start from defaults so unspecified flags inherit EvalConfig's own defaults. + base = EvalConfig() + parser = argparse.ArgumentParser( + prog="python -m java_codebase_rag.eval.runner", + description="Index a corpus, sweep RankConfigs, emit recall/precision/MRR.", + ) + parser.add_argument( + "corpus_dir", + nargs="?", + default=base.corpus_dir, + help=f"Java repo to index (default: {base.corpus_dir})", + ) + parser.add_argument("--index-dir", default=base.index_dir, + help="Lance+Ladybug index dir (default: temp dir).") + parser.add_argument("--results-dir", default=base.results_dir, + help=f"Where to write report.{{md,json}} (default: {base.results_dir})") + parser.add_argument("--max-queries", type=int, default=base.max_queries, + help=f"Cap on Tier-A queries (default: {base.max_queries}).") + parser.add_argument( + "--ks", default=",".join(str(k) for k in base.ks), + help="Comma-separated RRF k constants to sweep (default: %(default)s).", + ) + parser.add_argument("--tier-b", default=base.tier_b_path, + help="Optional path to a Tier-B ground-truth file (missing ⇒ disabled).") + parser.add_argument("--device", default=base.device, + help="SBERT device (default: SBERT_DEVICE env or auto).") + args = parser.parse_args(argv) + + try: + ks = tuple(int(k.strip()) for k in args.ks.split(",") if k.strip()) + except ValueError: + parser.error(f"--ks must be comma-separated ints, got {args.ks!r}") + if not ks: + parser.error("--ks must contain at least one value") + + return EvalConfig( + corpus_dir=args.corpus_dir, + index_dir=args.index_dir, + results_dir=args.results_dir, + tier_b_path=args.tier_b, + ks=ks, + max_queries=args.max_queries, + device=args.device, + ) + + +if __name__ == "__main__": + cfg = _build_eval_config_from_args() + try: + report = run_eval(cfg) + except Exception: + traceback.print_exc() + sys.exit(1) + print(report.out_dir) + sys.exit(0) diff --git a/src/java_codebase_rag/search/search_lancedb.py b/src/java_codebase_rag/search/search_lancedb.py index fd095ed..2003f67 100644 --- a/src/java_codebase_rag/search/search_lancedb.py +++ b/src/java_codebase_rag/search/search_lancedb.py @@ -32,6 +32,7 @@ DEFAULT_RANK_CONFIG, DEDUP_OVERFETCH, RankConfig, + build_fts_query, _ACTION_VERB_BONUS, _ACTION_VERB_PREFIXES, _HYBRID_SCORE_MAX, @@ -669,8 +670,18 @@ def _bm25_candidate_rows( degradation; the vector path is unaffected). """ # 1. BM25 candidate fetch via the FTS index. + # Pre-split the query with the same tokenizer the ``sym_fts`` index uses + # (``search_text`` stores ``_split_identifier`` tokens). LadybugDB FTS's own + # tokenizer does NOT split camelCase, so a raw ``DistributionChunkService`` + # would match nothing — ``build_fts_query`` mirrors what the lexical backend + # does at search_lexical.py (run_lexical_search), keeping index/query token + # spaces aligned. An empty split (degenerate / stopword-only query) → no FTS + # candidates → degrade silently to the vector path. + fts_query = build_fts_query(query) + if not fts_query or not fts_query.strip(): + return [] try: - fts = search_lexical.fetch_fts_candidates(g, query, filter=None, path_contains=None) + fts = search_lexical.fetch_fts_candidates(g, fts_query, filter=None, path_contains=None) except Exception as exc: # noqa: BLE001 — silent degradation _debug_ctx(f"bm25 FTS fetch raised: {exc!r}") return [] diff --git a/src/java_codebase_rag/search/search_scoring.py b/src/java_codebase_rag/search/search_scoring.py index 9868e40..5a63828 100644 --- a/src/java_codebase_rag/search/search_scoring.py +++ b/src/java_codebase_rag/search/search_scoring.py @@ -105,9 +105,9 @@ def _rrf_max(num_lists: int, k: int = 60) -> float: # Allowed list names in a RankConfig: vector is always required (it is the # backbone retrieval signal); graph and bm25 are optional fusion participants. -# NOTE: "bm25" is honored as a config element from Task 2, but the BM25 candidate -# fetch is wired only in Task 4 — until then a configured "bm25" list yields no -# candidates and behaves as a 2-list (vector+graph) fusion. +# The "bm25" list is wired in ``search_lancedb._graph_expand_merge`` (LadybugDB +# FTS candidate fetch), which fuses BM25-ranked Symbol candidates as a third +# RRF list alongside vector and graph. _RANK_LIST_NAMES: frozenset[str] = frozenset({"vector", "graph", "bm25"}) @@ -147,13 +147,14 @@ def __post_init__(self) -> None: raise ValueError(f"RankConfig.rrf_k must be an int >= 1, got {self.rrf_k!r}") -# Production default: ship the 3-list config (vector+graph+bm25). The "bm25" -# element is inert until Task 4 wires the BM25 candidate fetch; until then this -# is effectively a 2-list (vector+graph) fusion, identical to pre-Task-2 behavior. +# Production default: ship the 3-list config (vector+graph+bm25). The BM25 list +# is wired in ``search_lancedb._graph_expand_merge``; on installs without the +# vector stack, or when the FTS index is unavailable, it degrades silently to +# the 2-list (vector+graph) fusion. DEFAULT_RANK_CONFIG = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60) # Eval convenience: the historical 2-list (vector+graph) fusion, used by -# evaluation harnesses that do not want the bm25 list even after Task 4. +# evaluation harnesses that isolate the vector+graph baseline from the bm25 list. BASELINE_2LIST_CONFIG = RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=60) diff --git a/tests/search/test_search_lancedb.py b/tests/search/test_search_lancedb.py index a844c11..3c114e9 100644 --- a/tests/search/test_search_lancedb.py +++ b/tests/search/test_search_lancedb.py @@ -730,17 +730,31 @@ def test_refine_java_start_lines_skips_nonjava_and_method_chunks() -> None: def _eval_pred(rows: list[dict], pred: str | None) -> list[dict]: - """Tiny SQL-ish predicate evaluator for test fakes (AND / IN / <> / =).""" + """Tiny SQL-ish predicate evaluator for test fakes (AND / IN / <> / =). + + Splits on `` AND `` WITHOUT destroying parens (the prior implementation + pre-stripped ``(``/``)`` everywhere, which wiped the ``IN (...)`` marker so + the ``primary_type_fqn IN (...)`` filter never evaluated and rows that + should be filtered passed). ``_combine_predicates`` wraps each conjunct in a + paren pair when there are several, so we strip ONE outer paren layer per + clause while preserving the inner ``IN (...)`` parens needed to extract the + value list. + """ if not pred: return list(rows) + clauses: list[str] = [] + for c in pred.split(" AND "): + c = c.strip() + # Strip ONE outer wrapping paren pair (multi-predicate case); leave + # ``col IN (...)`` and its inner value-list parens intact. + if c.startswith("(") and c.endswith(")"): + c = c[1:-1].strip() + if c: + clauses.append(c) out: list[dict] = [] - clauses = [c.strip().strip("()") for c in pred.replace("(", " ").replace(")", " ").split("AND")] for r in rows: keep = True for clause in clauses: - clause = clause.strip() - if not clause: - continue if " IN (" in clause: col = clause.split(" IN ", 1)[0].strip() vals_part = clause[clause.index("(") + 1 : clause.rindex(")")] @@ -850,12 +864,17 @@ def _patch_bm25_environment(monkeypatch, *, fts_result, chunk_rows) -> _Recordin def test_bm25_candidate_rows_orders_by_bm25_score(monkeypatch) -> None: - """(a) BM25 candidates are emitted in BM25-desc order with the owning score attached.""" + """(a) BM25 candidates are emitted in BM25-desc order with the owning score attached. + + FTS rows are supplied SCRAMBLED relative to score order [C(10), A(20), B(30)] + so only a real score-desc sort produces the asserted [B, A, C] output — a + first-seen/passthrough bug would yield [C, A, B] instead. + """ fts_result = { "rows": [ - {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, - {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, {"id": "sc", "fqn": "com.x.C", "kind": "class", "name": "C"}, + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, ], "scores": {"sb": 30.0, "sa": 20.0, "sc": 10.0}, } @@ -867,7 +886,7 @@ def test_bm25_candidate_rows_orders_by_bm25_score(monkeypatch) -> None: db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) out = search_lancedb._bm25_candidate_rows( - g=object(), query="q", uri="mem://", db=db, + g=object(), query="query", uri="mem://", db=db, extra_predicates=[], columns={"primary_type_fqn"}, ) fqns = [r["primary_type_fqn"] for r in out] @@ -881,7 +900,7 @@ def test_bm25_candidate_rows_fts_unavailable_returns_empty(monkeypatch) -> None: db = _patch_bm25_environment(monkeypatch, fts_result=None, chunk_rows=[]) out = search_lancedb._bm25_candidate_rows( - g=object(), query="q", uri="mem://", db=db, + g=object(), query="query", uri="mem://", db=db, extra_predicates=[], columns={"primary_type_fqn"}, ) assert out == [] @@ -904,7 +923,7 @@ def test_bm25_candidate_rows_respects_filter(monkeypatch) -> None: db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) out = search_lancedb._bm25_candidate_rows( - g=object(), query="q", uri="mem://", db=db, + g=object(), query="query", uri="mem://", db=db, extra_predicates=["primary_type_fqn <> 'com.x.B'"], columns={"primary_type_fqn"}, ) @@ -912,6 +931,37 @@ def test_bm25_candidate_rows_respects_filter(monkeypatch) -> None: assert fqns == ["com.x.A"] +def test_bm25_candidate_rows_in_predicate_excludes_non_matching_types(monkeypatch) -> None: + """(c2) The ``primary_type_fqn IN (...)`` predicate (built from ordered_types) + excludes chunks whose FQN is not among the BM25-ordered types. + + Regression: the ``_RecordingDb`` fake's ``_eval_pred`` previously destroyed + parens before checking ``" IN ("``, so the IN predicate never evaluated and + non-matching chunks leaked through. FTS yields only ``com.x.A`` (so + ordered_types=['com.x.A'] and the IN predicate is ``primary_type_fqn IN + ('com.x.A')``); a ``com.x.B`` chunk sitting in the table must be filtered out. + """ + fts_result = { + "rows": [ + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + ], + "scores": {"sa": 20.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("b.java", "com.x.B", 1, 10), # NOT in ordered_types — must be filtered + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + fqns = [r["primary_type_fqn"] for r in out] + assert fqns == ["com.x.A"] + assert "com.x.B" not in fqns # filtered out by the IN (...) predicate + + def test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order(monkeypatch) -> None: """(d) Multiple chunks per type stay grouped, in table order, ordered by BM25 rank.""" fts_result = { @@ -929,7 +979,7 @@ def test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order(monkeypat db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) out = search_lancedb._bm25_candidate_rows( - g=object(), query="q", uri="mem://", db=db, + g=object(), query="query", uri="mem://", db=db, extra_predicates=[], columns={"primary_type_fqn"}, ) keys = [(r["primary_type_fqn"], r["range_start"]) for r in out] @@ -943,11 +993,16 @@ def test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order(monkeypat def test_bm25_candidate_rows_dedup_members_of_same_type_keep_max(monkeypatch) -> None: """(d2) Two member symbols (#-fqns) of the SAME type dedup to one type entry, - emitted ONCE at the MAX BM25 score among the members.""" + emitted ONCE at the MAX BM25 score among the members. + + FTS rows are ordered LOWER-score-first [sm2(20), sm1(30)] so a first-wins + bug would yield 20.0 and only a keep-max implementation yields the asserted + 30.0 — this discriminates keep-max from first-seen-wins. + """ fts_result = { "rows": [ - {"id": "sm1", "fqn": "com.x.A#method1()", "kind": "method", "name": "method1"}, {"id": "sm2", "fqn": "com.x.A#method2()", "kind": "method", "name": "method2"}, + {"id": "sm1", "fqn": "com.x.A#method1()", "kind": "method", "name": "method1"}, ], "scores": {"sm1": 30.0, "sm2": 20.0}, } @@ -958,7 +1013,7 @@ def test_bm25_candidate_rows_dedup_members_of_same_type_keep_max(monkeypatch) -> db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) out = search_lancedb._bm25_candidate_rows( - g=object(), query="q", uri="mem://", db=db, + g=object(), query="query", uri="mem://", db=db, extra_predicates=[], columns={"primary_type_fqn"}, ) # The type appears exactly once (both member chunks present, but only one type rank). @@ -1135,3 +1190,202 @@ def test_run_search_bm25_degrades_silently_when_fts_missing(monkeypatch, tmp_pat # Degradation: 3-list with FTS-missing == 2-list baseline. assert [r["filename"] for r in three] == [r["filename"] for r in two] assert len(three) == len(two) and len(two) >= 1 + + +def _build_minimal_fts_graph(db_path, *, symbol_fqn: str, symbol_name: str) -> bool: + """Build a 1-Symbol LadybugDB graph with the real ``sym_fts`` BM25 index. + + Returns True when the FTS index was created (extension available); False when + the FTS extension could not load (so the caller can ``pytest.skip``). + + Mirrors the production write path: ``_drop_all`` + ``_create_schema`` for the + table structure, one Symbol node + one GraphMeta node, then + ``_ensure_symbol_fts_index`` to build the BM25 index over ``search_text``. + """ + import ladybug + + from java_codebase_rag.ast.ast_java import ONTOLOGY_VERSION + from java_codebase_rag.graph.build_ast_graph import ( + _compute_symbol_search_text, + _create_schema, + _drop_all, + _ensure_symbol_fts_index, + ) + from java_codebase_rag.search.search_scoring import SYMBOL_FTS_INDEX + + db_path.parent.mkdir(parents=True, exist_ok=True) + db = ladybug.Database(str(db_path)) + conn = ladybug.Connection(db) + try: + _drop_all(conn) + _create_schema(conn) + search_text = _compute_symbol_search_text( + name=symbol_name, fqn=symbol_fqn, signature="", + annotations=[], capabilities=[], package=symbol_fqn.rsplit(".", 1)[0], + ) + conn.execute( + "MERGE (s:Symbol {id: $id}) " + "SET s.kind = 'class', s.name = $name, s.fqn = $fqn, " + "s.package = $package, s.module = '', s.microservice = '', " + "s.filename = $filename, s.start_line = 1, s.end_line = 20, " + "s.start_byte = 0, s.end_byte = 200, " + "s.modifiers = $modifiers, s.annotations = $annotations, " + "s.capabilities = $capabilities, s.role = 'SERVICE', " + "s.signature = '', s.parent_id = '', s.resolved = false, " + "s.generated = false, s.generated_by = '', " + "s.search_text = $search_text", + { + "id": symbol_fqn, "name": symbol_name, "fqn": symbol_fqn, + "package": symbol_fqn.rsplit(".", 1)[0], + "filename": f"{symbol_name}.java", + "modifiers": [], "annotations": [], "capabilities": [], + "search_text": search_text, + }, + ) + # GraphMeta with current ontology so LadybugGraph.get() accepts it. + conn.execute( + "MERGE (m:GraphMeta {key: 'meta'}) " + "SET m.ontology_version = $ov, m.built_at = 0, m.source_root = '', " + "m.counts_json = '', m.parse_errors = 0", + {"ov": ONTOLOGY_VERSION}, + ) + _ensure_symbol_fts_index(conn, verbose=False) + idx = conn.execute("CALL SHOW_INDEXES() RETURN index_name") + names: set[str] = set() + while idx.has_next(): + names.add(idx.get_next()[0]) + return SYMBOL_FTS_INDEX in names + finally: + conn.close() + db.close() + + +def test_run_search_bm25_contributes_on_camelcase_query_via_real_fts(tmp_path) -> None: + """(h) A camelCase identifier query lands in sym_fts's token space and the BM25 + list contributes to the 3-list fusion. + + Regression for A-I1: ``_bm25_candidate_rows`` used to pass the RAW query to + ``fetch_fts_candidates``; LadybugDB FTS does not split camelCase, so a query + like ``DistributionChunkService`` matched nothing and BM25 silently no-op'd. + With the ``build_fts_query`` pre-split, the FTS path matches. + + Exercises the UN-stubbed FTS path: a real LadybugDB ``sym_fts`` index on a + 1-Symbol graph + a real LanceDB index with a matching chunk. Does NOT + monkeypatch ``fetch_fts_candidates``. + + BM25 contribution is asserted two ways: + 1. Direct (un-stubbed) ``fetch_fts_candidates(g, build_fts_query(name))`` + returns the namesake Symbol with a positive BM25 score; the RAW query + returns nothing (the bug). + 2. ``run_search`` under the 3-list config produces a row whose + ``_score_components`` carries ``rrf_raw`` — set only by ``_rrf_merge`` + when the bm25 list joined the fusion. The graph is edge-less, so the + second list is necessarily bm25; the 2-list baseline has no ``rrf_raw``. + """ + import uuid + + import lancedb + from sentence_transformers import SentenceTransformer + + from java_codebase_rag.ast.ast_java import ONTOLOGY_VERSION + from java_codebase_rag.graph.ladybug_queries import LadybugGraph + from java_codebase_rag.search import search_lexical + from java_codebase_rag.search.index_common import SBERT_MODEL + from java_codebase_rag.search.search_lancedb import TABLES, _query_vector + from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + build_fts_query, + ) + + symbol_name = "DistributionChunkService" # multi-token camelCase — the trigger + symbol_fqn = f"smoke.{symbol_name}" + + # 1. Real LadybugDB graph: 1 Symbol + sym_fts BM25 index. + ladybug_path = tmp_path / "code_graph.lbug" + fts_ready = _build_minimal_fts_graph( + ladybug_path, symbol_fqn=symbol_fqn, symbol_name=symbol_name, + ) + if not fts_ready: + pytest.skip("FTS extension unavailable in this environment") + # build_fts_query must actually split the identifier (else the test is moot). + assert build_fts_query(symbol_name) == "distribution chunk service" + + LadybugGraph.reset_for_path(None) + g = LadybugGraph.get(str(ladybug_path)) + + # 2. Direct (un-stubbed) FTS sanity check: pre-split matches, raw does not. + pre = search_lexical.fetch_fts_candidates( + g, build_fts_query(symbol_name), filter=None, path_contains=None, + ) + assert pre and pre["rows"], "pre-split query must match via the real sym_fts index" + assert symbol_fqn in {r["fqn"] for r in pre["rows"]} + assert any(v > 0.0 for v in (pre.get("scores") or {}).values()) + raw = search_lexical.fetch_fts_candidates( + g, symbol_name, filter=None, path_contains=None, + ) + assert not raw or not raw.get("rows"), ( + "raw camelCase query should NOT match (FTS does not split camelCase); " + f"got: {raw}" + ) + + # 3. Real LanceDB index: 1 chunk matching the Symbol's enclosing type. + uri = str(tmp_path / "ldb") + model = SentenceTransformer(SBERT_MODEL, device="cpu", trust_remote_code=True) + text = "service that distributes chunks across the pipeline stages" + emb = _query_vector(model, text) + row = { + "id": str(uuid.uuid4()), + "filename": f"smoke/{symbol_name}.java", + "text": text, + "language": "java", + "range_start": 0, + "range_end": 500, + "start": {"line": 1, "byte_offset": 0}, + "end": {"line": 20, "byte_offset": 400}, + "embedding": emb, + "package": "smoke", + "module": "smoke", + "microservice": "smoke", + "primary_type_fqn": symbol_fqn, + "primary_type_kind": "class", + "role": "SERVICE", + "annotations_on_type": [], + "symbols": ["distribute"], + "ontology_version": ONTOLOGY_VERSION, + "capabilities": [], + } + db = lancedb.connect(uri) + db.create_table(TABLES["java"], [row], mode="create") + + common = dict( + uri=uri, table_keys=["java"], limit=5, path_substring=None, + model_name=SBERT_MODEL, device="cpu", model=model, + graph_expand=True, expand_depth=1, ladybug_path=str(ladybug_path), + ) + # 4. 3-list run: bm25 joins the fusion → _rrf_merge runs → rrf_raw appears. + three = run_search(symbol_name, rank_config=DEFAULT_RANK_CONFIG, **common) + assert three, "expected non-empty results for the namesake query" + has_rrf_raw = [ + r for r in three if "rrf_raw" in (r.get("_score_components") or {}) + ] + assert has_rrf_raw, ( + "3-list run should reflect bm25 fusion (rrf_raw set by _rrf_merge); " + f"components: {[r.get('_score_components') for r in three]}" + ) + # rrf_raw > 0 confirms the bm25 list contributed a positive rank contribution. + assert any( + float((r.get("_score_components") or {}).get("rrf_raw", 0.0)) > 0.0 + for r in has_rrf_raw + ) + # The namesake type is present in the 3-list result. + assert symbol_fqn in {r.get("primary_type_fqn") for r in three} + + # 5. 2-list baseline (vector+graph): edge-less graph → no fusion → no rrf_raw. + # Drop the cached singleton so the baseline run reopens cleanly. + LadybugGraph.reset_for_path(None) + two = run_search(symbol_name, rank_config=BASELINE_2LIST_CONFIG, **common) + assert all( + "rrf_raw" not in (r.get("_score_components") or {}) for r in two + ), "2-list baseline should not produce rrf_raw (graph is edge-less)" + From da3fd57320868293317c2171334aaa24b3cb24fc Mon Sep 17 00:00:00 2001 From: Dmitry Teryaev Date: Sun, 12 Jul 2026 14:28:49 +0300 Subject: [PATCH 15/15] =?UTF-8?q?docs(search):=20corrected=20post-fix=20ev?= =?UTF-8?q?al=20numbers=20=E2=80=94=20decisive=20BM25=20win=20(issue=20#43?= =?UTF-8?q?1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-A-I1-fix eval (build_fts_query now pre-splits camelCase): 3-list fusion at k=60 more than DOUBLES MRR (0.3044->0.6205, +104%) and recall@1 (0.220->0.490, +123%), recall@10 0.535->0.860. Latency cost ~+25-30ms p50 (~10%), not 0 as the bug-muted prior measurement suggested. Ship k=60 (conservative; k=30 narrowly best on identifier-heavy Tier-A, within noise). Records the query-preprocessing bug that muted the initial eval. Co-Authored-By: Claude --- docs/ARCHITECTURE.md | 2 +- docs/DESIGN.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index e2ebf90..5e4aa2c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -86,7 +86,7 @@ MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* **Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_lexical` (helpers from `search_scoring`). The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs. -**BM25 is also first-class on the primary (vector) path, not only the fallback.** `search_lancedb._graph_expand_merge` fuses **three** RRF lists — vector hits + graph-expand hits + BM25 hits — where the BM25 list is sourced from the same `sym_fts` index (via `search_lexical._try_fts_candidates`), resolved to chunk rows in BM25 rank order and re-filtered by the same LanceDB predicates as the vector list. The list-set and RRF `k` are runtime-injectable via `RankConfig` (`search_scoring.py`; default = 3-list, `k=60`), so the eval can A-B 2-list vs 3-list and sweep `k`. If the FTS extension/index is unavailable, the BM25 list is empty and the fusion degrades silently to the 2-list vector+graph ranking (no exception, no advisory) — so airgapped installs see no regression. Quality is measured by the **eval harness** (`java_codebase_rag.eval`: recall@k / precision@k / MRR over a corpus, with a Tier-A auto ground-truth derived per-Symbol and an optional Tier-B operator-authored file). On shopizer (n=400 of 2322 type-level symbols) the 3-list fusion at `k∈[60,120]` beats the 2-list baseline on every metric (MRR 0.3043→0.3082, recall@1 +0.005, recall@10 +0.0025) with **negligible latency cost** (~0 ms for the BM25 hop); `k=30` regresses recall@10 and is not used. The improvement is modest because Tier-A queries are identifier-derived (BM25's home turf) — a future Tier-B natural-language ground truth is expected to show a larger delta. +**BM25 is also first-class on the primary (vector) path, not only the fallback.** `search_lancedb._graph_expand_merge` fuses **three** RRF lists — vector hits + graph-expand hits + BM25 hits — where the BM25 list is sourced from the same `sym_fts` index (via `search_lexical._try_fts_candidates`), resolved to chunk rows in BM25 rank order and re-filtered by the same LanceDB predicates as the vector list. The list-set and RRF `k` are runtime-injectable via `RankConfig` (`search_scoring.py`; default = 3-list, `k=60`), so the eval can A-B 2-list vs 3-list and sweep `k`. If the FTS extension/index is unavailable, the BM25 list is empty and the fusion degrades silently to the 2-list vector+graph ranking (no exception, no advisory) — so airgapped installs see no regression. Quality is measured by the **eval harness** (`java_codebase_rag.eval`: recall@k / precision@k / MRR over a corpus, with a Tier-A auto ground-truth derived per-Symbol and an optional Tier-B operator-authored file). On shopizer (n=400 of 2322 type-level symbols) the 3-list fusion at `k=60` decisively beats the 2-list baseline on every metric: **MRR 0.3044→0.6205 (+104%)**, **recall@1 0.220→0.490 (+123%)**, recall@10 0.535→0.860 (+61%), recall@20→0.905. The gain is large because Tier-A queries are identifier-derived (BM25's home turf) — exact-identifier matches anchor the dense ranking exactly as the hybrid thesis predicts; a future Tier-B natural-language ground truth is expected to show a smaller-but-positive delta (NL queries favor semantic vectors). The BM25 hop costs **~+25-30 ms p50 (~10%)** per query. `k∈{30,60,90,120}` all beat baseline; `k=30` narrowly edges `k=60` on this identifier-heavy eval (MRR 0.631 vs 0.620, within noise on n=400), but `k=60` ships as the conservative, regime-robust choice — re-tune when Tier-B NL ground truth exists. (An initial eval run reported a much smaller delta; that was muted by a query-preprocessing bug — camelCase identifiers weren't reaching the FTS tokenizer — fixed in `search_lancedb._bm25_candidate_rows` via `search_scoring.build_fts_query`.) ### Watch path (`jrag watch`) — warm reads + freshness diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 966ac65..86d3f1e 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -19,7 +19,7 @@ One repo, two stores, two audiences: ## Core principles 1. **Deterministic extraction, not LLM extraction.** tree-sitter parses every file; a two-phase build (parse all nodes, then resolve edges against the complete registry) eliminates forward-reference gaps. Reproducible, runs in seconds, no ~30% silent file-skip rate. (DKB = Deterministic Knowledge Base; benchmark + rationale in `docs/paper`.) -2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. Lexical (BM25) is a **first-class** third signal on the primary path, not just the macOS-Intel fallback: the vector read path fuses vector + graph-expand + BM25 (Okapi BM25 over the `Symbol.search_text` FTS index), because exact-identifier matches (`DistributionChunkService`) are where dense embeddings are weakest and BM25 is strongest — the standard hybrid win for code retrieval. On shopizer this beats the 2-list baseline on every ranking metric at ~0 latency cost; the list-set and RRF `k` are runtime-injectable (`RankConfig`) and measured by an in-repo eval harness (recall/precision@k, MRR). +2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. Lexical (BM25) is a **first-class** third signal on the primary path, not just the macOS-Intel fallback: the vector read path fuses vector + graph-expand + BM25 (Okapi BM25 over the `Symbol.search_text` FTS index), because exact-identifier matches (`DistributionChunkService`) are where dense embeddings are weakest and BM25 is strongest — the standard hybrid win for code retrieval. On shopizer (identifier-derived Tier-A ground truth) this **more than doubles** MRR and recall@1 vs the 2-list baseline (exact-identifier anchoring), at a modest ~10% per-query latency cost; the list-set and RRF `k` are runtime-injectable (`RankConfig`) and measured by an in-repo eval harness (recall/precision@k, MRR). 3. **Walk at read time; don't precompute answers.** `neighbors` is exactly one hop. Multi-hop traces, impact analysis, "explain feature X" are the **agent's** reasoning over repeated one-hop calls. There is deliberately no magic impact/trace tool. 4. **Static analysis is a lower bound.** `CALLS` excludes reflection, Spring AOP proxies, dynamic dispatch. `resolved=false` means *external* (JDK/Spring), not *missing*. Never present this as proof of a runtime call path. 5. **Empty results must be honest, not silent.** Every empty hit is classified: `correct_empty` (genuine leaf), `not_in_project`, `external_dependency`, or `refine_query` — with did-you-mean, vocabulary context, and (for hard absence) an auditable proof.