diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a6c854d..5e4aa2c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -68,7 +68,7 @@ java_codebase_rag/pipeline.py ``` MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* - ├─ search ─▶ search_lancedb.run_search (vector / hybrid; optional graph-expand + RRF rank fusion) + ├─ search ─▶ search_lancedb.run_search (vector / hybrid; graph-expand + 3-list RRF: vector + graph + BM25) │ └─ lancedb import absent (Intel Mac) → search_lexical (BM25 over Symbol FTS index; heuristic scan fallback) ├─ find / describe / neighbors ─▶ ladybug_queries.LadybugGraph (Cypher) └─ resolve ─▶ resolve_service.resolve_v2 (cascade → status one | many | none) @@ -78,7 +78,7 @@ MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* | Tool | Backing | Notes | | --- | --- | --- | -| `search` | Lance vector/hybrid, or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring` | +| `search` | Lance vector/hybrid with 3-list RRF (vector + graph + BM25), or BM25 lexical fallback | dedup by FQN; role weights via `search_scoring`; list-set + `k` via injectable `RankConfig` | | `find` | Ladybug Cypher | required `NodeFilter`; strict per-kind frame | | `describe` | Ladybug Cypher | node record + `edge_summary` (composed/override rollups) | | `neighbors` | Ladybug Cypher | one hop; `direction` + `edge_types` required; dot-key composed edges | @@ -86,6 +86,8 @@ MCP tool call (server.py) ──asyncio.to_thread──▶ mcp_v2.* **Lexical fallback** is selected by import availability (`mcp_v2` guards `from search_lancedb import …`): same row contract, flagged via `lexical_mode` + advisory. It is **BM25-first**: `build_ast_graph` indexes `Symbol.search_text` (camelCase-split token soup) under a LadybugDB FTS index (`sym_fts`, Okapi BM25), and `search_lexical` fetches top-K candidates via `QUERY_FTS_INDEX` then re-ranks them with the name/type/fqn/role heuristic in `search_lexical` (helpers from `search_scoring`). The FTS index auto-maintains on `increment`; the heuristic scan is the fallback when the index/extension is absent (older graph, offline first run). **`jrag` CLI** calls the same `mcp_v2.*` functions — identical backends, only rendering differs. +**BM25 is also first-class on the primary (vector) path, not only the fallback.** `search_lancedb._graph_expand_merge` fuses **three** RRF lists — vector hits + graph-expand hits + BM25 hits — where the BM25 list is sourced from the same `sym_fts` index (via `search_lexical._try_fts_candidates`), resolved to chunk rows in BM25 rank order and re-filtered by the same LanceDB predicates as the vector list. The list-set and RRF `k` are runtime-injectable via `RankConfig` (`search_scoring.py`; default = 3-list, `k=60`), so the eval can A-B 2-list vs 3-list and sweep `k`. If the FTS extension/index is unavailable, the BM25 list is empty and the fusion degrades silently to the 2-list vector+graph ranking (no exception, no advisory) — so airgapped installs see no regression. Quality is measured by the **eval harness** (`java_codebase_rag.eval`: recall@k / precision@k / MRR over a corpus, with a Tier-A auto ground-truth derived per-Symbol and an optional Tier-B operator-authored file). On shopizer (n=400 of 2322 type-level symbols) the 3-list fusion at `k=60` decisively beats the 2-list baseline on every metric: **MRR 0.3044→0.6205 (+104%)**, **recall@1 0.220→0.490 (+123%)**, recall@10 0.535→0.860 (+61%), recall@20→0.905. The gain is large because Tier-A queries are identifier-derived (BM25's home turf) — exact-identifier matches anchor the dense ranking exactly as the hybrid thesis predicts; a future Tier-B natural-language ground truth is expected to show a smaller-but-positive delta (NL queries favor semantic vectors). The BM25 hop costs **~+25-30 ms p50 (~10%)** per query. `k∈{30,60,90,120}` all beat baseline; `k=30` narrowly edges `k=60` on this identifier-heavy eval (MRR 0.631 vs 0.620, within noise on n=400), but `k=60` ships as the conservative, regime-robust choice — re-tune when Tier-B NL ground truth exists. (An initial eval run reported a much smaller delta; that was muted by a query-preprocessing bug — camelCase identifiers weren't reaching the FTS tokenizer — fixed in `search_lancedb._bm25_candidate_rows` via `search_scoring.build_fts_query`.) + ### Watch path (`jrag watch`) — warm reads + freshness When a `jrag watch` daemon is running, the **read path gains a warm hop**: the `jrag` read handlers ask the daemon over a Unix socket for the already-built payload instead of cold-loading the model and graph. The daemon reuses the MCP server's warm-cache posture — a process-singleton `_st_model` (SBERT) and a `LadybugGraph` — served to the CLI, so each query skips the per-call torch/model load. Output is byte-identical to the cold path (the same payload cores in `read_payloads.py` run either way). With no daemon running, the client transparently takes the cold path — the daemon is a pure accelerator, never a dependency. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 50b37b1..86d3f1e 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -19,7 +19,7 @@ One repo, two stores, two audiences: ## Core principles 1. **Deterministic extraction, not LLM extraction.** tree-sitter parses every file; a two-phase build (parse all nodes, then resolve edges against the complete registry) eliminates forward-reference gaps. Reproducible, runs in seconds, no ~30% silent file-skip rate. (DKB = Deterministic Knowledge Base; benchmark + rationale in `docs/paper`.) -2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. +2. **Structure complements vectors — it does not replace them.** Two stores from the same sources. Semantic questions → vector; structural questions → graph; fused via RRF (Reciprocal Rank Fusion) only where each adds signal. Lexical (BM25) is a **first-class** third signal on the primary path, not just the macOS-Intel fallback: the vector read path fuses vector + graph-expand + BM25 (Okapi BM25 over the `Symbol.search_text` FTS index), because exact-identifier matches (`DistributionChunkService`) are where dense embeddings are weakest and BM25 is strongest — the standard hybrid win for code retrieval. On shopizer (identifier-derived Tier-A ground truth) this **more than doubles** MRR and recall@1 vs the 2-list baseline (exact-identifier anchoring), at a modest ~10% per-query latency cost; the list-set and RRF `k` are runtime-injectable (`RankConfig`) and measured by an in-repo eval harness (recall/precision@k, MRR). 3. **Walk at read time; don't precompute answers.** `neighbors` is exactly one hop. Multi-hop traces, impact analysis, "explain feature X" are the **agent's** reasoning over repeated one-hop calls. There is deliberately no magic impact/trace tool. 4. **Static analysis is a lower bound.** `CALLS` excludes reflection, Spring AOP proxies, dynamic dispatch. `resolved=false` means *external* (JDK/Spring), not *missing*. Never present this as proof of a runtime call path. 5. **Empty results must be honest, not silent.** Every empty hit is classified: `correct_empty` (genuine leaf), `not_in_project`, `external_dependency`, or `refine_query` — with did-you-mean, vocabulary context, and (for hard absence) an auditable proof. diff --git a/docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md b/docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md new file mode 100644 index 0000000..46bf1e2 --- /dev/null +++ b/docs/superpowers/plans/active/2026-07-12-hybrid-bm25-rrf.md @@ -0,0 +1,412 @@ +# Hybrid BM25 + Vector RRF Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Promote LadybugDB Okapi BM25 to a first-class, always-on third RRF list on the vector/hybrid search read path, and build a recall/precision@k + MRR eval harness to measure and tune it. + +**Architecture:** The vector path (`search_lancedb._graph_expand_merge`) currently fuses 2 lists (vector + graph-expand) via the N-list-generic `_rrf_merge`. We add a third list — BM25 candidates from the existing `search_lexical._try_fts_candidates` helper (fork A, merged) — resolved to chunk rows in BM25 rank order, filter the result through the same LanceDB predicates as the vector path, and fuse with RRF. A new dep-free `RankConfig` makes `{lists, rrf_k}` runtime-injectable so the eval can A-B 2-list vs 3-list and sweep `k`. The `score_components["bm25"]` entry surfaces under `explain=True`. + +**Tech Stack:** Python 3.11 (`.venv/bin/python`, editable install), LadybugDB 0.17.1 FTS (Okapi BM25) via the `sym_fts` index, LanceDB 0.34, pydantic v2, pytest. + +**Spec:** `docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md` (commit `8368e71`). **Issue:** #431. **Deferred follow-ups:** #434–#439. + +## Global Constraints + +- Use `.venv/bin/python` and `.venv/bin/pip` only — never system `python`/`pip`. Editable install only; if pytest complains about a stale install, run `.venv/bin/pip install -e ".[dev]"`. +- `search/search_scoring.py` MUST NOT import lancedb / torch / sentence_transformers / cocoindex — it is imported on graph-only (macOS Intel) installs where those are absent. `RankConfig` and any new metric helpers that the eval imports without the vector stack MUST live there (or another dep-free module). +- `SYMBOL_FTS_INDEX = "sym_fts"` (`search_scoring.py:20`) is the single source of truth for the FTS index name — never inline the string. +- Erase stale manual indexes before running tests: `rm -rf tests/*/.java-codebase-rag tests/*/.java-codebase-rag.{yml,hosts}`. Tests build their own index in a temp dir; never commit one under `tests/`. +- Develop against the search subset (`-k "lexical or search or hybrid or rrf"`); run the full suite once at the end. +- `SearchHit` / `SearchOutput` / `NodeFilter` schemas (`mcp/mcp_v2.py`) are UNCHANGED — `score_components` is an open dict. +- FTS unavailable must degrade SILENTLY to 2-list ranking on the vector path: no exception, no advisory, no exit-code change. + +--- + +## File Structure + +| Path | Responsibility | Status | +|---|---|---| +| `src/java_codebase_rag/search/search_scoring.py` | Dep-free scoring/dedup primitives. Add `RankConfig` dataclass; derive `_HYBRID_SCORE_MAX` RRF term from list count; add `bm25=` token to `explain_score_components`. | Modify | +| `src/java_codebase_rag/search/search_lancedb.py` | Vector/hybrid read path. Add `_bm25_candidate_rows` helper; thread `rank_config` through `run_search` → `_graph_expand_merge`; pass BM25 list to `_rrf_merge`. | Modify | +| `src/java_codebase_rag/search/search_lexical.py` | Lexical backend. Possibly widen visibility of `_try_fts_candidates` for cross-module reuse. | Modify (minimal) | +| `src/java_codebase_rag/eval/__init__.py` | Eval package marker. | Create | +| `src/java_codebase_rag/eval/metrics.py` | Dep-free IR metrics: recall@k, precision@k, MRR. | Create | +| `src/java_codebase_rag/eval/ground_truth.py` | Tier-A auto ground-truth generator (Symbol→query). Tier-B YAML/JSON loader. | Create | +| `src/java_codebase_rag/eval/runner.py` | Build shopizer index, run 2-list baseline + 3-list k-sweep via injected `RankConfig`, write Markdown + JSON results. | Create | +| `tests/search/test_search_scoring.py` | Unit tests for `RankConfig`, `_HYBRID_SCORE_MAX` derivation, `bm25=` token. | Modify | +| `tests/search/test_search_lancedb.py` | Unit + integration tests for `_bm25_candidate_rows`, rank-config injection, FTS-unavailable degradation. | Modify | +| `tests/eval/test_metrics.py`, `tests/eval/test_ground_truth.py`, `tests/eval/test_runner.py` | Unit tests for eval harness machinery (metric math, generator determinism, runner smoke). | Create | + +--- + +## Task 1: Derive `_HYBRID_SCORE_MAX` from RRF list count + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py:87-93` +- Test: `tests/search/test_search_scoring.py` + +**Interfaces:** +- Produces: a new module-level function `_rrf_max(num_lists: int, k: int = 60) -> float` returning `num_lists / (k + 1)`. `_HYBRID_SCORE_MAX` is redefined to use `_rrf_max(2)` (preserving today's exact `2.0/61.0` value) PLUS the unchanged bonus terms (`max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS`). The numeric value of `_HYBRID_SCORE_MAX` MUST stay byte-identical to today (it still describes the shipped 2-list hybrid display path until Task 4 flips production to 3-list). +- Consumes: nothing new. + +- [ ] **Step 1: Write failing tests** + +In `tests/search/test_search_scoring.py` add: + +(a) `test_rrf_max_formula`: assert `_rrf_max(2, 60)` equals `2.0 / 61.0` (within 1e-12); `_rrf_max(3, 60)` equals `3.0 / 61.0`; `_rrf_max(3, 30)` equals `3.0 / 31.0`. + +(b) `test_hybrid_score_max_unchanged`: assert the module attribute `_HYBRID_SCORE_MAX` equals the literal `(2.0/61.0) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS` computed in the test from the same imported constants — i.e. the refactor introduces no numeric drift. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py::test_rrf_max_formula -v` (and the second test). +Expected: FAIL — `_rrf_max` not defined (ImportError/AttributeError). + +- [ ] **Step 3: Implement** + +Add `_rrf_max(num_lists, k=60)` returning `num_lists / (k + 1)`. Redefine `_HYBRID_SCORE_MAX` to call `_rrf_max(2)` for the RRF term; leave the bonus terms and the surrounding comment unchanged except to note the RRF term is now derived. Do not change any other constant. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py -v` +Expected: PASS (both new tests + all existing scoring tests). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py tests/search/test_search_scoring.py` +Run: `git commit -m "refactor(scoring): derive _HYBRID_SCORE_MAX RRF term from list count"` + +--- + +## Task 2: Add `RankConfig` and thread it through the read path + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py` (add `RankConfig`) +- Modify: `src/java_codebase_rag/search/search_lancedb.py:635-646` (`_graph_expand_merge` signature), `:887-897` (`run_search` call site), and `run_search` signature (line ~764) +- Test: `tests/search/test_search_scoring.py`, `tests/search/test_search_lancedb.py` + +**Interfaces:** +- Produces (in `search_scoring.py`, dep-free): + - `@dataclass(frozen=True) class RankConfig:` with fields: + - `lists: frozenset[str]` — subset of `{"vector", "graph", "bm25"}`. Must contain `"vector"`. Validation in `__post_init__`: raise `ValueError` if `"vector"` not in `lists`, or if any element is outside the allowed set, or if `lists` is empty. + - `rrf_k: int = 60` — must be `>= 1` (else `ValueError`). + - `DEFAULT_RANK_CONFIG = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60)` — production default. (3-list is the shipped behavior after Task 4; until Task 4 lands the BM25 list, the `"bm25"` element is honored but yields an empty list, so behavior is effectively 2-list. This is intentional — Task 4 only adds the non-empty BM25 list.) + - `BASELINE_2LIST_CONFIG = RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=60)` — eval convenience. +- Produces (in `search_lancedb.py`): + - `_graph_expand_merge` gains keyword-only param `rank_config: RankConfig = DEFAULT_RANK_CONFIG`. It passes `k=rank_config.rrf_k` into `_rrf_merge`. It fuses exactly the lists named in `rank_config.lists` (vector always present; graph present unless omitted; bm25 handled in Task 4, empty until then). + - `run_search` gains keyword-only param `rank_config: RankConfig = DEFAULT_RANK_CONFIG`, forwarded to `_graph_expand_merge`. +- Consumes: Task 1's `_rrf_max`. + +- [ ] **Step 1: Write failing tests** + +(a) In `test_search_scoring.py`: +- `test_rank_config_defaults`: `DEFAULT_RANK_CONFIG.lists == frozenset({"vector","graph","bm25"})`, `.rrf_k == 60`. +- `test_rank_config_validation`: constructing `RankConfig(lists=frozenset({"graph"}))` raises `ValueError` (no vector); `RankConfig(lists=frozenset({"vector","nope"}))` raises `ValueError` (unknown list); `RankConfig(lists=frozenset({"vector"}), rrf_k=0)` raises `ValueError`. +- `test_rank_config_frozen`: mutating `DEFAULT_RANK_CONFIG.rrf_k = 5` raises `FrozenInstanceError`. + +(b) In `test_search_lancedb.py`: +- `test_graph_expand_merge_honors_injected_k` (monkeypatch-based): construct a tiny scenario where `_graph_expand_merge` is called with `rank_config=RankConfig(lists=frozenset({"vector","graph"}), rrf_k=30)`; assert the fused rows' `_score_components["rrf_raw"]` normalization reflects `k=30` (i.e. max = `2/31`), proving `k` is injected. Use the existing fixture/monkeypatch pattern already present in this test file for `_graph_expand_merge`; if none exists, build the minimal stub `LadybugGraph` + `_search_one_table` doubles the file already uses elsewhere. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py::test_rank_config_defaults tests/search/test_search_lancedb.py::test_graph_expand_merge_honors_injected_k -v` +Expected: FAIL — `RankConfig` not defined / `rank_config` param not accepted. + +- [ ] **Step 3: Implement** + +Add the `RankConfig` dataclass + the two module constants in `search_scoring.py` (with `from dataclasses import dataclass`). In `search_lancedb.py`: add the `rank_config` keyword-only param to `_graph_expand_merge` and `run_search`; pass `k=rank_config.rrf_k` to `_rrf_merge` in `_graph_expand_merge`; forward the param at the call site (line ~888). Do NOT yet add the BM25 list (Task 4) — this task only plumbs injection and keeps behavior identical to today because the default omits no behavior the current code has (graph still fuses; bm25 not yet fetched). + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/ -k "scoring or lancedb or hybrid" -v` +Expected: PASS — including the new tests, with zero regressions (behavior unchanged). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py src/java_codebase_rag/search/search_lancedb.py tests/search/test_search_scoring.py tests/search/test_search_lancedb.py` +Run: `git commit -m "feat(search): injectable RankConfig for RRF list-set and k"` + +--- + +## Task 3: Add `bm25=` token to `explain_score_components` + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py:342-393` +- Test: `tests/search/test_search_scoring.py` + +**Interfaces:** +- Produces: `explain_score_components` emits a `bm25=` token when called with `hybrid=True` and the component dict contains a truthy `"bm25"` key. Format: `bm25={float(value):.3f}`. It is appended after the existing `rrf=` token (within the `elif hybrid:` branch) and before the shared role/symbol/import tokens. When `"bm25"` is absent or zero, no token is emitted (consistent with how `symbol_bonus`/`role_weight` are conditionally shown). +- Consumes: nothing new. + +- [ ] **Step 1: Write failing tests** + +In `test_search_scoring.py`: +- `test_explain_bm25_token_present`: `explain_score_components({"rrf_raw": 0.03, "bm25": 12.5}, hybrid=True)` returns a string containing `"rrf=0.030"` AND `"bm25=12.500"` (order: rrf before bm25). +- `test_explain_bm25_token_absent_when_zero_or_missing`: `explain_score_components({"rrf_raw": 0.03}, hybrid=True)` does NOT contain `"bm25="`; same for `{"rrf_raw": 0.03, "bm25": 0.0}`. +- `test_explain_bm25_only_in_hybrid`: `explain_score_components({"bm25": 12.5}, lexical=True)` must NOT emit a `bm25=` token (the lexical branch keeps its existing `relevance=`/`name=` tokens — `bm25=` is hybrid-only). + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py::test_explain_bm25_token_present -v` +Expected: FAIL — no `bm25=` token produced. + +- [ ] **Step 3: Implement** + +In the `elif hybrid:` branch of `explain_score_components`, after appending the `rrf=` token, read `comps.get("bm25")` and, when truthy, append `f"bm25={float(bm25):.3f}"`. No other branch changes. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/test_search_scoring.py -v` +Expected: PASS (new + existing explain tests). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py tests/search/test_search_scoring.py` +Run: `git commit -m "feat(scoring): surface bm25= token in hybrid explain output"` + +--- + +## Task 4: BM25 candidate fetch + wire as third RRF list + +**Files:** +- Modify: `src/java_codebase_rag/search/search_lancedb.py` (new `_bm25_candidate_rows` near `_graph_expand_merge` line 635; edits inside `_graph_expand_merge` at 709-716; default config already 3-list from Task 2) +- Modify: `src/java_codebase_rag/search/search_lexical.py:210` (widen visibility of `_try_fts_candidates` if needed for cross-module import — e.g. expose a thin public alias, do not change behavior) +- Test: `tests/search/test_search_lancedb.py` + +**Interfaces:** + +- Produces (in `search_lancedb.py`): + - `def _bm25_candidate_rows(*, g: "LadybugGraph", query: str, uri: str, db: object, extra_predicates: list[str], columns: list[str]) -> list[dict]:` + - Behavior: + 1. Call `search_lexical._try_fts_candidates(g, query, filter=None, path_contains=None)`. If it returns `None` (FTS extension/index unavailable) OR its `"rows"` is empty, return `[]`. + 2. From the result: `"scores"` is `{symbol_node_id: bm25_float}`, `"rows"` is a list of Symbol dicts each carrying at least `id` and a fully-qualified name field (the Symbol FQN — use the same key the lexical backend exposes for the FQN; confirm against `_SYMBOL_RETURN` in `search_lexical.py`). Build `ordered_fqns`: the distinct FQNs of the returned symbols, sorted by their BM25 score descending (ties broken by FQN ascending for determinism). Build `fqn_to_bm25: dict[str,float]` mapping each FQN to its max BM25 score among its symbols. + 3. Fetch chunk rows from LanceDB for exactly those FQNs, filter-respecting: build `preds = list(extra_predicates) + _build_extra_predicates(columns=columns, fqn_in=ordered_fqns)` and query the `java` table WITHOUT a vector ranking (filter-only `.where()` query through the same table-access pattern `_search_one_table` uses to open the table — but omitting the vector search so the rows are not re-ranked by similarity). Return columns must include at least `filename`, `range_start`, `range_end`, `primary_type_fqn`, plus whatever `_apply_chunk_hints` / `_refine_java_start_lines` need (mirror the column set `_search_one_table` returns for `java`). + 4. Group fetched chunks by `primary_type_fqn`. Emit chunk rows ordered by the BM25 rank of their owning FQN: iterate `ordered_fqns`; for each FQN that has chunks, emit its chunks in their natural table order. Each emitted chunk row is the same dict shape as a `vector_row`/`graph_row` and additionally has `_score_components["bm25"] = round(float(fqn_to_bm25[fqn]), 4)`. Chunks whose FQN was filtered out by `extra_predicates` are never fetched, so filter parity with the vector path is automatic. + 5. Return the ordered chunk-row list (may be empty). + - Error handling: any exception from the FTS call or the LanceDB fetch is caught and the function returns `[]` (silent degradation — matches the spec's "FTS failure is a silent no-op on the vector path"). Log via `_debug_ctx` (the existing helper in this module) at debug level. + + - `_graph_expand_merge` change: when `"bm25"` is in `rank_config.lists`, after computing `graph_rows`, call `_bm25_candidate_rows(g=g, query=query, uri=uri, db=db, extra_predicates=extra_predicates, columns=_table_columns(uri, TABLES["java"], db))`. Then, when building the lists for `_rrf_merge`, pass `[vector_rows, graph_rows, bm25_rows]` with `k=rank_config.rrf_k`. The `query` string must be threaded into `_graph_expand_merge` as a new keyword-only param (it is not currently a param — add `query: str` to the signature and forward it from `run_search`, which already has `query`). When `"bm25"` is NOT in `rank_config.lists`, omit the BM25 list entirely (2-list behavior, used by the eval baseline). + - Note on `graph_rows` presence: if `"graph"` is in `rank_config.lists` the graph list is fetched as today; if omitted, only vector + (optionally) bm25 are fused. The `vector` list is always present (validated by `RankConfig`). + +- Consumes: Task 2's `RankConfig` / `DEFAULT_RANK_CONFIG`; `search_lexical._try_fts_candidates`; `search_scoring.SYMBOL_FTS_INDEX`; `_build_extra_predicates`, `_table_columns`, `_debug_ctx`, `TABLES`. + +- [ ] **Step 1: Write failing tests** + +In `test_search_lancedb.py` (monkeypatch `LadybugGraph`, `_search_one_table`/table access, and `search_lexical._try_fts_candidates` — use the doubling patterns already in this file): + +(a) `test_bm25_candidate_rows_orders_by_bm25_score`: stub `_try_fts_candidates` to return symbols for FQNs `["B", "A", "C"]` with BM25 scores `{B: 30, A: 20, C: 10}`; stub the LanceDB chunk fetch to return one chunk per FQN. Assert the returned list is ordered `B, A, C` (BM25 desc) and each row's `_score_components["bm25"]` equals the owning FQN's score (B→30.0, A→20.0, C→10.0). + +(b) `test_bm25_candidate_rows_fts_unavailable_returns_empty`: stub `_try_fts_candidates` to return `None`; assert `_bm25_candidate_rows(...)` returns `[]` and no LanceDB fetch is attempted (assert the table-access double is never called). + +(c) `test_bm25_candidate_rows_respects_filter`: stub `_try_fts_candidates` to return FQNs `["A","B"]`; pass `extra_predicates=["primary_type_fqn <> 'B'"]` (or the project's equivalent SQL form) and assert only FQN `A`'s chunks are emitted (the predicate is applied at the chunk fetch, so B is filtered). + +(d) `test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order`: stub `_try_fts_candidates` returning FQN `A` (score 20) and FQN `B` (score 10); stub the chunk fetch to return 2 chunks for A and 1 for B. Assert output order is `[A_chunk1, A_chunk2, B_chunk1]` and all three carry the correct `bm25` component. + +(e) `test_graph_expand_merge_includes_bm25_list`: with `rank_config=DEFAULT_RANK_CONFIG` (3-list), monkeypatch `_bm25_candidate_rows` to return a known non-empty chunk list, and assert the fused result contains rows whose `_score_components` includes contributions from the BM25 list (a row appearing ONLY in the BM25 list is present in the fused output and has `_score_components["bm25"]` set). + +(f) `test_graph_expand_merge_omits_bm25_when_excluded`: with `rank_config=BASELINE_2LIST_CONFIG`, assert `_bm25_candidate_rows` is never called and the fused result matches today's 2-list behavior. + +(g) Integration: `test_run_search_bm25_degrades_silently_when_fts_missing` — run `run_search` against a fixture index whose LadybugDB graph has NO `sym_fts` index (or monkeypatch `_ensure_fts_loaded` → False); assert it returns results with no exception, no row has a `bm25` component, and ranking matches the pre-change 2-list vector path (compare against a baseline snapshot or a `BASELINE_2LIST_CONFIG` run — they must be equal). + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/search/test_search_lancedb.py -k "bm25_candidate or graph_expand_merge or run_search_bm25" -v` +Expected: FAIL — `_bm25_candidate_rows` undefined; `query` param not threaded. + +- [ ] **Step 3: Implement** + +Add `_bm25_candidate_rows` per the Produces contract above. Thread `query` into `_graph_expand_merge` and forward from `run_search`. Modify `_graph_expand_merge` to conditionally fetch + fuse the BM25 list based on `rank_config.lists`. If `_try_fts_candidates` is not importable from `search_lexical` due to its leading underscore, add a thin non-underscore alias in `search_lexical.py` (e.g. `fetch_fts_candidates = _try_fts_candidates`) without changing `_try_fts_candidates` behavior, and import the alias. Apply `_apply_chunk_hints` and `_refine_java_start_lines` to BM25 rows before returning (consistency with graph_rows handling at lines 701-702). + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/search/ -k "lexical or search or hybrid or rrf or bm25" -v` +Expected: PASS — all new tests green, no regressions in the existing search subset (104+ passed baseline from worktree setup). + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/search/search_lancedb.py src/java_codebase_rag/search/search_lexical.py tests/search/test_search_lancedb.py` +Run: `git commit -m "feat(search): fuse LadybugDB BM25 as third RRF list on vector path"` + +--- + +## Task 5: Eval metrics module (recall@k, precision@k, MRR) + +**Files:** +- Create: `src/java_codebase_rag/eval/__init__.py`, `src/java_codebase_rag/eval/metrics.py` +- Test: `tests/eval/__init__.py`, `tests/eval/test_metrics.py` + +**Interfaces:** +- Produces (`eval/metrics.py`, dep-free, pure functions): + - `def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:` — fraction of `relevant` appearing in `retrieved[:k]`. Returns `0.0` if `relevant` is empty. Value in `[0.0, 1.0]`. + - `def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:` — `|retrieved[:k] ∩ relevant| / k`. Returns `0.0` if `k == 0`. Value in `[0.0, 1.0]`. + - `def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:` — `1.0 / rank` of the first `retrieved` item that is in `relevant` (1-indexed); `0.0` if none. + - `def mean(values: list[float]) -> float:` — arithmetic mean, `0.0` for empty. + - `def aggregate(per_query: list[dict]) -> dict[str, float]:` — given a list of per-query dicts each containing keys `recall@1, recall@5, recall@10, recall@20, precision@5, mrr`, return the mean of each across all queries, keyed identically (e.g. `{"recall@10": 0.42, "mrr": 0.55, ...}`). +- Consumes: nothing. + +- [ ] **Step 1: Write failing tests** + +In `tests/eval/test_metrics.py`, hand-computed cases: +- `test_recall_at_k`: `retrieved=["a","b","c"], relevant={"b","d"}, k=3` → `0.5` (b found, d not); `k=1` → `0.0`; `relevant=set()` → `0.0`; `k=10` (longer than retrieved) → `0.5`. +- `test_precision_at_k`: `retrieved=["a","b","c"], relevant={"b"}, k=2` → `0.5`; `k=3` → `0.333...`; `k=0` → `0.0`. +- `test_reciprocal_rank`: `retrieved=["a","b","c"], relevant={"b"}` → `0.5`; `relevant={"z"}` → `0.0`; first-position match `relevant={"a"}` → `1.0`. +- `test_mean`: `[1.0, 0.0, 0.5]` → `0.5`; `[]` → `0.0`. +- `test_aggregate`: two per-query dicts `{"recall@10":1.0,"mrr":1.0, ...}` and `{"recall@10":0.0,"mrr":0.0, ...}` aggregate to `recall@10==0.5, mrr==0.5`. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/eval/test_metrics.py -v` +Expected: FAIL — module/import not found. + +- [ ] **Step 3: Implement** + +Create `eval/__init__.py` (empty) and `eval/metrics.py` with the five functions per contract. Pure stdlib only. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/eval/test_metrics.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/eval/__init__.py src/java_codebase_rag/eval/metrics.py tests/eval/__init__.py tests/eval/test_metrics.py` +Run: `git commit -m "feat(eval): IR metrics — recall@k, precision@k, reciprocal rank, aggregate"` + +--- + +## Task 6: Eval ground-truth — Tier-A generator + Tier-B loader + +**Files:** +- Create: `src/java_codebase_rag/eval/ground_truth.py` +- Test: `tests/eval/test_ground_truth.py` + +**Interfaces:** +- Produces (`eval/ground_truth.py`): + - `@dataclass(frozen=True) class LabeledQuery:` fields `query: str`, `relevant: frozenset[str]` (set of relevant Symbol FQNs), `tier: str` ("A" or "B"). + - `def build_tier_a(symbols: Iterable[SymbolLike]) -> list[LabeledQuery]:` — for each symbol produce queries from its simple name via `search_scoring._split_identifier`: one camelCase-joined identifier form and one space-joined lowercase token form (e.g. for `DistributionChunkService` produce `"DistributionChunkService"` and `"distribution chunk service"`). `relevant` = `frozenset({symbol.fqn})`. Skip symbols whose simple name splits to fewer than 2 tokens or is shorter than 3 chars (noise). `tier="A"`. Deterministic: output sorted by `(query, fqn)`. + - `def load_tier_b(path: str | Path) -> list[LabeledQuery]:` — parse a YAML (`.yaml`/`.yml`) or JSON (`.json`) file whose schema is a list of `{query: str, relevant: [str, ...]}` objects; return `LabeledQuery(query=..., relevant=frozenset(...), tier="B")`. Raise `FileNotFoundError` if missing (the runner treats absence as "Tier-B disabled", so the runner checks existence before calling). `SymbolLike` is a small structural type (duck-typed) exposing `.fqn: str` and `.name: str` (the simple name). +- Consumes: `search_scoring._split_identifier` (dep-free). YAML via `yaml.safe_load` if PyYAML is already a dependency (check `pyproject.toml`); if not, support JSON only and document that Tier-B YAML requires the existing YAML dep or add it. + +- [ ] **Step 1: Write failing tests** + +In `tests/eval/test_ground_truth.py`: +- `test_build_tier_a_deterministic`: feed a fixed list of `SymbolLike` (use simple objects/namedtuples with `.fqn`/`.name`) including `name="DistributionChunkService"`; assert the output contains `LabeledQuery("DistributionChunkService", frozenset({fqn}), "A")` and `LabeledQuery("distribution chunk service", frozenset({fqn}), "A")`; assert running twice yields identical lists; assert output is sorted by `(query, fqn)`. +- `test_build_tier_a_skips_noise`: a symbol with `name="A"` (1 char) and one with `name="Do"` (single token after split) produce no queries. +- `test_load_tier_b_yaml` (if YAML available) / `test_load_tier_b_json`: write a temp file with two entries; assert parsed into two `LabeledQuery` with `tier="B"` and correct `relevant` frozensets. Assert `load_tier_b` on a non-existent path raises `FileNotFoundError`. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/eval/test_ground_truth.py -v` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +Create `eval/ground_truth.py` per contract. Reuse `search_scoring._split_identifier` for query derivation so tokenization parity with the FTS index holds (the index is built from the same splitter). Check `pyproject.toml` for PyYAML before using `yaml`. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/eval/test_ground_truth.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/eval/ground_truth.py tests/eval/test_ground_truth.py` +Run: `git commit -m "feat(eval): Tier-A auto ground-truth generator + Tier-B loader"` + +--- + +## Task 7: Eval runner — index shopizer, sweep configs, emit results + +**Files:** +- Create: `src/java_codebase_rag/eval/runner.py` +- Test: `tests/eval/test_runner.py` + +**Interfaces:** +- Produces (`eval/runner.py`): + - `@dataclass(frozen=True) class EvalConfig:` fields: `corpus_dir: str` (default `~/jrag-bench/shopizer`), `index_dir: str` (temp dir; created by the runner), `results_dir: str` (default `~/jrag-bench/shopizer/results`), `tier_b_path: str | None = None`, `ks: tuple[int, ...] = (30, 60, 90, 120)`, `top_k_metrics: tuple[int, ...] = (1, 5, 10, 20)`, `model_name: str = ` (read from the same default `run_search` consumers use — confirm against `search_lancedb.run_search`'s `model_name` default). + - `def run_eval(cfg: EvalConfig) -> EvalReport:` orchestration: + 1. Build a fresh shopizer index into `cfg.index_dir` using the project's existing index entry point (the `jrag`/`java-codebase-rag` CLI index command — invoke programmatically via the function the CLI wraps, not a subprocess, so errors surface). Locate the indexer function by searching the `cli`/pipeline module for the index entry the console script calls. + 2. Open the index for query: obtain the LanceDB `uri`/`db` handles and the LadybugDB graph path the same way `run_search`'s callers do (mirror the MCP `search_v2` wiring in `mcp/mcp_v2.py`). Load a `SentenceTransformer` model once (reuse `search_lancedb`'s model-loading helper). + 3. Build ground truth: enumerate `Symbol` nodes from the graph (mirror `search_lexical`'s symbol enumeration), `build_tier_a(...)`; if `cfg.tier_b_path` exists, `load_tier_b(...)` and concatenate. + 4. For each config to evaluate — `BASELINE_2LIST_CONFIG` (k=60) and `DEFAULT_RANK_CONFIG`-shape at each `k` in `cfg.ks` (i.e. `RankConfig(lists=frozenset({"vector","graph","bm25"}), rrf_k=k)`) — for each `LabeledQuery`: call `search_lancedb.run_search(query=..., uri=uri, table_keys=["java"], limit=max(cfg.top_k_metrics), path_substring=None, model_name=cfg.model_name, model=, rank_config=)`, measure wall-clock per query, map returned hits' `primary_type_fqn` (or `fqn`) to retrieved FQN list, compute per-query recall@k/precision@k/mrr, aggregate. + 5. Return an `EvalReport` dataclass with per-config aggregated metrics + per-config p50 latency in ms. + 6. Persist: write `results//report.md` (a Markdown table: rows = configs, columns = recall@1, recall@5, recall@10, recall@20, precision@5, mrr, p50_latency_ms) and `report.json` (raw `EvalReport` as JSON). +- Consumes: Tasks 2/4 (`RankConfig`, `BASELINE_2LIST_CONFIG`, `DEFAULT_RANK_CONFIG`, `run_search` with `rank_config`), Task 5 metrics, Task 6 ground truth, the existing indexer + index-opening + model-loading helpers. + +- [ ] **Step 1: Write failing tests** + +In `tests/eval/test_runner.py` (do NOT depend on the real shopizer corpus — use a tiny in-repo Java fixture corpus the test suite already indexes, or the smallest existing fixture): +- `test_eval_report_shape`: run `run_eval` against a tiny fixture corpus (point `corpus_dir` at a small fixture under `tests/`, a fresh temp `index_dir`); assert the returned `EvalReport` has one entry per config evaluated (`1 + len(ks)` entries), each entry contains all metric keys, and the latency field is a non-negative float. +- `test_eval_report_persists_files`: after `run_eval`, assert `/report.md` and `/report.json` exist and the Markdown contains a header row with all metric column names and one data row per config. +- `test_eval_tier_b_optional`: with `tier_b_path=None`, the run completes using only Tier-A ground truth (no exception). With `tier_b_path` pointing to a temp file with one entry, the report still produces all configs (Tier-B queries are included but the test only asserts the run completes and shape is correct — not specific numbers). +- A smoke marker test `test_runner_smoke_exists` is NOT needed if the above cover it; numbers are research outputs, never asserted. + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/eval/test_runner.py -v` +Expected: FAIL — `runner` module not found. + +- [ ] **Step 3: Implement** + +Create `eval/runner.py` per contract. Reuse `run_search` directly (not the MCP layer) so `rank_config` is injectable. Keep the indexer/model/index-open wiring thin by delegating to existing helpers; if a helper doesn't exist for programmatic index-open, mirror the minimal calls `mcp_v2.search_v2` makes. Guard the whole run so a missing corpus raises a clear `FileNotFoundError` with the path (operators pointed at `~/jrag-bench/shopizer`). + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/eval/ -v` +Expected: PASS. (These tests build a real small index and may take longer than unit tests — that's acceptable; they are not part of the fast search subset.) + +- [ ] **Step 5: Commit** + +Run: `git add src/java_codebase_rag/eval/runner.py tests/eval/test_runner.py` +Run: `git commit -m "feat(eval): shopizer recall/precision@k + MRR runner with k-sweep"` + +--- + +## Task 8: Run eval on shopizer, tune k, set default, docs, full suite + +**Files:** +- Modify: `src/java_codebase_rag/search/search_scoring.py` (`DEFAULT_RANK_CONFIG.rrf_k` → eval winner) +- Modify: `docs/ARCHITECTURE.md` (HOW: vector read path now 3-list; rank-config injection point) +- Modify: `docs/DESIGN.md` (WHAT/WHY: BM25 promoted to first-class; reference the eval) +- Possibly `docs/CONFIGURATION.md` (only if any operator-visible surface changed — none should; if not, skip) +- Test: full suite + +**Interfaces:** none new. + +- [ ] **Step 1: Run the eval on the real corpus** + +Precondition: `~/jrag-bench/shopizer` exists with the shopizer source. If absent, STOP and report — the operator must clone shopizer there before this task can complete. +Run: `.venv/bin/python -m java_codebase_rag.eval.runner` (or the chosen invocation). Inspect `~/jrag-bench/shopizer/results//report.md`. + +- [ ] **Step 2: Pick the winning k** + +Per the spec win criterion: choose the `k` (among 30/60/90/120) maximizing **MRR** with **no regression on recall@10** vs the 2-list baseline; recall@1 is the tiebreak. If no 3-list config beats baseline on MRR, keep `DEFAULT_RANK_CONFIG` at 3-list @ k=60 anyway (lexical-first-class is the goal) and record the result honestly in the docs + the PR description. Open a follow-up issue if negative. + +- [ ] **Step 3: Set the default k** + +Update `DEFAULT_RANK_CONFIG` in `search_scoring.py` so `rrf_k` equals the winning k (if unchanged from 60, note that explicitly). Commit. + +- [ ] **Step 4: Update docs** + +Update `docs/ARCHITECTURE.md` read-path description to the 3-list RRF and mention `RankConfig` as the injection point. Update `docs/DESIGN.md` to reflect BM25 as first-class on the primary path, referencing the eval result (winning k + headline metric). Do NOT change operator docs unless an operator-facing surface changed (none did). + +- [ ] **Step 5: Run the full suite once** + +Erase stale indexes: `rm -rf tests/*/.java-codebase-rag tests/*/.java-codebase-rag.{yml,hosts}` +Run: `.venv/bin/python -m pytest tests/ -q` +Expected: PASS (0 failures). Report any failures before proceeding. + +- [ ] **Step 6: Commit** + +Run: `git add src/java_codebase_rag/search/search_scoring.py docs/ARCHITECTURE.md docs/DESIGN.md` +Run: `git commit -m "feat(search): promote BM25 to first-class hybrid RRF list (issue #431)"` + +--- + +## Self-Review Notes (resolved during authoring) + +- **Code leakage:** Steps describe behavior, data shapes, signatures, and exact expected test results — no method bodies or test code. +- **Self-containment:** Every task carries full Consumes/Produces contracts; Tasks 5–7 do not require reading the spec to implement. +- **Spec coverage:** 3-list fusion (T2/T4), `_HYBRID_SCORE_MAX` derivation (T1), `bm25=` explain (T3), filter-respect + FTS-unavailable degradation (T4g), rank-config injection (T2), eval harness with Tier-A/Tier-B + recall/precision/MRR + k-sweep + latency + win criterion (T5–T8) — all spec sections mapped. +- **Type consistency:** `RankConfig.lists`/`rrf_k`, `DEFAULT_RANK_CONFIG`, `BASELINE_2LIST_CONFIG`, `_bm25_candidate_rows`, `EvalConfig`, `EvalReport`, `LabeledQuery` — names used consistently across tasks. +- **Deferred (issues #434–#439):** absence unification, did-you-mean, no-vectors, airgapped FTS, NDCG, latency gating — all explicitly out of scope and filed. diff --git a/docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md b/docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md new file mode 100644 index 0000000..b14c712 --- /dev/null +++ b/docs/superpowers/specs/active/2026-07-12-hybrid-bm25-rrf-design.md @@ -0,0 +1,119 @@ +# Hybrid BM25 + Vector Ranking via RRF (fork B, issue #431) + +**Status:** Active — design approved 2026-07-12. +**Tracks:** [issue #431](https://github.com/HumanBean17/java-codebase-rag/issues/431). +**Depends on:** Fork A (PR #432, merged) — `Symbol.search_text` column + `sym_fts` LadybugDB FTS index at ontology v19. **Satisfied.** + +## Summary + +Promote LadybugDB Okapi BM25 from a fallback-only signal (macOS-Intel lexical path) to a first-class, always-on **third input** to the vector/hybrid read path's RRF (Reciprocal Rank Fusion). Exact-identifier lexical matches then anchor dense rankings on the primary path. + +``` +final_rank = RRF([vector_hits, graph_expand_hits, bm25_hits], k=tuned) +``` + +## Background & current state + +- The vector path (`search/search_lancedb.py`) fuses **2 lists** — `vector_rows` + `graph_rows` — in `_graph_expand_merge` via `_rrf_merge(k=60)`. +- `_rrf_merge` is already N-list generic; the only "2-ness" lives in (a) the caller and (b) the `_HYBRID_SCORE_MAX` constant in `search/search_scoring.py` (`2.0/61.0` term). +- Fork A shipped `Symbol.search_text` + `sym_fts` (ontology v19) and made BM25 the primary path in `search/search_lexical.py` via `_try_fts_candidates(g, query, filter, path_contains)`, returning `{rows, bm25_scores}`. The hand-rolled token-overlap scan (PR #403) survives only as the lexical fallback. +- `absence_diagnosis` is orthogonal (vocabulary q-gram + difflib did-you-mean); it does not consume BM25 today. +- No eval harness exists in-repo. + +## Goals + +- Add BM25 as an always-on third RRF list on the vector path. +- Make ranking config (`{list-set, k}`) runtime-injectable. +- Build a recall/precision @k + MRR eval harness over `~/jrag-bench/shopizer`. +- Tune RRF `k` for the 3-list fusion on that eval; ship the winner. +- Surface a `bm25` score component on the vector path under `explain=True`. + +## Non-goals + +- No change to `SearchHit` / `SearchOutput` / `NodeFilter` schemas (`score_components` is an open dict). +- No change to `absence_diagnosis` (deferred — see Follow-ups). +- No change to the `sym_fts` index or `search_text` column (fork A). +- No change to the LanceDB-internal `.text(fts_text)` single-table hybrid mode. +- No physical unification of the lexical-only and vector code paths (deferred). +- No `--no-vectors` distribution (deferred). +- No new operator-facing CLI flag for rank config (the injection is internal). + +## Architecture & data flow + +**Current:** +``` +search_v2 → search_lancedb.run_search + → vector query (LanceDB) → vector_rows + → graph-expand (LadybugGraph) → graph_rows + → _rrf_merge([vector_rows, graph_rows], k=60) +``` + +**Proposed:** +``` +search_v2 → search_lancedb.run_search + → vector query (LanceDB) → vector_rows + → graph-expand (LadybugGraph) → graph_rows + → BM25 FTS query (LadybugGraph sym_fts) → bm25_rows [always-on] + → _rrf_merge([vector_rows, graph_rows, bm25_rows], k=tuned) +``` + +- The BM25 fetch reuses `search_lexical._try_fts_candidates` — the same code the lexical backend uses. **No FTS-query, tokenizer, or filter-translation logic is duplicated.** `SYMBOL_FTS_INDEX` (search_scoring.py) remains the single source of truth for the index name. +- **Symbol→chunk resolution:** BM25 candidates are `Symbol` nodes; `_rrf_merge` dedups on `(filename, range_start, range_end)`. BM25 Symbols resolve to chunk(s) via the same expansion `graph_rows` already use. Orphaned Symbols (no chunk) are dropped. A Symbol mapping to multiple chunks expands to all (BM25 is a per-Signal, per-Symbol rank position). +- **FTS unavailable / empty query / over-restrictive filter:** the BM25 list is empty; 3-list `_rrf_merge` with one empty list degrades cleanly to current 2-list behavior. **No exception, no advisory.** This covers the airgapped case silently. +- **macOS-Intel lexical-only path:** behavior unchanged in this PR. Conceptually "hybrid with an empty vector list," but not physically unified (deferred). + +## Components + +| File | Change | +|---|---| +| `search/search_lancedb.py` | `_graph_expand_merge` (or a new sibling helper) calls `search_lexical._try_fts_candidates` → `bm25_rows`; resolves Symbol→chunk; passes `[vector_rows, graph_rows, bm25_rows]` to `_rrf_merge`. Reads `{list-set, k}` from an injected rank config (default = 3-list, tuned-k). Populates `row["_score_components"]["bm25"]`. | +| `search/search_scoring.py` | `_HYBRID_SCORE_MAX` RRF term derived from list count (`num_lists/(k+1)`) instead of hard-coded `2.0/61.0`. `explain_score_components` gains a `bm25=` token in the hybrid branch. | +| `search/search_lexical.py` | No behavioral change. (Possibly expose `_try_fts_candidates` for reuse; minimal.) | +| `eval/` (new package) | New top-level package + CLI runner. Indexes shopizer into a temp dir; builds Tier-A ground truth; runs 2-list baseline, 3-list candidate, and k-sweep; writes Markdown + JSON results. | + +**Score components (vector path, `explain=True`):** adds `bm25` (raw Okapi BM25 score; `0` if the hit did not come from the BM25 list), alongside existing `rrf_raw`, `hybrid_rrf`, `role_weight`, `symbol_bonus`, `import_penalty`. + +**Rank-config contract (internal):** `_graph_expand_merge` accepts a config selecting list-set ∈ `{{vector,graph}, {vector,graph,bm25}}` and integer `k`. Production default = `{vector,graph,bm25}` at the tuned k. The eval and tests inject alternatives. + +## Eval harness + +- **Location:** new `eval/` package (not under `tests/`); invoked via a runner entry point. Not a CI pass/fail gate. +- **Corpus:** `~/jrag-bench/shopizer`, indexed into a temp dir (fresh, never committed), mirroring `tests/conftest.py` hygiene. +- **Ground truth — Tier A (auto-generated, ships with harness):** for each indexed `Symbol`, queries derived from name/FQN tokens (e.g. `"DistributionChunkService"`, `"distribution chunk service"`), `relevant = {that symbol}`. Deterministic, free, no manual authoring. Tests the identifier regime BM25 should dominate. +- **Ground truth — Tier B (user-authored, optional):** `~/jrag-bench/shopizer/ground_truth.{json,yaml}` of natural-language intent queries → relevant symbols. Loaded if present; reported separately. +- **Metrics:** Recall@k (k ∈ {1,5,10,20}); Precision@k; **MRR (primary)**; recall@10 as no-regression guardrail; recall@1 as tiebreak. Binary relevance (graded/NDCG deferred). +- **Comparisons (single pass, one index):** 2-list @ k=60 (baseline); 3-list @ k ∈ {30,60,90,120}. Latency (p50 ms) measured per config. +- **Output:** Markdown table + raw JSON to `~/jrag-bench/shopizer/results//`. +- **Win criterion:** ship the 3-list config at the k maximizing **MRR** with no regression on recall@10 vs baseline. Negative results are reported honestly, not hidden; if no k beats baseline, ship anyway (lexical-first-class is the goal) and open a follow-up. + +## Testing + +- **Unit (CI):** `_rrf_merge` over 3/N lists (dedup, `num_lists/(k+1)` normalization, empty-list degradation); `_HYBRID_SCORE_MAX` derivation for list counts 1/2/3; `bm25` component population + `explain=True` gating; `explain_score_components` `bm25=` token; filter-respect (BM25 Symbol outside `NodeFilter` dropped); rank-config injection honoring injected `{list-set, k}`. +- **Integration (CI):** no-regression on existing `tests/search/` fixtures; FTS-unavailable on vector path → silent 2-list degradation (monkeypatch FTS load, assert no exception/advisory). +- **Eval-harness (CI, lightweight):** Tier-A generator deterministic on a fixture; recall@k & MRR math unit-tested against hand-computed cases; full-harness smoke run on a tiny fixture asserts *output produced* (not specific ranking numbers). +- **Not CI-gated:** the actual shopizer recall/MRR numbers (research artifacts, non-hermetic). +- **Baseline:** develop against the search subset; full suite once at end. + +## Error handling & latency + +- FTS unavailable / degenerate query / over-restrictive filter → empty BM25 list → silent degradation to 2-list. No new exceptions, exit codes, or advisories. +- `QUERY_FTS_INDEX` is a DB-side indexed query returning `top:=200` rows, comparable to the existing graph-expand hop. The eval measures the per-search latency delta (2-list vs 3-list); gating/caching deferred pending that measurement. + +## Deferred / follow-ups (GitHub issues to be opened) + +1. BM25-fed did-you-mean in `absence_diagnosis` (with its own MRR eval). +2. Physically unify the lexical-only and vector read paths ("hybrid with empty vector list"). +3. `--no-vectors` distribution mode. +4. Airgapped/bundled FTS install path (`INSTALL FTS` fetches from `extension.ladybugdb.com` on first use). +5. NDCG@k with graded relevance. +6. Latency gating (query-shape) or BM25-result caching, if the eval shows material cost. + +## References + +- `search/search_lancedb.py:_graph_expand_merge` (RRF caller), `:_rrf_merge` (N-list core). +- `search/search_scoring.py:_HYBRID_SCORE_MAX`, `:SYMBOL_FTS_INDEX`, `:explain_score_components`, `:build_fts_query`. +- `search/search_lexical.py:_try_fts_candidates`, `:_ensure_fts_loaded`, `:_CANDIDATE_LIMIT_CAP`. +- `graph/build_ast_graph.py:_compute_symbol_search_text`, `:_ensure_symbol_fts_index`. +- `ast/ast_java.py:ONTOLOGY_VERSION` (=19). +- `mcp/mcp_v2.py:SearchHit`, `:SearchOutput`, `:NodeFilter`, `:_row_to_search_hit`. +- `absence/absence_diagnosis.py:diagnose`. diff --git a/src/java_codebase_rag/eval/__init__.py b/src/java_codebase_rag/eval/__init__.py new file mode 100644 index 0000000..b8ae542 --- /dev/null +++ b/src/java_codebase_rag/eval/__init__.py @@ -0,0 +1 @@ +# Eval package for IR metrics diff --git a/src/java_codebase_rag/eval/ground_truth.py b/src/java_codebase_rag/eval/ground_truth.py new file mode 100644 index 0000000..3087ffd --- /dev/null +++ b/src/java_codebase_rag/eval/ground_truth.py @@ -0,0 +1,100 @@ +"""Eval ground-truth — Tier-A auto generator + Tier-B file loader. + +Tier-A derives labeled queries deterministically from indexed symbols (no +manual labeling). Tier-B loads hand-curated labeled queries from YAML/JSON. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable, Protocol + +import yaml + +from java_codebase_rag.search.search_scoring import _split_identifier + + +class SymbolLike(Protocol): + """Structural type for symbols — duck-typed .fqn / .name.""" + + fqn: str + name: str + + +@dataclass(frozen=True) +class LabeledQuery: + """A labeled retrieval query and its set of relevant Symbol FQNs.""" + + query: str + relevant: frozenset[str] + tier: str + + +def build_tier_a(symbols: Iterable[SymbolLike]) -> list[LabeledQuery]: + """Auto-generate labeled queries from each symbol's simple name. + + For each symbol two query strings are derived from its simple name: + 1. The original simple name verbatim (e.g. ``"DistributionChunkService"``) + — matches identifier-joined index text. + 2. A space-joined lowercase token form (e.g. ``"distribution chunk service"``) + produced via ``search_scoring._split_identifier`` so tokenization parity + with the FTS index holds. + + Symbols whose simple name splits to fewer than 2 tokens or is shorter than + 3 characters are skipped (noise). Output is deterministic, sorted by + ``(query, fqn)``; all entries carry ``tier="A"``. + """ + out: list[LabeledQuery] = [] + for sym in symbols: + name: str = sym.name + if len(name) < 3: + continue + tokens = _split_identifier(name) + if len(tokens) < 2: + continue + fqn: str = sym.fqn + relevant = frozenset({fqn}) + # 1. identifier-joined form = ORIGINAL simple name (preserve case). + out.append(LabeledQuery(name, relevant, "A")) + # 2. space-joined lowercase token form. + out.append(LabeledQuery(" ".join(tokens), relevant, "A")) + out.sort(key=lambda q: (q.query, next(iter(q.relevant)))) + return out + + +def load_tier_b(path: str | Path) -> list[LabeledQuery]: + """Load hand-curated Tier-B labeled queries from a YAML (``.yaml``/``.yml``) + or JSON (``.json``) file. + + Schema: a list of ``{query: str, relevant: [str, ...]}`` objects. + + Raises: + FileNotFoundError: if the path does not exist (the runner checks + existence before calling, treating absence as "Tier-B disabled"). + """ + p = Path(path) + if not p.exists(): + raise FileNotFoundError(f"Tier-B ground-truth file not found: {p}") + + suffix = p.suffix.lower() + raw = p.read_text() + if suffix in (".yaml", ".yml"): + data = yaml.safe_load(raw) + elif suffix == ".json": + data = json.loads(raw) + else: + # Fall back to YAML (superset of JSON) for unknown extensions. + data = yaml.safe_load(raw) + + out: list[LabeledQuery] = [] + for entry in data or []: + out.append( + LabeledQuery( + query=str(entry["query"]), + relevant=frozenset(entry.get("relevant", []) or []), + tier="B", + ) + ) + return out diff --git a/src/java_codebase_rag/eval/metrics.py b/src/java_codebase_rag/eval/metrics.py new file mode 100644 index 0000000..487ad9d --- /dev/null +++ b/src/java_codebase_rag/eval/metrics.py @@ -0,0 +1,107 @@ +"""IR evaluation metrics — pure functions, stdlib only. + +Functions take `retrieved: list[str]` (ordered list of retrieved FQN ids) +and `relevant: set[str]` (ground-truth relevant set). +""" + +from __future__ import annotations + + +def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float: + """Fraction of relevant documents appearing in retrieved[:k]. + + Args: + retrieved: Ordered list of retrieved document IDs. + relevant: Set of ground-truth relevant document IDs. + k: Cut-off rank (1-indexed). + + Returns: + Recall@k in [0.0, 1.0]. Returns 0.0 if relevant is empty. + """ + if not relevant: + return 0.0 + + retrieved_at_k = set(retrieved[:k]) + relevant_retrieved = retrieved_at_k.intersection(relevant) + + return len(relevant_retrieved) / len(relevant) + + +def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float: + """Precision at cut-off k: |retrieved[:k] ∩ relevant| / k. + + Args: + retrieved: Ordered list of retrieved document IDs. + relevant: Set of ground-truth relevant document IDs. + k: Cut-off rank (1-indexed). + + Returns: + Precision@k in [0.0, 1.0]. Returns 0.0 if k == 0. + """ + if k == 0: + return 0.0 + + retrieved_at_k = set(retrieved[:k]) + relevant_retrieved = retrieved_at_k.intersection(relevant) + + return len(relevant_retrieved) / k + + +def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float: + """Reciprocal rank: 1.0 / rank of first retrieved relevant document. + + Args: + retrieved: Ordered list of retrieved document IDs. + relevant: Set of ground-truth relevant document IDs. + + Returns: + Reciprocal rank in [0.0, 1.0]. Returns 0.0 if no relevant document + is retrieved. Ranks are 1-indexed (first hit = 1.0). + """ + for rank, doc_id in enumerate(retrieved, start=1): + if doc_id in relevant: + return 1.0 / rank + + return 0.0 + + +def mean(values: list[float]) -> float: + """Arithmetic mean of a list of floats. + + Args: + values: List of float values. + + Returns: + Arithmetic mean. Returns 0.0 for empty list. + """ + if not values: + return 0.0 + + return sum(values) / len(values) + + +def aggregate(per_query: list[dict]) -> dict[str, float]: + """Aggregate per-query metrics into means across all queries. + + Takes a list of per-query metric dicts and computes the mean of each + metric across all queries. All dicts must have the same keys. + + Args: + per_query: List of dicts, each containing metric names as keys + and float values (e.g., {"recall@10": 1.0, "mrr": 0.5, ...}). + + Returns: + Dict with same keys as input, containing mean of each metric + across all queries (e.g., {"recall@10": 0.42, "mrr": 0.55, ...}). + """ + if not per_query: + return {} + + metric_names = list(per_query[0].keys()) + result: dict[str, float] = {} + + for metric in metric_names: + values = [query_metrics[metric] for query_metrics in per_query] + result[metric] = mean(values) + + return result diff --git a/src/java_codebase_rag/eval/runner.py b/src/java_codebase_rag/eval/runner.py new file mode 100644 index 0000000..87a71c2 --- /dev/null +++ b/src/java_codebase_rag/eval/runner.py @@ -0,0 +1,556 @@ +"""Eval runner — index a corpus, sweep RankConfigs, emit recall/precision/MRR. + +This is the integration layer of the eval harness (Task 7 of the hybrid-BM25 +RRF plan). Unlike ``eval/metrics.py`` and ``eval/ground_truth.py`` (pure +stdlib), the runner MAY import the full vector stack (torch / lancedb / +sentence_transformers) and invokes the operator CLI to build a real index. + +Pipeline (``run_eval``): + +1. Build a fresh index into ``cfg.index_dir`` via the operator CLI + (``java-codebase-rag init``) as a **subprocess** — the stable operator + surface. Reaching into cocoindex/pipeline internals is fragile (process env, + progress renderers), so the subprocess wins on robustness. A non-zero exit + surfaces stdout/stderr in the raised ``RuntimeError``. +2. Open the index for query: set ``JAVA_CODEBASE_RAG_INDEX_DIR`` so + ``resolve_ladybug_path`` + ``run_search``'s URI resolve to our temp index; + load ``SentenceTransformer`` once and reuse. +3. Enumerate ``Symbol`` nodes from the LadybugDB graph (mirrors + ``search_lexical``) and build Tier-A ground truth; optionally concat Tier-B. +4. For each ``RankConfig`` (``BASELINE_2LIST_CONFIG`` + a 3-list config per + swept ``k``), run ``run_search`` per query, map rows to ``primary_type_fqn`` + and compute per-query recall@k / precision@k / MRR via ``eval.metrics``. +5. Aggregate, persist Markdown + JSON under + ``//report.{md,json}`` (timestamped subdir so + successive sweeps don't clobber each other). + +Granularity note (metric mapping) +--------------------------------- +Tier-A ``build_tier_a`` sets ``relevant = {symbol.fqn}`` where the fqn may be a +MEMBER fqn (``com.x.A#processClientMessage()``). ``run_search`` rows carry +``primary_type_fqn`` = the enclosing TYPE fqn (``com.x.A``, no ``#``). To make +both sides type-level, the runner normalizes member→type via +``search_lexical._enclosing_type_fqn`` BEFORE scoring. This keeps Task 6's +``build_tier_a`` untouched. +""" + +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +import tempfile +import time +import traceback +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +from java_codebase_rag.eval.ground_truth import LabeledQuery, build_tier_a, load_tier_b +from java_codebase_rag.eval import metrics as M +from java_codebase_rag.graph.ladybug_queries import LadybugGraph, resolve_ladybug_path +from java_codebase_rag.search.index_common import SBERT_MODEL +from java_codebase_rag.search.search_lexical import _enclosing_type_fqn, _SYMBOL_RETURN +from java_codebase_rag.search.search_lancedb import run_search +from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + RankConfig, +) + +# Markdown table columns — order-stable, mirrored by the test suite. +METRIC_COLUMNS: tuple[str, ...] = ( + "recall@1", + "recall@5", + "recall@10", + "recall@20", + "precision@5", + "mrr", + "p50_latency_ms", +) + + +# Type-level Symbol kinds emitted by the graph (ast_java._TYPE_KINDS + +# build_ast_graph._TYPE_KINDS). Confirmed literals — note it's "annotation", +# NOT "annotation_type". Tier-A recall is measured at type level (chunks carry +# primary_type_fqn), so member-symbol queries are redundant; defaulting +# symbol_kinds to these type kinds both bounds fan-out and is semantically right. +_TYPE_SYMBOL_KINDS: tuple[str, ...] = ( + "class", + "interface", + "enum", + "annotation", + "record", +) + + +@dataclass(frozen=True) +class EvalConfig: + """Configuration for a single eval run. + + ``index_dir`` may be empty — the runner creates a temp dir in that case + (and writes it back into the returned ``EvalReport``). + """ + + corpus_dir: str = field( + default_factory=lambda: str(Path.home() / "jrag-bench" / "shopizer") + ) + index_dir: str = "" + results_dir: str = field( + default_factory=lambda: str(Path.home() / "jrag-bench" / "shopizer" / "results") + ) + tier_b_path: str | None = None + ks: tuple[int, ...] = (30, 60, 90, 120) + top_k_metrics: tuple[int, ...] = (1, 5, 10, 20) + model_name: str = SBERT_MODEL + device: str | None = field( + default_factory=lambda: os.environ.get("SBERT_DEVICE") or None + ) + # When set, _enumerate_symbols filters Symbol nodes to these kinds (type + # level by default). None = all kinds (escape hatch). Bounds Tier-A fan-out + # AND matches the type-level recall granularity. + symbol_kinds: tuple[str, ...] | None = _TYPE_SYMBOL_KINDS + # Deterministic cap on Tier-A LabeledQuery items produced (after the kind + # filter). Tier-B queries are NOT capped (operator-curated). See run_eval. + max_queries: int = 400 + + def __post_init__(self) -> None: + if self.max_queries < 1: + raise ValueError( + f"EvalConfig.max_queries must be >= 1, got {self.max_queries}" + ) + + +@dataclass(frozen=True) +class ConfigMetrics: + """Aggregated metrics for one RankConfig under test.""" + + config_name: str + metrics: dict[str, float] + num_queries: int + rrf_k: int + lists: tuple[str, ...] + + +@dataclass(frozen=True) +class EvalReport: + """Result of a full eval sweep.""" + + configs: list[ConfigMetrics] + timestamp: str + num_queries: int + corpus_dir: str + index_dir: str + # Tier-A queries available BEFORE the max_queries cap was applied (Tier-B + # excluded from this count). Recorded so capped runs stay interpretable. + num_queries_available: int = 0 + # Absolute path to the timestamped output dir holding report.md / report.json. + out_dir: str = "" + + def to_json(self) -> str: + return json.dumps(asdict(self), indent=2, sort_keys=True) + + +# --------------------------------------------------------------------------- +# Index build (subprocess over the operator CLI — the stable surface) +# --------------------------------------------------------------------------- + + +def _build_index_subprocess(*, corpus_dir: str, index_dir: str) -> None: + """Build a fresh index via ``java-codebase-rag init``. + + Raises FileNotFoundError if the corpus is missing, RuntimeError on a + non-zero CLI exit (surfacing clipped stdout/stderr). + """ + if not Path(corpus_dir).is_dir(): + raise FileNotFoundError( + f"eval corpus_dir does not exist: {corpus_dir} " + "(point EvalConfig.corpus_dir at a checked-out Java repo)" + ) + + Path(index_dir).mkdir(parents=True, exist_ok=True) + env = { + **os.environ, + "JAVA_CODEBASE_RAG_INDEX_DIR": str(Path(index_dir).resolve()), + "JAVA_CODEBASE_RAG_SOURCE_ROOT": str(Path(corpus_dir).resolve()), + } + cmd = [ + sys.executable, + "-m", + "java_codebase_rag.cli", + "init", + "--source-root", + str(Path(corpus_dir).resolve()), + "--index-dir", + str(Path(index_dir).resolve()), + "--quiet", + ] + proc = subprocess.run( + cmd, + capture_output=True, + text=True, + env=env, + timeout=int(os.environ.get("JAVA_CODEBASE_RAG_EVAL_INDEX_TIMEOUT", "1800")), + ) + if proc.returncode != 0: + raise RuntimeError( + f"java-codebase-rag init exited {proc.returncode} for corpus {corpus_dir}.\n" + f"--- stdout (clipped 8000) ---\n{proc.stdout[-8000:]}\n" + f"--- stderr (clipped 8000) ---\n{proc.stderr[-8000:]}" + ) + + +# --------------------------------------------------------------------------- +# Symbol enumeration (mirror search_lexical) +# --------------------------------------------------------------------------- + + +class _SymbolRow: + """Attribute-view over a Cypher row dict (build_tier_a ducks on .fqn/.name).""" + + __slots__ = ("fqn", "name", "kind") + + def __init__(self, row: dict[str, Any]) -> None: + self.fqn = str(row.get("fqn") or "") + self.name = str(row.get("name") or "") + self.kind = str(row.get("kind") or "") + + +def _enumerate_symbols( + graph: LadybugGraph, *, symbol_kinds: tuple[str, ...] | None +) -> list[_SymbolRow]: + """Return Symbol rows as duck-typed objects (.fqn / .name) for build_tier_a. + + ``LadybugGraph._rows`` returns dicts; ``build_tier_a``'s ``SymbolLike`` + protocol ducks on attributes, so we wrap each row. + + When ``symbol_kinds`` is not None, filters to those kinds via a + parameterized ``WHERE s.kind IN $kinds`` predicate — type-level only by + default, which both bounds fan-out and matches the type-level recall + granularity (chunks carry ``primary_type_fqn``). + """ + if symbol_kinds is None: + cypher = f"MATCH (s:Symbol) RETURN {_SYMBOL_RETURN}" + rows = graph._rows(cypher) # noqa: SLF001 + else: + cypher = f"MATCH (s:Symbol) WHERE s.kind IN $kinds RETURN {_SYMBOL_RETURN}" + rows = graph._rows(cypher, {"kinds": list(symbol_kinds)}) # noqa: SLF001 + return [_SymbolRow(r) for r in rows] + + +# --------------------------------------------------------------------------- +# Metric computation +# --------------------------------------------------------------------------- + + +def _retrieved_fqns(rows: list[dict]) -> list[str]: + """Map search rows to ordered, deduped type-level FQNs.""" + out: list[str] = [] + seen: set[str] = set() + for r in rows: + fqn = r.get("primary_type_fqn") or r.get("fqn") or "" + if not fqn: + continue + if fqn in seen: + continue + seen.add(fqn) + out.append(fqn) + return out + + +def _relevant_type_fqns(labeled: LabeledQuery) -> set[str]: + """Normalize member FQNs to enclosing-type FQNs so both sides are type-level.""" + return {_enclosing_type_fqn(fqn) for fqn in labeled.relevant if fqn} + + +def _p50(values: list[float]) -> float: + if not values: + return 0.0 + s = sorted(values) + mid = len(s) // 2 + if len(s) % 2: + return float(s[mid]) + return float((s[mid - 1] + s[mid]) / 2.0) + + +def _eval_one_config( + *, + config_name: str, + rank_config: RankConfig, + queries: list[LabeledQuery], + uri: str, + ladybug_path: str, + model, + model_name: str, + device: str | None, + top_k_metrics: tuple[int, ...], + limit: int, +) -> ConfigMetrics: + """Run one RankConfig over all queries; return aggregated ConfigMetrics.""" + per_query: list[dict[str, float]] = [] + latencies_ms: list[float] = [] + + for q in queries: + t0 = time.perf_counter() + rows = run_search( + q.query, + uri=uri, + table_keys=["java"], + limit=limit, + offset=0, + path_substring=None, + model_name=model_name, + device=device, + model=model, + rank_config=rank_config, + graph_expand=True, + expand_depth=1, + ladybug_path=ladybug_path, + dedup_by_fqn=True, + ) + elapsed_ms = (time.perf_counter() - t0) * 1000.0 + latencies_ms.append(elapsed_ms) + + retrieved = _retrieved_fqns(rows) + relevant = _relevant_type_fqns(q) + if not relevant: + # No relevant set (e.g. Tier-B with empty relevant) — skip from scoring. + continue + + qm: dict[str, float] = {} + for k in top_k_metrics: + qm[f"recall@{k}"] = M.recall_at_k(retrieved, relevant, k) + qm["precision@5"] = M.precision_at_k(retrieved, relevant, 5) + qm["mrr"] = M.reciprocal_rank(retrieved, relevant) + per_query.append(qm) + + agg = M.aggregate(per_query) + metrics: dict[str, float] = {} + for k in top_k_metrics: + metrics[f"recall@{k}"] = float(agg.get(f"recall@{k}", 0.0)) + metrics["precision@5"] = float(agg.get("precision@5", 0.0)) + metrics["mrr"] = float(agg.get("mrr", 0.0)) + metrics["p50_latency_ms"] = _p50(latencies_ms) + + return ConfigMetrics( + config_name=config_name, + metrics=metrics, + num_queries=len(per_query), + rrf_k=rank_config.rrf_k, + lists=tuple(sorted(rank_config.lists)), + ) + + +# --------------------------------------------------------------------------- +# Markdown rendering +# --------------------------------------------------------------------------- + + +def _render_markdown(report: EvalReport) -> str: + lines: list[str] = [] + lines.append(f"# Eval Report — {report.timestamp}") + lines.append("") + lines.append( + f"Corpus: `{report.corpus_dir}` | Index: `{report.index_dir}` " + f"| Queries scored: {report.num_queries}" + ) + if report.num_queries_available: + lines.append( + f"Tier-A available (pre-cap): {report.num_queries_available} | " + f"Total scored (Tier-A kept + Tier-B): {report.num_queries}" + ) + lines.append("") + header = "| config | " + " | ".join(METRIC_COLUMNS) + " |" + sep = "| --- " * (len(METRIC_COLUMNS) + 1) + "|" + lines.append(header) + lines.append(sep) + for entry in report.configs: + cells = [entry.config_name] + for col in METRIC_COLUMNS: + v = entry.metrics.get(col, 0.0) + if col == "p50_latency_ms": + cells.append(f"{v:.1f}") + else: + cells.append(f"{v:.4f}") + lines.append("| " + " | ".join(cells) + " |") + lines.append("") + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Orchestration +# --------------------------------------------------------------------------- + + +def run_eval(cfg: EvalConfig) -> EvalReport: + """Build a fresh index, sweep RankConfigs, return an EvalReport. + + See module docstring for the pipeline and the metric-granularity note. + """ + # Late import — torch/lancedb only needed for the run, not for module import. + from sentence_transformers import SentenceTransformer + + # Resolve index_dir (temp dir if blank). + if not cfg.index_dir: + index_dir = tempfile.mkdtemp(prefix="jrag-eval-") + else: + index_dir = cfg.index_dir + Path(index_dir).mkdir(parents=True, exist_ok=True) + + # 1. Build the index (subprocess). + _build_index_subprocess(corpus_dir=cfg.corpus_dir, index_dir=index_dir) + + # 2. Wire the process env so resolve_ladybug_path + run_search's URI hit our index. + os.environ["JAVA_CODEBASE_RAG_INDEX_DIR"] = str(Path(index_dir).resolve()) + os.environ.setdefault( + "JAVA_CODEBASE_RAG_SOURCE_ROOT", str(Path(cfg.corpus_dir).resolve()) + ) + uri = str(Path(index_dir).resolve()) + ladybug_path = resolve_ladybug_path(None) + + # Load the model ONCE — pass into every run_search call. + model = SentenceTransformer( + cfg.model_name, device=cfg.device, trust_remote_code=True + ) + + # 3. Open graph + build ground truth. + # Reset the LadybugGraph singleton — a prior test/process may have cached + # a different path. We force-bind to our index's graph. + LadybugGraph.reset_for_path(None) + graph = LadybugGraph.get(ladybug_path) + + symbols = _enumerate_symbols(graph, symbol_kinds=cfg.symbol_kinds) + tier_a = list(build_tier_a(symbols)) + num_queries_available = len(tier_a) + # Deterministic cap on Tier-A: sort by (query, fqn) and keep the first + # max_queries. fqn lives as the sole element of each LabeledQuery.relevant + # (build_tier_a sets relevant = {symbol.fqn}). Tier-B is operator-curated + # and NOT capped. + tier_a_sorted = sorted(tier_a, key=lambda q: (q.query, next(iter(q.relevant)))) + tier_a_kept = tier_a_sorted[: cfg.max_queries] + queries = list(tier_a_kept) + # Tier-B is optional: a configured but missing path means "Tier-B disabled" + # (matches load_tier_b's docstring). load_tier_b itself still raises + # FileNotFoundError when called directly on a genuinely missing path. + if cfg.tier_b_path and Path(cfg.tier_b_path).exists(): + queries.extend(load_tier_b(cfg.tier_b_path)) + + # 4. Enumerate configs: BASELINE_2LIST_CONFIG (k=60) + 3-list at each swept k. + limit = max(cfg.top_k_metrics) + configs: list[tuple[str, RankConfig]] = [ + ("baseline_2list_k60", BASELINE_2LIST_CONFIG), + ] + for k in cfg.ks: + configs.append( + ( + f"hybrid_3list_k{k}", + RankConfig( + lists=frozenset({"vector", "graph", "bm25"}), + rrf_k=k, + ), + ) + ) + + # 5. Run + aggregate. + results: list[ConfigMetrics] = [] + for name, rc in configs: + results.append( + _eval_one_config( + config_name=name, + rank_config=rc, + queries=queries, + uri=uri, + ladybug_path=ladybug_path, + model=model, + model_name=cfg.model_name, + device=cfg.device, + top_k_metrics=cfg.top_k_metrics, + limit=limit, + ) + ) + + timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + # Namespace outputs under the timestamp so successive sweeps don't clobber. + out_dir = str(Path(cfg.results_dir) / timestamp) + report = EvalReport( + configs=results, + timestamp=timestamp, + num_queries=len(queries), + corpus_dir=cfg.corpus_dir, + index_dir=index_dir, + num_queries_available=num_queries_available, + out_dir=out_dir, + ) + + # 6. Persist into //report.{md,json}. + _persist(report, cfg.results_dir) + return report + + +def _persist(report: EvalReport, results_dir: str) -> None: + # Write into a timestamped subdir so successive sweeps don't clobber. + out = Path(results_dir) / report.timestamp + out.mkdir(parents=True, exist_ok=True) + (out / "report.md").write_text(_render_markdown(report)) + (out / "report.json").write_text(report.to_json()) + + +def _build_eval_config_from_args(argv: list[str] | None = None) -> EvalConfig: + """Parse CLI args into an EvalConfig. Defaults mirror EvalConfig fields.""" + # Start from defaults so unspecified flags inherit EvalConfig's own defaults. + base = EvalConfig() + parser = argparse.ArgumentParser( + prog="python -m java_codebase_rag.eval.runner", + description="Index a corpus, sweep RankConfigs, emit recall/precision/MRR.", + ) + parser.add_argument( + "corpus_dir", + nargs="?", + default=base.corpus_dir, + help=f"Java repo to index (default: {base.corpus_dir})", + ) + parser.add_argument("--index-dir", default=base.index_dir, + help="Lance+Ladybug index dir (default: temp dir).") + parser.add_argument("--results-dir", default=base.results_dir, + help=f"Where to write report.{{md,json}} (default: {base.results_dir})") + parser.add_argument("--max-queries", type=int, default=base.max_queries, + help=f"Cap on Tier-A queries (default: {base.max_queries}).") + parser.add_argument( + "--ks", default=",".join(str(k) for k in base.ks), + help="Comma-separated RRF k constants to sweep (default: %(default)s).", + ) + parser.add_argument("--tier-b", default=base.tier_b_path, + help="Optional path to a Tier-B ground-truth file (missing ⇒ disabled).") + parser.add_argument("--device", default=base.device, + help="SBERT device (default: SBERT_DEVICE env or auto).") + args = parser.parse_args(argv) + + try: + ks = tuple(int(k.strip()) for k in args.ks.split(",") if k.strip()) + except ValueError: + parser.error(f"--ks must be comma-separated ints, got {args.ks!r}") + if not ks: + parser.error("--ks must contain at least one value") + + return EvalConfig( + corpus_dir=args.corpus_dir, + index_dir=args.index_dir, + results_dir=args.results_dir, + tier_b_path=args.tier_b, + ks=ks, + max_queries=args.max_queries, + device=args.device, + ) + + +if __name__ == "__main__": + cfg = _build_eval_config_from_args() + try: + report = run_eval(cfg) + except Exception: + traceback.print_exc() + sys.exit(1) + print(report.out_dir) + sys.exit(0) diff --git a/src/java_codebase_rag/search/search_lancedb.py b/src/java_codebase_rag/search/search_lancedb.py index a4958b6..2003f67 100644 --- a/src/java_codebase_rag/search/search_lancedb.py +++ b/src/java_codebase_rag/search/search_lancedb.py @@ -20,6 +20,7 @@ from java_codebase_rag.ast.chunk_heuristics import analyze_chunk, looks_like_code_identifier from java_codebase_rag.search.index_common import SBERT_MODEL +from java_codebase_rag.search import search_lexical from java_codebase_rag.config import maybe_expand_embedding_model_path, resolved_sbert_model_for_process_env # Scoring & dedup primitives live in `search_scoring` (dependency-free — no @@ -27,7 +28,11 @@ # graph-only (macOS Intel) installs where this module is unimportable. Re-exported # here for backward compatibility (`from search_lancedb import _clamp01`, etc.). from java_codebase_rag.search.search_scoring import ( # noqa: F401 + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, DEDUP_OVERFETCH, + RankConfig, + build_fts_query, _ACTION_VERB_BONUS, _ACTION_VERB_PREFIXES, _HYBRID_SCORE_MAX, @@ -632,9 +637,150 @@ def _attach_neighbor_context( _debug_ctx(f"attached context to {attached}/{len(java_rows)} java rows") +def _bm25_candidate_rows( + *, + g: object, + query: str, + uri: str, + db: object, + extra_predicates: list[str], + columns: set[str], + limit: int = 100, +) -> list[dict]: + """Fetch BM25-ranked Symbol candidates from the FTS index and resolve them to + chunk rows in BM25 rank order. Returns ``[]`` on any failure (silent degradation). + + Pipeline: + 1. ``search_lexical.fetch_fts_candidates(g, query)`` → BM25-ranked Symbols + + a ``{symbol_node_id: bm25_score}`` map. ``None`` / empty → return ``[]``. + 2. Map each Symbol fqn to its enclosing TYPE fqn (``primary_type_fqn`` has no + ``#``; a member ``Type#method`` maps to ``Type``). Dedupe by type fqn, + keeping the MAX BM25 score among same-type symbols. + 3. Order type fqns by BM25 desc (fqn asc tiebreak — deterministic). + 4. Fetch chunk rows from LanceDB with a FILTER-ONLY query (no vector ranking, + so BM25 order is preserved). Predicates = caller's ``extra_predicates`` + + the ``primary_type_fqn IN (...)`` built from the ordered types — preserving + filter parity with the vector path. + 5. Group fetched chunks by ``primary_type_fqn``; emit in BM25 rank order, + each chunk carrying ``_score_components["bm25"]``. + 6. Apply ``_apply_chunk_hints`` + ``_refine_java_start_lines`` for consistency + with graph_rows handling. + + Any exception (FTS or LanceDB) → ``_debug_ctx`` log + return ``[]`` (silent + degradation; the vector path is unaffected). + """ + # 1. BM25 candidate fetch via the FTS index. + # Pre-split the query with the same tokenizer the ``sym_fts`` index uses + # (``search_text`` stores ``_split_identifier`` tokens). LadybugDB FTS's own + # tokenizer does NOT split camelCase, so a raw ``DistributionChunkService`` + # would match nothing — ``build_fts_query`` mirrors what the lexical backend + # does at search_lexical.py (run_lexical_search), keeping index/query token + # spaces aligned. An empty split (degenerate / stopword-only query) → no FTS + # candidates → degrade silently to the vector path. + fts_query = build_fts_query(query) + if not fts_query or not fts_query.strip(): + return [] + try: + fts = search_lexical.fetch_fts_candidates(g, fts_query, filter=None, path_contains=None) + except Exception as exc: # noqa: BLE001 — silent degradation + _debug_ctx(f"bm25 FTS fetch raised: {exc!r}") + return [] + if not fts or not fts.get("rows"): + return [] + sym_rows = fts["rows"] + scores = fts.get("scores") or {} + + # 2. Map symbol fqns → enclosing type fqns; keep MAX bm25 per type. + type_fqn_to_bm25: dict[str, float] = {} + for r in sym_rows: + fqn = r.get("fqn") + if not fqn: + continue + type_fqn = search_lexical.enclosing_type_fqn(str(fqn)) + if not type_fqn: + continue + score = float(scores.get(r.get("id"), 0.0)) + prev = type_fqn_to_bm25.get(type_fqn) + if prev is None or score > prev: + type_fqn_to_bm25[type_fqn] = score + if not type_fqn_to_bm25: + return [] + + # 3. Deterministic ordering: BM25 desc, fqn asc. + ordered_types = sorted( + type_fqn_to_bm25.keys(), + key=lambda f: (-type_fqn_to_bm25[f], f), + ) + + # 4. Filter-only chunk fetch (NO vector ranking → BM25 order preserved). The + # ``primary_type_fqn IN (...)`` predicate must be buildable; if the index is so + # old that the column is absent, we can't restrict the fetch and degrade to []. + if "primary_type_fqn" not in columns: + _debug_ctx("bm25 fetch skipped: primary_type_fqn column absent from schema") + return [] + preds = list(extra_predicates) + _build_extra_predicates( + columns=columns, + role=None, module=None, microservice=None, + package_prefix=None, fqn_in=ordered_types, + ) + combined_pred = _combine_predicates(preds) + base_cols = ["filename", "text", "start", "end"] + for col in ("range_start", "range_end"): + if col in columns: + base_cols.append(col) + java_extra = [c for c in JAVA_ENRICHED_COLUMNS if c in columns] + select_cols = [*base_cols, "language", *java_extra] + + try: + tbl = db.open_table(TABLES["java"]) + # LanceDB 0.34 filter-only path: search() with no vector arg issues a + # non-vector scan; .where/.select/.limit/.to_list returns rows in table + # order without re-ranking by similarity. (tbl.query() is NOT available in + # 0.34; to_lance().scanner() requires pylance, which isn't installed on the + # PEP 508 graph-only profile — search() with no vector is the supported API.) + q = tbl.search().select(select_cols).limit( + max(limit, len(ordered_types) * 4) + ) + if combined_pred: + q = q.where(combined_pred, prefilter=True) + with _silence_lance_autoproj_warnings(): + fetched = q.to_list() + except Exception as exc: # noqa: BLE001 — silent degradation + _debug_ctx(f"bm25 chunk fetch failed: {exc!r}") + return [] + + # 5. Group by primary_type_fqn; emit in BM25 rank order. + by_type: dict[str, list[dict]] = {} + for ch in fetched: + tf = ch.get("primary_type_fqn") + if tf is None: + continue + by_type.setdefault(str(tf), []).append(ch) + + out: list[dict] = [] + for type_fqn in ordered_types: + chunks = by_type.get(type_fqn) + if not chunks: + continue # filtered out by extra_predicates / absent from index + bm25_val = round(float(type_fqn_to_bm25[type_fqn]), 4) + for ch in chunks: + ch["_kind"] = "java" + ch["_hybrid"] = False + ch.setdefault("_score_components", {})["bm25"] = bm25_val + ch["start"] = coerce_position_field(ch.get("start")) + ch["end"] = coerce_position_field(ch.get("end")) + out.append(ch) + + # 6. Consistency with graph_rows handling. + _apply_chunk_hints(out) + _refine_java_start_lines(out) + return out + + def _graph_expand_merge( vector_rows: list[dict], *, + query: str, query_vec: np.ndarray, db: object, uri: str, @@ -642,8 +788,24 @@ def _graph_expand_merge( extra_predicates: list[str], expand_depth: int, ladybug_path: str | None, + rank_config: RankConfig = DEFAULT_RANK_CONFIG, ) -> list[dict]: - """Expand vector top-k through the LadybugDB graph and fuse (RRF) with the original list.""" + """Expand vector top-k through the graph and/or fuse BM25, then RRF-merge. + + Which lists contribute is controlled by ``rank_config.lists``: + - ``"vector"`` — always present (the backbone; validated by RankConfig). + - ``"graph"`` — graph expand + fetch (skipped entirely when absent). + - ``"bm25"`` — LadybugDB FTS candidate fetch fused as a third list. + + Silent degradation: any failure in the graph or BM25 path drops just that list; + the vector list is never lost. Returns ``vector_rows`` unchanged when no + auxiliary list yields rows. + """ + want_graph = "graph" in rank_config.lists + want_bm25 = "bm25" in rank_config.lists + if not want_graph and not want_bm25: + return vector_rows + # Lazy import so the module works without ladybug installed when graph_expand=False. try: from java_codebase_rag.graph.ladybug_queries import LadybugGraph @@ -653,67 +815,97 @@ def _graph_expand_merge( if not LadybugGraph.exists(ladybug_path): return vector_rows - seed_fqns = sorted({r.get("primary_type_fqn") for r in vector_rows if r.get("primary_type_fqn")}) - if not seed_fqns: - return vector_rows + java_cols = _table_columns(uri, TABLES["java"], db) - try: - graph = LadybugGraph.get(ladybug_path) - structural = graph.expand_fqns(seed_fqns, depth=expand_depth) - method_pairs = graph.expand_methods( - seed_fqns, depth=expand_depth, exclude_external=True, - ) - expand_weight_by_fqn: dict[str, float] = {} - for f in structural: - if f: - expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), 1.0) - for f, conf in method_pairs: - if f: - expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), conf) - neighbor_fqns = list(dict.fromkeys( - list(structural) + [f for f, _ in method_pairs], - )) - except Exception: - return vector_rows + # --- graph list --- + graph_rows: list[dict] = [] + expand_weight_by_fqn: dict[str, float] = {} + if want_graph: + seed_fqns = sorted({r.get("primary_type_fqn") for r in vector_rows if r.get("primary_type_fqn")}) + neighbor_fqns: list[str] = [] + if seed_fqns: + try: + graph_obj = LadybugGraph.get(ladybug_path) + structural = graph_obj.expand_fqns(seed_fqns, depth=expand_depth) + method_pairs = graph_obj.expand_methods( + seed_fqns, depth=expand_depth, exclude_external=True, + ) + for f in structural: + if f: + expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), 1.0) + for f, conf in method_pairs: + if f: + expand_weight_by_fqn[f] = max(expand_weight_by_fqn.get(f, 0.0), conf) + neighbor_fqns = list(dict.fromkeys( + list(structural) + [f for f, _ in method_pairs], + )) + except Exception: + neighbor_fqns = [] + + novel = [fqn for fqn in neighbor_fqns if fqn and fqn not in set(seed_fqns)] + if novel: + extra = list(extra_predicates) + extra.extend(_build_extra_predicates( + columns=java_cols, + role=None, module=None, microservice=None, + package_prefix=None, fqn_in=novel, + )) + try: + graph_rows = _search_one_table( + TABLES["java"], + uri=uri, db=db, query_vec=query_vec, + limit=max(limit, 20), + path_predicate=None, kind="java", + hybrid=False, fts_text=None, + extra_predicates=extra, + ) + except Exception: + graph_rows = [] + _apply_chunk_hints(graph_rows) + _refine_java_start_lines(graph_rows) + graph_rows.sort(key=_vector_sort_key) + for r in graph_rows: + r["_graph_expanded"] = True + r["_graph_expand_weight"] = expand_weight_by_fqn.get( + r.get("primary_type_fqn"), 1.0, + ) + + # --- bm25 list --- + bm25_rows: list[dict] = [] + if want_bm25: + try: + graph_obj = LadybugGraph.get(ladybug_path) + except Exception: + graph_obj = None + if graph_obj is not None: + bm25_rows = _bm25_candidate_rows( + g=graph_obj, + query=query, + uri=uri, + db=db, + extra_predicates=extra_predicates, + columns=java_cols, + limit=limit, + ) - novel = [fqn for fqn in neighbor_fqns if fqn and fqn not in set(seed_fqns)] - if not novel: + # --- RRF fusion (only lists that yielded rows beyond vector) --- + lists: list[list[dict]] = [vector_rows] + row_weights: list[Callable[[dict], float] | None] = [None] + if want_graph and graph_rows: + lists.append(graph_rows) + row_weights.append(lambda row: float(row.get("_graph_expand_weight", 1.0))) + if want_bm25 and bm25_rows: + lists.append(bm25_rows) + row_weights.append(None) + + if len(lists) == 1: return vector_rows - extra = list(extra_predicates) - extra.extend(_build_extra_predicates( - columns=_table_columns(uri, TABLES["java"], db), - role=None, module=None, microservice=None, - package_prefix=None, fqn_in=novel, - )) - - try: - graph_rows = _search_one_table( - TABLES["java"], - uri=uri, db=db, query_vec=query_vec, - limit=max(limit, 20), - path_predicate=None, kind="java", - hybrid=False, fts_text=None, - extra_predicates=extra, - ) - except Exception: - return vector_rows - _apply_chunk_hints(graph_rows) - _refine_java_start_lines(graph_rows) - graph_rows.sort(key=_vector_sort_key) - for r in graph_rows: - r["_graph_expanded"] = True - r["_graph_expand_weight"] = expand_weight_by_fqn.get( - r.get("primary_type_fqn"), 1.0, - ) - fused = _rrf_merge( - [vector_rows, graph_rows], - row_weight_for_list_index=[ - None, - lambda row: float(row.get("_graph_expand_weight", 1.0)), - ], + return _rrf_merge( + lists, + k=rank_config.rrf_k, + row_weight_for_list_index=row_weights, ) - return fused def _rrf_merge( @@ -790,6 +982,7 @@ def run_search( generated_only: bool = False, exclude_generated: bool = False, dedup_by_fqn: bool = False, + rank_config: RankConfig = DEFAULT_RANK_CONFIG, ) -> list[dict]: effective_hybrid = hybrid effective_fts = fts_text @@ -887,6 +1080,7 @@ def run_search( if graph_expand and key == "java" and expand_depth > 0: rows = _graph_expand_merge( rows, + query=query, query_vec=query_vec, db=db, uri=uri, @@ -894,6 +1088,7 @@ def run_search( extra_predicates=extra_java, expand_depth=expand_depth, ladybug_path=ladybug_path, + rank_config=rank_config, ) # Dedup by primary_type_fqn after all sorting/merging, before windowing diff --git a/src/java_codebase_rag/search/search_lexical.py b/src/java_codebase_rag/search/search_lexical.py index 24cf562..964ace5 100644 --- a/src/java_codebase_rag/search/search_lexical.py +++ b/src/java_codebase_rag/search/search_lexical.py @@ -127,6 +127,11 @@ def _enclosing_type_fqn(fqn: str) -> str: return fqn.split("#", 1)[0] if fqn else fqn +# Non-underscore aliases for cross-module callers (search_lancedb's BM25 fusion). +# Behavior is identical; the leading-underscore originals stay module-private. +enclosing_type_fqn = _enclosing_type_fqn + + def _resolve_source_root(graph: LadybugGraph) -> str: """Authoritative source root is the one cached on the graph at index time.""" try: @@ -255,6 +260,11 @@ def _try_fts_candidates( return {"rows": rows, "scores": scores} +# Non-underscore alias for cross-module callers (search_lancedb's BM25 fusion on the +# vector path). Behavior is identical to the leading-underscore original. +fetch_fts_candidates = _try_fts_candidates + + def run_lexical_search( query: str, *, diff --git a/src/java_codebase_rag/search/search_scoring.py b/src/java_codebase_rag/search/search_scoring.py index b94336d..5a63828 100644 --- a/src/java_codebase_rag/search/search_scoring.py +++ b/src/java_codebase_rag/search/search_scoring.py @@ -13,6 +13,7 @@ import json import re +from dataclasses import dataclass # Name of the LadybugDB FTS (Okapi BM25) index over Symbol.search_text (fork A). # Shared by the build path (build_ast_graph._ensure_symbol_fts_index) and the @@ -84,13 +85,86 @@ "DTO": -0.08, } + +def _rrf_max(num_lists: int, k: int = 60) -> float: + """Return the theoretical maximum RRF score for N-list fusion. + + Reciprocal Rank Fusion (RRF) bounds each contribution to ≤ 1/(rank + k). + For N fused lists, the maximum possible sum is N/(k + 1) (achieved when + an item ranks #1 across all lists). + + Args: + num_lists: Number of ranked lists being fused (e.g., 2 for vector+lexical). + k: The RRF constant (default 60 per the original paper). + + Returns: + The maximum RRF contribution: num_lists / (k + 1). + """ + return num_lists / (k + 1) + + +# Allowed list names in a RankConfig: vector is always required (it is the +# backbone retrieval signal); graph and bm25 are optional fusion participants. +# The "bm25" list is wired in ``search_lancedb._graph_expand_merge`` (LadybugDB +# FTS candidate fetch), which fuses BM25-ranked Symbol candidates as a third +# RRF list alongside vector and graph. +_RANK_LIST_NAMES: frozenset[str] = frozenset({"vector", "graph", "bm25"}) + + +@dataclass(frozen=True) +class RankConfig: + """Which ranked lists to fuse and the RRF constant to fuse them with. + + This is a dep-free value object (no lancedb/torch) so it can be constructed + on every install flavor, including graph-only (macOS Intel). It is plumbed + through ``run_search`` → ``_graph_expand_merge`` to control (a) which lists + contribute to the final RRF fusion and (b) the ``k`` constant passed into + ``_rrf_merge``. + + Attributes: + lists: Subset of ``{"vector", "graph", "bm25"}``. Must contain + ``"vector"`` (the backbone retrieval signal) and be non-empty. + rrf_k: The RRF constant (default 60 per the original paper). Must be ≥ 1. + """ + + lists: frozenset[str] + rrf_k: int = 60 + + def __post_init__(self) -> None: + if not isinstance(self.lists, frozenset) or not self.lists: + raise ValueError("RankConfig.lists must be a non-empty frozenset") + if "vector" not in self.lists: + raise ValueError( + "RankConfig.lists must contain 'vector' (the backbone signal)" + ) + unknown = self.lists - _RANK_LIST_NAMES + if unknown: + raise ValueError( + f"RankConfig.lists has unknown names {sorted(unknown)!r}; " + f"allowed: {sorted(_RANK_LIST_NAMES)!r}" + ) + if not isinstance(self.rrf_k, int) or self.rrf_k < 1: + raise ValueError(f"RankConfig.rrf_k must be an int >= 1, got {self.rrf_k!r}") + + +# Production default: ship the 3-list config (vector+graph+bm25). The BM25 list +# is wired in ``search_lancedb._graph_expand_merge``; on installs without the +# vector stack, or when the FTS index is unavailable, it degrades silently to +# the 2-list (vector+graph) fusion. +DEFAULT_RANK_CONFIG = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60) + +# Eval convenience: the historical 2-list (vector+graph) fusion, used by +# evaluation harnesses that isolate the vector+graph baseline from the bm25 list. +BASELINE_2LIST_CONFIG = RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=60) + + # Theoretical maximum for hybrid composite score (used for display normalization). # Hybrid sort metric: raw_rrf * (import_factor if import_heavy else 1) # + role_weight + symbol_bonus # where raw_rrf ≤ 2/(k+1) for 2-list RRF, role_weight ≤ max(_ROLE_SCORE_WEIGHTS), # and symbol_bonus ≤ _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS. -# The import factor is ≤ 1, so we use the raw max (2/61). -_HYBRID_SCORE_MAX = (2.0 / 61.0) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS +# The import factor is ≤ 1, so we use the raw max (derived via _rrf_max(2)). +_HYBRID_SCORE_MAX = _rrf_max(2) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS def _query_tokens(query: str) -> set[str]: @@ -374,6 +448,9 @@ def explain_score_components( rrf = comps.get("rrf_raw") or comps.get("hybrid_rrf") if rrf is not None: parts.append(f"rrf={float(rrf):.3f}") + bm25 = comps.get("bm25") + if bm25: + parts.append(f"bm25={float(bm25):.3f}") else: d = comps.get("distance") if d is not None: diff --git a/tests/eval/__init__.py b/tests/eval/__init__.py new file mode 100644 index 0000000..3f8d105 --- /dev/null +++ b/tests/eval/__init__.py @@ -0,0 +1 @@ +# Test package marker diff --git a/tests/eval/test_ground_truth.py b/tests/eval/test_ground_truth.py new file mode 100644 index 0000000..d1804f0 --- /dev/null +++ b/tests/eval/test_ground_truth.py @@ -0,0 +1,115 @@ +"""Tests for eval.ground_truth — Tier-A generator + Tier-B loader.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from java_codebase_rag.eval.ground_truth import ( + LabeledQuery, + build_tier_a, + load_tier_b, +) + + +class _Sym: + """Minimal structural symbol stand-in (duck-typed .fqn / .name).""" + + def __init__(self, fqn: str, name: str) -> None: + self.fqn = fqn + self.name = name + + +class TestBuildTierA: + def test_build_tier_a_deterministic(self) -> None: + syms = [ + _Sym("com.example.DistributionChunkService", "DistributionChunkService"), + _Sym("com.example.Other", "Other"), + ] + out = build_tier_a(syms) + + fqn = "com.example.DistributionChunkService" + assert LabeledQuery("DistributionChunkService", frozenset({fqn}), "A") in out + assert LabeledQuery( + "distribution chunk service", frozenset({fqn}), "A" + ) in out + + # Deterministic: same input -> identical output + assert build_tier_a(syms) == out + + # Sorted by (query, fqn) + keys = [(q.query, next(iter(q.relevant))) for q in out] + assert keys == sorted(keys) + + def test_build_tier_a_skips_noise(self) -> None: + syms = [ + _Sym("com.example.A", "A"), # <3 chars + _Sym("com.example.Do", "Do"), # splits to single token + ] + assert build_tier_a(syms) == [] + + +class TestLoadTierB: + def test_load_tier_b_yaml(self, tmp_path: Path) -> None: + path = tmp_path / "tier_b.yaml" + path.write_text( + "- query: distribution chunk service\n" + " relevant:\n" + " - com.example.DistributionChunkService\n" + "- query: user service\n" + " relevant:\n" + " - com.example.UserService\n" + " - com.other.UserService\n" + ) + out = load_tier_b(path) + assert out == [ + LabeledQuery( + "distribution chunk service", + frozenset({"com.example.DistributionChunkService"}), + "B", + ), + LabeledQuery( + "user service", + frozenset({"com.example.UserService", "com.other.UserService"}), + "B", + ), + ] + + def test_load_tier_b_json(self, tmp_path: Path) -> None: + path = tmp_path / "tier_b.json" + path.write_text( + json.dumps( + [ + { + "query": "distribution chunk service", + "relevant": ["com.example.DistributionChunkService"], + }, + { + "query": "user service", + "relevant": [ + "com.example.UserService", + "com.other.UserService", + ], + }, + ] + ) + ) + out = load_tier_b(path) + assert out == [ + LabeledQuery( + "distribution chunk service", + frozenset({"com.example.DistributionChunkService"}), + "B", + ), + LabeledQuery( + "user service", + frozenset({"com.example.UserService", "com.other.UserService"}), + "B", + ), + ] + + def test_load_tier_b_missing_raises(self, tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError): + load_tier_b(tmp_path / "does_not_exist.yaml") diff --git a/tests/eval/test_metrics.py b/tests/eval/test_metrics.py new file mode 100644 index 0000000..0ca089b --- /dev/null +++ b/tests/eval/test_metrics.py @@ -0,0 +1,100 @@ +"""Tests for eval.metrics — hand-computed cases to verify metric math.""" + +import pytest +from java_codebase_rag.eval.metrics import ( + recall_at_k, + precision_at_k, + reciprocal_rank, + mean, + aggregate, +) + + +class TestRecallAtK: + def test_recall_at_k_basic(self): + retrieved = ["a", "b", "c"] + relevant = {"b", "d"} + k = 3 + assert recall_at_k(retrieved, relevant, k) == 0.5 # b found, d not + + def test_recall_at_k_small_k(self): + retrieved = ["a", "b", "c"] + relevant = {"b", "d"} + k = 1 + assert recall_at_k(retrieved, relevant, k) == 0.0 # b not in first position + + def test_recall_at_k_empty_relevant(self): + retrieved = ["a", "b", "c"] + relevant = set() + k = 3 + assert recall_at_k(retrieved, relevant, k) == 0.0 # empty relevant set + + def test_recall_at_k_k_larger_than_retrieved(self): + retrieved = ["a", "b", "c"] + relevant = {"b", "d"} + k = 10 # longer than retrieved + assert recall_at_k(retrieved, relevant, k) == 0.5 # only b found + + +class TestPrecisionAtK: + def test_precision_at_k_basic(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + k = 2 + assert precision_at_k(retrieved, relevant, k) == 0.5 # 1 out of 2 + + def test_precision_at_k_full_retrieved(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + k = 3 + assert precision_at_k(retrieved, relevant, k) == 1.0 / 3.0 # 1 out of 3 + + def test_precision_at_k_zero_k(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + k = 0 + assert precision_at_k(retrieved, relevant, k) == 0.0 # k == 0 + + +class TestReciprocalRank: + def test_reciprocal_rank_second_position(self): + retrieved = ["a", "b", "c"] + relevant = {"b"} + assert reciprocal_rank(retrieved, relevant) == 0.5 # 1/2 + + def test_reciprocal_rank_no_match(self): + retrieved = ["a", "b", "c"] + relevant = {"z"} + assert reciprocal_rank(retrieved, relevant) == 0.0 # no match + + def test_reciprocal_rank_first_position(self): + retrieved = ["a", "b", "c"] + relevant = {"a"} + assert reciprocal_rank(retrieved, relevant) == 1.0 # 1/1 + + +class TestMean: + def test_mean_basic(self): + values = [1.0, 0.0, 0.5] + assert mean(values) == 0.5 # (1.0 + 0.0 + 0.5) / 3 + + def test_mean_empty(self): + values = [] + assert mean(values) == 0.0 # empty list + + +class TestAggregate: + def test_aggregate_basic(self): + per_query = [ + {"recall@1": 1.0, "recall@5": 1.0, "recall@10": 1.0, "recall@20": 1.0, "precision@5": 1.0, "mrr": 1.0}, + {"recall@1": 0.0, "recall@5": 0.0, "recall@10": 0.0, "recall@20": 0.0, "precision@5": 0.0, "mrr": 0.0}, + ] + result = aggregate(per_query) + assert result == { + "recall@1": 0.5, + "recall@5": 0.5, + "recall@10": 0.5, + "recall@20": 0.5, + "precision@5": 0.5, + "mrr": 0.5, + } diff --git a/tests/eval/test_runner.py b/tests/eval/test_runner.py new file mode 100644 index 0000000..5d1eca2 --- /dev/null +++ b/tests/eval/test_runner.py @@ -0,0 +1,168 @@ +"""Tests for eval.runner — shopizer-style eval sweep over a tiny fixture corpus. + +These tests build a REAL (tiny) index via the operator CLI subprocess and run +``run_search`` under multiple ``RankConfig``s. They assert SHAPE and file +persistence only — never specific ranking numbers (those are research outputs). + +Skipped cleanly when the vector stack (torch / lancedb / sentence_transformers) +or the cocoindex CLI is unavailable (graph-only envs). +""" +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path + +import pytest + +# Skip the whole file when the vector stack is missing — runner needs it. +pytest.importorskip("lancedb") +pytest.importorskip("sentence_transformers") +pytest.importorskip("torch") + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent +TINY_CORPUS = REPO_ROOT / "tests" / "fixtures" / "cross_service_smoke" + +# Expected metric keys (table columns) — order-stable. +METRIC_KEYS = ( + "recall@1", + "recall@5", + "recall@10", + "recall@20", + "precision@5", + "mrr", + "p50_latency_ms", +) + + +def _cocoindex_available() -> bool: + """True when the cocoindex CLI sits next to the pytest interpreter.""" + return (Path(sys.executable).parent / "cocoindex").is_file() + + +pytestmark = pytest.mark.skipif( + not _cocoindex_available(), + reason="cocoindex CLI not installed in this venv; runner integration test needs the full stack", +) + + +def _cfg( + tmp_path: Path, + *, + tier_b_path: str | None = None, + tag: str = "run", + max_queries: int | None = None, +): + from java_codebase_rag.eval.runner import EvalConfig + + # Fresh index dir per tag — `init` refuses an occupied index_dir, and some + # tests invoke run_eval twice into the same tmp_path. + kwargs = dict( + corpus_dir=str(TINY_CORPUS), + index_dir=str(tmp_path / f"index_{tag}"), + results_dir=str(tmp_path / f"results_{tag}"), + tier_b_path=tier_b_path, + ks=(60,), # single k keeps the smoke fast; shape is what we assert + top_k_metrics=(1, 5, 10, 20), + ) + if max_queries is not None: + kwargs["max_queries"] = max_queries + return EvalConfig(**kwargs) + + +def test_eval_report_shape(tmp_path): + from java_codebase_rag.eval.runner import run_eval + + report = run_eval(_cfg(tmp_path)) + + # 1 baseline + 1 per swept k. + assert len(report.configs) == 1 + 1 + + for entry in report.configs: + assert entry.config_name + assert entry.num_queries >= 0 + for key in METRIC_KEYS: + assert key in entry.metrics, f"missing metric {key} in {entry.config_name}" + assert isinstance(entry.metrics[key], float) + assert entry.metrics["p50_latency_ms"] >= 0.0 + + +def test_eval_report_persists_files(tmp_path): + from java_codebase_rag.eval.runner import run_eval + + cfg = _cfg(tmp_path) + report = run_eval(cfg) + + # Outputs are namespaced under a timestamped subdir (no flat-path clobbering). + out_dir = Path(cfg.results_dir) / report.timestamp + md_path = out_dir / "report.md" + json_path = out_dir / "report.json" + assert md_path.is_file(), f"missing report.md at {md_path}" + assert json_path.is_file(), f"missing report.json at {json_path}" + # EvalReport also exposes the resolved out_dir. + assert report.out_dir == str(out_dir) + + md = md_path.read_text() + # Header row mentions every metric column. + for key in METRIC_KEYS: + assert key in md, f"report.md header missing column {key}" + # One data row per config (count pipe-led rows under the header). + data_rows = [ln for ln in md.splitlines() if ln.startswith("| ") and "-" not in ln[:3]] + assert len(data_rows) >= len(report.configs) + + payload = json.loads(json_path.read_text()) + assert "configs" in payload + assert len(payload["configs"]) == len(report.configs) + + +def test_eval_tier_b_optional(tmp_path): + """tier_b_path=None completes on Tier-A only; a one-entry file still works.""" + from java_codebase_rag.eval.runner import run_eval + + # None — no exception. + report = run_eval(_cfg(tmp_path, tier_b_path=None, tag="a")) + assert len(report.configs) == 1 + 1 + + # With a single Tier-B entry in a temp file. + tier_b = tmp_path / "tier_b.json" + tier_b.write_text( + json.dumps([{"query": "OrderService", "relevant": ["com.example.OrderService"]}]) + ) + report_b = run_eval(_cfg(tmp_path, tier_b_path=str(tier_b), tag="b")) + assert len(report_b.configs) == 1 + 1 + for entry in report_b.configs: + for key in METRIC_KEYS: + assert key in entry.metrics + + +def test_eval_max_queries_caps_tier_a(tmp_path): + """EvalConfig.max_queries deterministically caps the Tier-A query set. + + On the tiny fixture the default kind filter yields a handful of type-level + symbols, so a max_queries=2 cap must keep num_queries_available >= 2 while + the actually-scored Tier-A count is exactly 2 (no Tier-B here). Also + confirms the report records the pre-cap availability. + """ + from java_codebase_rag.eval.runner import run_eval + + report = run_eval(_cfg(tmp_path, max_queries=2, tag="cap")) + + # Pre-cap Tier-A availability is recorded and at least 2 (the fixture has + # multiple type-level symbols; the cap must have had something to bite on). + assert report.num_queries_available >= 2 + # With no Tier-B, num_queries == Tier-A kept == max_queries(2). + assert report.num_queries == 2 + # Every config scored exactly the capped query count. + for entry in report.configs: + assert entry.num_queries <= 2 + + +def test_eval_max_queries_validation(): + """max_queries < 1 is rejected with ValueError at construction time.""" + from java_codebase_rag.eval.runner import EvalConfig + + with pytest.raises(ValueError): + EvalConfig(max_queries=0) + # Boundary: 1 is accepted. + EvalConfig(max_queries=1) diff --git a/tests/search/test_search_lancedb.py b/tests/search/test_search_lancedb.py index cf889cd..3c114e9 100644 --- a/tests/search/test_search_lancedb.py +++ b/tests/search/test_search_lancedb.py @@ -11,7 +11,7 @@ pytest.importorskip("sentence_transformers") from java_codebase_rag.search import search_lancedb -from java_codebase_rag.search.search_lancedb import JAVA_ENRICHED_COLUMNS, _rrf_merge +from java_codebase_rag.search.search_lancedb import JAVA_ENRICHED_COLUMNS, _rrf_merge, run_search def test_rrf_merge_weights_second_list_by_row() -> None: @@ -67,6 +67,76 @@ def test_java_enriched_columns_include_symbol_identity_fields() -> None: assert "metadata" in JAVA_ENRICHED_COLUMNS +def test_graph_expand_merge_honors_injected_k(monkeypatch) -> None: + """RankConfig.rrf_k flows through _graph_expand_merge into _rrf_merge. + + With k=30 injected, a row reinforced at rank 0 across the vector and graph + lists has ``rrf_raw = 2/(k+1) = 2/31``. Under the old default k=60 it would + be 2/61 — strictly smaller — so observing 2/31 proves the injected k wins. + """ + import sys + import types + + from java_codebase_rag.search.search_scoring import RankConfig + + # Stub the lazily-imported LadybugGraph so the function reaches the merge. + class _FakeGraph: + @staticmethod + def exists(path): + return True + + @staticmethod + def get(path): + class _G: + def expand_fqns(self, fqns, depth): + # One novel FQN so the function proceeds to graph fetch + fuse. + return ["com.example.Other"] + + def expand_methods(self, fqns, depth, exclude_external=False): + return [] + return _G() + + fake_mod = types.ModuleType("java_codebase_rag.graph.ladybug_queries") + fake_mod.LadybugGraph = _FakeGraph + monkeypatch.setitem(sys.modules, "java_codebase_rag.graph.ladybug_queries", fake_mod) + + # Same (filename, range_start, range_end) key in both lists → rank-0 in both, + # so raw RRF = 1/(k+1) + 1/(k+1) = 2/(k+1). + vector_rows = [ + {"filename": "a.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.example.Foo"}, + ] + graph_rows = [ + {"filename": "a.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.example.Other"}, + ] + + monkeypatch.setattr(search_lancedb, "_search_one_table", lambda *a, **kw: graph_rows) + monkeypatch.setattr(search_lancedb, "_table_columns", lambda *a, **kw: set()) + monkeypatch.setattr(search_lancedb, "_build_extra_predicates", lambda **kw: []) + monkeypatch.setattr(search_lancedb, "_apply_chunk_hints", lambda rows: None) + monkeypatch.setattr(search_lancedb, "_refine_java_start_lines", lambda rows: None) + monkeypatch.setattr(search_lancedb, "_vector_sort_key", lambda r: 0.0) + + result = search_lancedb._graph_expand_merge( + vector_rows, + query="query", + query_vec=np.zeros(3), + db=object(), + uri="mem://", + limit=10, + extra_predicates=[], + expand_depth=1, + ladybug_path=None, + rank_config=RankConfig(lists=frozenset({"vector", "graph"}), rrf_k=30), + ) + + top = result[0] + # k=30 → raw = 2/31; would be 2/61 if the default leaked through. + assert top["_score_components"]["rrf_raw"] == pytest.approx(2.0 / 31.0, abs=1e-12) + assert top["_score_components"]["rrf_raw"] > 2.0 / 61.0 + + def test_search_one_table_selects_symbol_identity_columns_when_schema_has_them(monkeypatch) -> None: selected: list[str] = [] @@ -654,3 +724,668 @@ def test_refine_java_start_lines_skips_nonjava_and_method_chunks() -> None: assert rows[0]["start"]["line"] == 7 assert rows[1]["start"]["line"] == 30 assert "start" not in rows[2] + + +# ---------- Task 4: BM25 candidate fetch + third RRF list ---------- + + +def _eval_pred(rows: list[dict], pred: str | None) -> list[dict]: + """Tiny SQL-ish predicate evaluator for test fakes (AND / IN / <> / =). + + Splits on `` AND `` WITHOUT destroying parens (the prior implementation + pre-stripped ``(``/``)`` everywhere, which wiped the ``IN (...)`` marker so + the ``primary_type_fqn IN (...)`` filter never evaluated and rows that + should be filtered passed). ``_combine_predicates`` wraps each conjunct in a + paren pair when there are several, so we strip ONE outer paren layer per + clause while preserving the inner ``IN (...)`` parens needed to extract the + value list. + """ + if not pred: + return list(rows) + clauses: list[str] = [] + for c in pred.split(" AND "): + c = c.strip() + # Strip ONE outer wrapping paren pair (multi-predicate case); leave + # ``col IN (...)`` and its inner value-list parens intact. + if c.startswith("(") and c.endswith(")"): + c = c[1:-1].strip() + if c: + clauses.append(c) + out: list[dict] = [] + for r in rows: + keep = True + for clause in clauses: + if " IN (" in clause: + col = clause.split(" IN ", 1)[0].strip() + vals_part = clause[clause.index("(") + 1 : clause.rindex(")")] + vals = [v.strip().strip("'") for v in vals_part.split(",")] + if str(r.get(col)) not in vals: + keep = False + break + elif "<>" in clause: + col, val = [p.strip() for p in clause.split("<>", 1)] + val = val.strip("'") + if str(r.get(col)) == val: + keep = False + break + elif "=" in clause: + col, val = [p.strip() for p in clause.split("=", 1)] + val = val.strip("'") + if str(r.get(col)) != val: + keep = False + break + if keep: + out.append(r) + return out + + +class _FilterQuery: + """Fake LanceDB filter-only query: search().where().select().limit().to_list().""" + + def __init__(self, rows: list[dict]) -> None: + self._rows = rows + self._pred: str | None = None + self._limit: int | None = None + + def where(self, pred: str | None, prefilter: bool = False) -> "_FilterQuery": + self._pred = pred + return self + + def select(self, _cols: list[str]) -> "_FilterQuery": + return self + + def limit(self, n: int) -> "_FilterQuery": + self._limit = n + return self + + def to_list(self) -> list[dict]: + filtered = _eval_pred(self._rows, self._pred) + if self._limit is not None: + filtered = filtered[: self._limit] + return list(filtered) + + +class _FilterTable: + """Fake LanceDB table: open_table(...).search() with no vector → filter-only.""" + + def __init__(self, rows: list[dict]) -> None: + self._rows = rows + + def search(self, *args, **kwargs) -> _FilterQuery: + # Filter-only when no positional vector arg is passed. + return _FilterQuery(self._rows) + + +class _RecordingDb: + """Fake DB recording open_table calls; returns a per-table _FilterTable.""" + + def __init__(self) -> None: + self.tables: dict[str, _FilterTable] = {} + self.opened: list[str] = [] + + def add(self, name: str, rows: list[dict]) -> None: + self.tables[name] = _FilterTable(rows) + + def open_table(self, name: str) -> _FilterTable: + self.opened.append(name) + return self.tables.get(name) or _FilterTable([]) + + +def _bm25_chunk(filename: str, fqn: str, rs: int, re_: int) -> dict: + return { + "filename": filename, + "range_start": rs, + "range_end": re_, + "primary_type_fqn": fqn, + "text": f"body of {fqn}", + "language": "java", + "start": {"line": rs, "byte_offset": 0}, + "end": {"line": re_, "byte_offset": 100}, + } + + +def _patch_bm25_environment(monkeypatch, *, fts_result, chunk_rows) -> _RecordingDb: + """Wire the common monkeypatches for _bm25_candidate_rows unit tests.""" + from java_codebase_rag.search import search_lexical + + monkeypatch.setattr(search_lexical, "fetch_fts_candidates", lambda *a, **kw: fts_result) + monkeypatch.setattr(search_lexical, "enclosing_type_fqn", lambda fqn: fqn.split("#", 1)[0]) + monkeypatch.setattr(search_lancedb, "_apply_chunk_hints", lambda rows: None) + monkeypatch.setattr(search_lancedb, "_refine_java_start_lines", lambda rows: None) + # Provide a schema that includes the enriched java columns the helper selects. + monkeypatch.setattr( + search_lancedb, "_table_columns", + lambda *a, **kw: {"filename", "text", "start", "end", "language", "range_start", + "range_end", "primary_type_fqn", "role", "package"}, + ) + db = _RecordingDb() + db.add(search_lancedb.TABLES["java"], list(chunk_rows)) + return db + + +def test_bm25_candidate_rows_orders_by_bm25_score(monkeypatch) -> None: + """(a) BM25 candidates are emitted in BM25-desc order with the owning score attached. + + FTS rows are supplied SCRAMBLED relative to score order [C(10), A(20), B(30)] + so only a real score-desc sort produces the asserted [B, A, C] output — a + first-seen/passthrough bug would yield [C, A, B] instead. + """ + fts_result = { + "rows": [ + {"id": "sc", "fqn": "com.x.C", "kind": "class", "name": "C"}, + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, + ], + "scores": {"sb": 30.0, "sa": 20.0, "sc": 10.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("b.java", "com.x.B", 1, 10), + _bm25_chunk("c.java", "com.x.C", 1, 10), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + fqns = [r["primary_type_fqn"] for r in out] + assert fqns == ["com.x.B", "com.x.A", "com.x.C"] + scores = {r["primary_type_fqn"]: r["_score_components"]["bm25"] for r in out} + assert scores == {"com.x.B": 30.0, "com.x.A": 20.0, "com.x.C": 10.0} + + +def test_bm25_candidate_rows_fts_unavailable_returns_empty(monkeypatch) -> None: + """(b) FTS returns None → [] and no LanceDB fetch is attempted.""" + db = _patch_bm25_environment(monkeypatch, fts_result=None, chunk_rows=[]) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + assert out == [] + assert db.opened == [] # no chunk fetch attempted + + +def test_bm25_candidate_rows_respects_filter(monkeypatch) -> None: + """(c) extra_predicates flow into the chunk fetch and filter out types.""" + fts_result = { + "rows": [ + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, + ], + "scores": {"sa": 20.0, "sb": 10.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("b.java", "com.x.B", 1, 10), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=["primary_type_fqn <> 'com.x.B'"], + columns={"primary_type_fqn"}, + ) + fqns = [r["primary_type_fqn"] for r in out] + assert fqns == ["com.x.A"] + + +def test_bm25_candidate_rows_in_predicate_excludes_non_matching_types(monkeypatch) -> None: + """(c2) The ``primary_type_fqn IN (...)`` predicate (built from ordered_types) + excludes chunks whose FQN is not among the BM25-ordered types. + + Regression: the ``_RecordingDb`` fake's ``_eval_pred`` previously destroyed + parens before checking ``" IN ("``, so the IN predicate never evaluated and + non-matching chunks leaked through. FTS yields only ``com.x.A`` (so + ordered_types=['com.x.A'] and the IN predicate is ``primary_type_fqn IN + ('com.x.A')``); a ``com.x.B`` chunk sitting in the table must be filtered out. + """ + fts_result = { + "rows": [ + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + ], + "scores": {"sa": 20.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("b.java", "com.x.B", 1, 10), # NOT in ordered_types — must be filtered + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + fqns = [r["primary_type_fqn"] for r in out] + assert fqns == ["com.x.A"] + assert "com.x.B" not in fqns # filtered out by the IN (...) predicate + + +def test_bm25_candidate_rows_multiple_chunks_per_symbol_preserve_order(monkeypatch) -> None: + """(d) Multiple chunks per type stay grouped, in table order, ordered by BM25 rank.""" + fts_result = { + "rows": [ + {"id": "sa", "fqn": "com.x.A", "kind": "class", "name": "A"}, + {"id": "sb", "fqn": "com.x.B", "kind": "class", "name": "B"}, + ], + "scores": {"sa": 20.0, "sb": 10.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("a.java", "com.x.A", 11, 20), + _bm25_chunk("b.java", "com.x.B", 1, 10), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + keys = [(r["primary_type_fqn"], r["range_start"]) for r in out] + assert keys == [("com.x.A", 1), ("com.x.A", 11), ("com.x.B", 1)] + for r in out: + if r["primary_type_fqn"] == "com.x.A": + assert r["_score_components"]["bm25"] == 20.0 + else: + assert r["_score_components"]["bm25"] == 10.0 + + +def test_bm25_candidate_rows_dedup_members_of_same_type_keep_max(monkeypatch) -> None: + """(d2) Two member symbols (#-fqns) of the SAME type dedup to one type entry, + emitted ONCE at the MAX BM25 score among the members. + + FTS rows are ordered LOWER-score-first [sm2(20), sm1(30)] so a first-wins + bug would yield 20.0 and only a keep-max implementation yields the asserted + 30.0 — this discriminates keep-max from first-seen-wins. + """ + fts_result = { + "rows": [ + {"id": "sm2", "fqn": "com.x.A#method2()", "kind": "method", "name": "method2"}, + {"id": "sm1", "fqn": "com.x.A#method1()", "kind": "method", "name": "method1"}, + ], + "scores": {"sm1": 30.0, "sm2": 20.0}, + } + chunk_rows = [ + _bm25_chunk("a.java", "com.x.A", 1, 10), + _bm25_chunk("a.java", "com.x.A", 11, 20), + ] + db = _patch_bm25_environment(monkeypatch, fts_result=fts_result, chunk_rows=chunk_rows) + + out = search_lancedb._bm25_candidate_rows( + g=object(), query="query", uri="mem://", db=db, + extra_predicates=[], columns={"primary_type_fqn"}, + ) + # The type appears exactly once (both member chunks present, but only one type rank). + type_fqns = [r["primary_type_fqn"] for r in out] + assert type_fqns.count("com.x.A") == 2 # 2 chunks of the same single type + assert set(type_fqns) == {"com.x.A"} # no other type leaked; deduped to one entry + # Keep-max: every emitted chunk carries the MAX member score (30.0), not 20.0. + for r in out: + assert r["_score_components"]["bm25"] == 30.0 + + +def _stub_ladybug_graph(monkeypatch) -> None: + """Install a LadybugGraph stub that exists but expands to nothing novel.""" + import sys + import types + + class _FakeGraph: + @staticmethod + def exists(_path): + return True + + @staticmethod + def get(_path): + class _G: + def expand_fqns(self, fqns, depth): + return [] + + def expand_methods(self, fqns, depth, exclude_external=False): + return [] + + return _G() + + fake_mod = types.ModuleType("java_codebase_rag.graph.ladybug_queries") + fake_mod.LadybugGraph = _FakeGraph + monkeypatch.setitem(sys.modules, "java_codebase_rag.graph.ladybug_queries", fake_mod) + + +def test_graph_expand_merge_includes_bm25_list(monkeypatch) -> None: + """(e) With DEFAULT_RANK_CONFIG (3-list), a BM25-only row surfaces in the fusion.""" + from java_codebase_rag.search.search_scoring import RankConfig + + _stub_ladybug_graph(monkeypatch) + rc = RankConfig(lists=frozenset({"vector", "graph", "bm25"}), rrf_k=60) + + vector_rows = [ + {"filename": "v.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.V"}, + ] + bm25_rows = [ + {"filename": "b.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.B", "_kind": "java", + "_score_components": {"bm25": 0.42}}, + ] + monkeypatch.setattr(search_lancedb, "_bm25_candidate_rows", lambda **kw: list(bm25_rows)) + monkeypatch.setattr(search_lancedb, "_table_columns", lambda *a, **kw: set()) + monkeypatch.setattr(search_lancedb, "_build_extra_predicates", lambda **kw: []) + # Force the graph path to produce nothing (no novel fqns via stub) → graph_rows = []. + + result = search_lancedb._graph_expand_merge( + vector_rows, + query="something", + query_vec=np.zeros(3), + db=object(), + uri="mem://", + limit=10, + extra_predicates=[], + expand_depth=1, + ladybug_path=None, + rank_config=rc, + ) + files = [r["filename"] for r in result] + assert "b.java" in files + bm25_row = next(r for r in result if r["filename"] == "b.java") + assert bm25_row["_score_components"]["bm25"] == 0.42 + + +def test_graph_expand_merge_omits_bm25_when_excluded(monkeypatch) -> None: + """(f) With BASELINE_2LIST_CONFIG, _bm25_candidate_rows is never called.""" + from java_codebase_rag.search.search_scoring import BASELINE_2LIST_CONFIG + + _stub_ladybug_graph(monkeypatch) + calls: list[dict] = [] + monkeypatch.setattr(search_lancedb, "_bm25_candidate_rows", + lambda **kw: calls.append(kw) or []) + monkeypatch.setattr(search_lancedb, "_table_columns", lambda *a, **kw: set()) + monkeypatch.setattr(search_lancedb, "_build_extra_predicates", lambda **kw: []) + + vector_rows = [ + {"filename": "v.java", "range_start": 1, "range_end": 10, + "primary_type_fqn": "com.V"}, + ] + result = search_lancedb._graph_expand_merge( + vector_rows, + query="something", + query_vec=np.zeros(3), + db=object(), + uri="mem://", + limit=10, + extra_predicates=[], + expand_depth=1, + ladybug_path=None, + rank_config=BASELINE_2LIST_CONFIG, + ) + assert calls == [] + # Result is the vector rows (graph produced nothing novel via stub). + assert result == vector_rows + + +def test_run_search_bm25_degrades_silently_when_fts_missing(monkeypatch, tmp_path) -> None: + """(g) When FTS is unavailable, the 3-list config degrades to the 2-list baseline. + + Builds a real LanceDB index with one java row, stubs LadybugGraph to exist (so + _graph_expand_merge runs) but expand to nothing, and forces _ensure_fts_loaded→False + so the BM25 fetch returns None. Asserts: no exception, no row carries a `bm25` + score component, and the result equals the BASELINE_2LIST_CONFIG run. + """ + import uuid + + import lancedb + from sentence_transformers import SentenceTransformer + + from java_codebase_rag.ast.ast_java import ONTOLOGY_VERSION + from java_codebase_rag.search import search_lexical + from java_codebase_rag.search.index_common import SBERT_MODEL + from java_codebase_rag.search.search_lancedb import TABLES, _query_vector + from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + ) + + _stub_ladybug_graph(monkeypatch) + # Force FTS unavailable → _try_fts_candidates returns None. + monkeypatch.setattr(search_lexical, "_ensure_fts_loaded", lambda g: False) + + uri = str(tmp_path / "ldb") + model = SentenceTransformer(SBERT_MODEL, device="cpu", trust_remote_code=True) + text = "service that processes inbound Kafka records on the listener endpoint" + emb = _query_vector(model, text) + row = { + "id": str(uuid.uuid4()), + "filename": "smoke/p/Svc.java", + "text": text, + "language": "java", + "range_start": 0, + "range_end": 500, + "start": {"line": 1, "byte_offset": 0}, + "end": {"line": 20, "byte_offset": 400}, + "embedding": emb, + "package": "p", + "module": "smoke", + "microservice": "smoke", + "primary_type_fqn": "p.Svc", + "primary_type_kind": "class", + "role": "SERVICE", + "annotations_on_type": [], + "symbols": ["process"], + "ontology_version": ONTOLOGY_VERSION, + "capabilities": [], + } + db = lancedb.connect(uri) + db.create_table(TABLES["java"], [row], mode="create") + + common = dict( + uri=uri, table_keys=["java"], limit=5, path_substring=None, + model_name=SBERT_MODEL, device="cpu", model=model, + graph_expand=True, expand_depth=1, + ) + three = run_search(text, rank_config=DEFAULT_RANK_CONFIG, **common) + two = run_search(text, rank_config=BASELINE_2LIST_CONFIG, **common) + + # No bm25 component anywhere on either run. + for rows in (three, two): + assert all("bm25" not in (r.get("_score_components") or {}) for r in rows) + # Degradation: 3-list with FTS-missing == 2-list baseline. + assert [r["filename"] for r in three] == [r["filename"] for r in two] + assert len(three) == len(two) and len(two) >= 1 + + +def _build_minimal_fts_graph(db_path, *, symbol_fqn: str, symbol_name: str) -> bool: + """Build a 1-Symbol LadybugDB graph with the real ``sym_fts`` BM25 index. + + Returns True when the FTS index was created (extension available); False when + the FTS extension could not load (so the caller can ``pytest.skip``). + + Mirrors the production write path: ``_drop_all`` + ``_create_schema`` for the + table structure, one Symbol node + one GraphMeta node, then + ``_ensure_symbol_fts_index`` to build the BM25 index over ``search_text``. + """ + import ladybug + + from java_codebase_rag.ast.ast_java import ONTOLOGY_VERSION + from java_codebase_rag.graph.build_ast_graph import ( + _compute_symbol_search_text, + _create_schema, + _drop_all, + _ensure_symbol_fts_index, + ) + from java_codebase_rag.search.search_scoring import SYMBOL_FTS_INDEX + + db_path.parent.mkdir(parents=True, exist_ok=True) + db = ladybug.Database(str(db_path)) + conn = ladybug.Connection(db) + try: + _drop_all(conn) + _create_schema(conn) + search_text = _compute_symbol_search_text( + name=symbol_name, fqn=symbol_fqn, signature="", + annotations=[], capabilities=[], package=symbol_fqn.rsplit(".", 1)[0], + ) + conn.execute( + "MERGE (s:Symbol {id: $id}) " + "SET s.kind = 'class', s.name = $name, s.fqn = $fqn, " + "s.package = $package, s.module = '', s.microservice = '', " + "s.filename = $filename, s.start_line = 1, s.end_line = 20, " + "s.start_byte = 0, s.end_byte = 200, " + "s.modifiers = $modifiers, s.annotations = $annotations, " + "s.capabilities = $capabilities, s.role = 'SERVICE', " + "s.signature = '', s.parent_id = '', s.resolved = false, " + "s.generated = false, s.generated_by = '', " + "s.search_text = $search_text", + { + "id": symbol_fqn, "name": symbol_name, "fqn": symbol_fqn, + "package": symbol_fqn.rsplit(".", 1)[0], + "filename": f"{symbol_name}.java", + "modifiers": [], "annotations": [], "capabilities": [], + "search_text": search_text, + }, + ) + # GraphMeta with current ontology so LadybugGraph.get() accepts it. + conn.execute( + "MERGE (m:GraphMeta {key: 'meta'}) " + "SET m.ontology_version = $ov, m.built_at = 0, m.source_root = '', " + "m.counts_json = '', m.parse_errors = 0", + {"ov": ONTOLOGY_VERSION}, + ) + _ensure_symbol_fts_index(conn, verbose=False) + idx = conn.execute("CALL SHOW_INDEXES() RETURN index_name") + names: set[str] = set() + while idx.has_next(): + names.add(idx.get_next()[0]) + return SYMBOL_FTS_INDEX in names + finally: + conn.close() + db.close() + + +def test_run_search_bm25_contributes_on_camelcase_query_via_real_fts(tmp_path) -> None: + """(h) A camelCase identifier query lands in sym_fts's token space and the BM25 + list contributes to the 3-list fusion. + + Regression for A-I1: ``_bm25_candidate_rows`` used to pass the RAW query to + ``fetch_fts_candidates``; LadybugDB FTS does not split camelCase, so a query + like ``DistributionChunkService`` matched nothing and BM25 silently no-op'd. + With the ``build_fts_query`` pre-split, the FTS path matches. + + Exercises the UN-stubbed FTS path: a real LadybugDB ``sym_fts`` index on a + 1-Symbol graph + a real LanceDB index with a matching chunk. Does NOT + monkeypatch ``fetch_fts_candidates``. + + BM25 contribution is asserted two ways: + 1. Direct (un-stubbed) ``fetch_fts_candidates(g, build_fts_query(name))`` + returns the namesake Symbol with a positive BM25 score; the RAW query + returns nothing (the bug). + 2. ``run_search`` under the 3-list config produces a row whose + ``_score_components`` carries ``rrf_raw`` — set only by ``_rrf_merge`` + when the bm25 list joined the fusion. The graph is edge-less, so the + second list is necessarily bm25; the 2-list baseline has no ``rrf_raw``. + """ + import uuid + + import lancedb + from sentence_transformers import SentenceTransformer + + from java_codebase_rag.ast.ast_java import ONTOLOGY_VERSION + from java_codebase_rag.graph.ladybug_queries import LadybugGraph + from java_codebase_rag.search import search_lexical + from java_codebase_rag.search.index_common import SBERT_MODEL + from java_codebase_rag.search.search_lancedb import TABLES, _query_vector + from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + build_fts_query, + ) + + symbol_name = "DistributionChunkService" # multi-token camelCase — the trigger + symbol_fqn = f"smoke.{symbol_name}" + + # 1. Real LadybugDB graph: 1 Symbol + sym_fts BM25 index. + ladybug_path = tmp_path / "code_graph.lbug" + fts_ready = _build_minimal_fts_graph( + ladybug_path, symbol_fqn=symbol_fqn, symbol_name=symbol_name, + ) + if not fts_ready: + pytest.skip("FTS extension unavailable in this environment") + # build_fts_query must actually split the identifier (else the test is moot). + assert build_fts_query(symbol_name) == "distribution chunk service" + + LadybugGraph.reset_for_path(None) + g = LadybugGraph.get(str(ladybug_path)) + + # 2. Direct (un-stubbed) FTS sanity check: pre-split matches, raw does not. + pre = search_lexical.fetch_fts_candidates( + g, build_fts_query(symbol_name), filter=None, path_contains=None, + ) + assert pre and pre["rows"], "pre-split query must match via the real sym_fts index" + assert symbol_fqn in {r["fqn"] for r in pre["rows"]} + assert any(v > 0.0 for v in (pre.get("scores") or {}).values()) + raw = search_lexical.fetch_fts_candidates( + g, symbol_name, filter=None, path_contains=None, + ) + assert not raw or not raw.get("rows"), ( + "raw camelCase query should NOT match (FTS does not split camelCase); " + f"got: {raw}" + ) + + # 3. Real LanceDB index: 1 chunk matching the Symbol's enclosing type. + uri = str(tmp_path / "ldb") + model = SentenceTransformer(SBERT_MODEL, device="cpu", trust_remote_code=True) + text = "service that distributes chunks across the pipeline stages" + emb = _query_vector(model, text) + row = { + "id": str(uuid.uuid4()), + "filename": f"smoke/{symbol_name}.java", + "text": text, + "language": "java", + "range_start": 0, + "range_end": 500, + "start": {"line": 1, "byte_offset": 0}, + "end": {"line": 20, "byte_offset": 400}, + "embedding": emb, + "package": "smoke", + "module": "smoke", + "microservice": "smoke", + "primary_type_fqn": symbol_fqn, + "primary_type_kind": "class", + "role": "SERVICE", + "annotations_on_type": [], + "symbols": ["distribute"], + "ontology_version": ONTOLOGY_VERSION, + "capabilities": [], + } + db = lancedb.connect(uri) + db.create_table(TABLES["java"], [row], mode="create") + + common = dict( + uri=uri, table_keys=["java"], limit=5, path_substring=None, + model_name=SBERT_MODEL, device="cpu", model=model, + graph_expand=True, expand_depth=1, ladybug_path=str(ladybug_path), + ) + # 4. 3-list run: bm25 joins the fusion → _rrf_merge runs → rrf_raw appears. + three = run_search(symbol_name, rank_config=DEFAULT_RANK_CONFIG, **common) + assert three, "expected non-empty results for the namesake query" + has_rrf_raw = [ + r for r in three if "rrf_raw" in (r.get("_score_components") or {}) + ] + assert has_rrf_raw, ( + "3-list run should reflect bm25 fusion (rrf_raw set by _rrf_merge); " + f"components: {[r.get('_score_components') for r in three]}" + ) + # rrf_raw > 0 confirms the bm25 list contributed a positive rank contribution. + assert any( + float((r.get("_score_components") or {}).get("rrf_raw", 0.0)) > 0.0 + for r in has_rrf_raw + ) + # The namesake type is present in the 3-list result. + assert symbol_fqn in {r.get("primary_type_fqn") for r in three} + + # 5. 2-list baseline (vector+graph): edge-less graph → no fusion → no rrf_raw. + # Drop the cached singleton so the baseline run reopens cleanly. + LadybugGraph.reset_for_path(None) + two = run_search(symbol_name, rank_config=BASELINE_2LIST_CONFIG, **common) + assert all( + "rrf_raw" not in (r.get("_score_components") or {}) for r in two + ), "2-list baseline should not produce rrf_raw (graph is edge-less)" + diff --git a/tests/search/test_search_scoring.py b/tests/search/test_search_scoring.py index 9d865d2..94cbcb7 100644 --- a/tests/search/test_search_scoring.py +++ b/tests/search/test_search_scoring.py @@ -10,7 +10,13 @@ import pytest from java_codebase_rag.search.search_scoring import ( + _ACTION_VERB_BONUS, + _HYBRID_SCORE_MAX, + _ROLE_SCORE_WEIGHTS, + _SYMBOL_MATCH_BONUS_CAP, + _TYPE_MATCH_BONUS_CAP, declaration_line_number, + explain_score_components, vector_display_score, ) @@ -83,3 +89,106 @@ def test_declaration_line_number_method_only_chunk_keeps_anchor() -> None: # Empty / missing inputs are safe. assert declaration_line_number(None, 5) == 5 assert declaration_line_number("public class X {", None) is None + + +# ---------- _rrf_max (Task 1) ---------- + + +def test_rrf_max_formula() -> None: + """Test the RRF max formula: num_lists / (k + 1).""" + from java_codebase_rag.search.search_scoring import _rrf_max + + # Test the exact formula with different inputs + assert _rrf_max(2, 60) == pytest.approx(2.0 / 61.0, abs=1e-12) + assert _rrf_max(3, 60) == pytest.approx(3.0 / 61.0, abs=1e-12) + assert _rrf_max(3, 30) == pytest.approx(3.0 / 31.0, abs=1e-12) + + +def test_hybrid_score_max_unchanged() -> None: + """Verify _HYBRID_SCORE_MAX preserves its exact numeric value after refactor.""" + # Compute the expected value from the same constants used in the definition + expected = (2.0 / 61.0) + max(_ROLE_SCORE_WEIGHTS.values()) + _SYMBOL_MATCH_BONUS_CAP + _TYPE_MATCH_BONUS_CAP + _ACTION_VERB_BONUS + + # The refactor must not introduce any numeric drift + assert _HYBRID_SCORE_MAX == pytest.approx(expected, abs=1e-12) + + +# ---------- RankConfig (Task 2) ---------- + + +def test_rank_config_defaults() -> None: + """DEFAULT_RANK_CONFIG ships the 3-list set (bm25 inert until Task 4) + and the canonical RRF k=60 from the original paper.""" + from java_codebase_rag.search.search_scoring import ( + BASELINE_2LIST_CONFIG, + DEFAULT_RANK_CONFIG, + ) + + assert DEFAULT_RANK_CONFIG.lists == frozenset({"vector", "graph", "bm25"}) + assert DEFAULT_RANK_CONFIG.rrf_k == 60 + # Eval convenience: omits bm25. + assert BASELINE_2LIST_CONFIG.lists == frozenset({"vector", "graph"}) + assert BASELINE_2LIST_CONFIG.rrf_k == 60 + + +def test_rank_config_validation() -> None: + """RankConfig validates its lists set and rrf_k range at construction.""" + from java_codebase_rag.search.search_scoring import RankConfig + + # Missing required "vector" element. + with pytest.raises(ValueError): + RankConfig(lists=frozenset({"graph"})) + # Unknown list name. + with pytest.raises(ValueError): + RankConfig(lists=frozenset({"vector", "nope"})) + # Empty set. + with pytest.raises(ValueError): + RankConfig(lists=frozenset()) + # rrf_k below 1. + with pytest.raises(ValueError): + RankConfig(lists=frozenset({"vector"}), rrf_k=0) + # Sanity: a minimal valid config constructs cleanly. + ok = RankConfig(lists=frozenset({"vector"}), rrf_k=1) + assert ok.lists == frozenset({"vector"}) + assert ok.rrf_k == 1 + + +def test_rank_config_frozen() -> None: + """RankConfig is frozen so configs can be safely shared as defaults.""" + from dataclasses import FrozenInstanceError + + from java_codebase_rag.search.search_scoring import DEFAULT_RANK_CONFIG + + with pytest.raises(FrozenInstanceError): + DEFAULT_RANK_CONFIG.rrf_k = 5 # type: ignore[misc] + + +# ---------- explain_score_components (Task 3) ---------- + + +def test_explain_bm25_token_present() -> None: + """When hybrid=True and bm25 is truthy, output contains both rrf= and bm25= tokens.""" + result = explain_score_components({"rrf_raw": 0.03, "bm25": 12.5}, hybrid=True) + assert "rrf=0.030" in result + assert "bm25=12.500" in result + # Verify order: rrf appears before bm25 + assert result.index("rrf=0.030") < result.index("bm25=12.500") + + +def test_explain_bm25_token_absent_when_zero_or_missing() -> None: + """When bm25 is missing or zero, no bm25= token is emitted.""" + # Missing bm25 + result1 = explain_score_components({"rrf_raw": 0.03}, hybrid=True) + assert "bm25=" not in result1 + # Zero bm25 + result2 = explain_score_components({"rrf_raw": 0.03, "bm25": 0.0}, hybrid=True) + assert "bm25=" not in result2 + + +def test_explain_bm25_only_in_hybrid() -> None: + """bm25= token is only emitted in hybrid mode, not lexical mode.""" + result = explain_score_components({"bm25": 12.5}, lexical=True) + assert "bm25=" not in result + # Lexical mode should still emit its standard tokens + assert "relevance=" in result or "name=" in result or not result # May be empty if no lexical comps +