Skip to content

fix(search): keep curated indexes consistent - #2

Open
kaykayyali wants to merge 6 commits into
devnill:mainfrom
kaykayyali:fix/issue-1-search-index-consistency
Open

fix(search): keep curated indexes consistent#2
kaykayyali wants to merge 6 commits into
devnill:mainfrom
kaykayyali:fix/issue-1-search-index-consistency

Conversation

@kaykayyali

@kaykayyali kaykayyali commented Jul 20, 2026

Copy link
Copy Markdown

Summary

  • replace obsolete same-path FTS identities when a note gains or changes its frontmatter ID
  • prune missing, out-of-vault, and hidden-folder notes so .trash and test data cannot remain searchable
  • expose pruning through Hybrid/auto, remove stale semantic vectors, and fail closed on invalid semantic paths
  • handle hidden symlink aliases and symlink-resolution failures safely
  • use the pinned USearch 2.23 count query parameter so semantic search actually runs
  • route public recall fallback through the same guarded Grep backend so deleted notes cannot reappear
  • reject stale FTS rows at read time even when a fresh scan marker skips maintenance
  • include Obsidian filename stems when validating wikilink targets

Why

Vault curation can change a note's identity or move notes into hidden folders while the physical file still exists. The search index previously retained obsolete rows, and Hybrid/auto could bypass or semantically resurrect them. Audit relation checks also treated filename-based wikilinks as broken when frontmatter titles differed.

Tests

  • uv run pytest tests/ -q
  • uv run ruff check src/cyberbrain/extractors/search_backends.py src/cyberbrain/mcp/tools/{audit,recall}.py tests/{test_audit,test_mcp_server,test_search_backends}.py
  • real USearch 2.23 probe: stale vector absent after prune, valid vector retained across save/reload, and public Hybrid search returned no stale result
  • live HTTP cb_recall probe: a term present only in .trash returned No notes found
  • live HTTP stale-FTS probe: a persisted hidden row plus fresh scan marker returned No notes found, then cleanup restored index parity
  • live HTTP MCP Hybrid prune probe followed by the vault acceptance verifier: 456 active notes, 456 indexed rows, 456 unique paths, zero issues

Closes #1

Hermes Agent added 4 commits July 20, 2026 07:21
Replace obsolete same-path identities, prune out-of-vault and hidden notes, and accept filename stems when auditing Obsidian links.

Closes devnill#1
Delegate shared curation pruning to the FTS layer and harden hidden-path checks against symlink aliases and loops.
Remove invalid semantic keys and fail closed at query time so Hybrid search cannot resurrect pruned notes.
USearch 2.23 names the result-count parameter count; using k silently disabled semantic search behind the fallback path.
@kaykayyali
kaykayyali force-pushed the fix/issue-1-search-index-consistency branch from 7be9b13 to ccb5f8b Compare July 20, 2026 07:54
Hermes Agent added 2 commits July 20, 2026 08:16
Reuse the guarded GrepBackend for public recall so an empty index result cannot resurrect hidden or out-of-vault notes.
Filter indexed paths through the active-vault guard so a fresh scan marker cannot expose a hidden stale row through public recall.
@kaykayyali

Copy link
Copy Markdown
Author

Independent immutable-head review approved 8a38f4c53b338b2195588533cfb837afd76be0fa (tree deb3ac9e) with no blocking findings.

The review traced every public cb_recall path: direct FTS, Hybrid FTS + semantic fusion, empty/error grep fallbacks, final reads, and stale pruning. Isolated adversarial probes confirmed hidden, out-of-vault, and symlink-invalid paths are rejected.

Final deployed verification:

  • full test suite passes
  • changed-file Ruff passes
  • live stale-FTS + fresh-marker probe returns No notes found
  • vault acceptance: 456 active / 456 indexed / 456 unique / 0 issues

Ready for an upstream maintainer to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix stale search entries after vault curation

1 participant