fix: rebuild index when mode adds capabilities

[astandrik]

Summary

Fixes #1273
Related: #563

• persist the effective index capability level in the Project node as index_mode
• force a full rebuild when the requested mode needs capabilities missing from the stored index
• on downgrades, keep requested discovery/reporting scope separate from the stronger stored extraction scope
• conservatively rebuild legacy, missing, malformed, or non-exact mode metadata
• export persistent artifacts exactly once, after atomic database publication
• keep the implementation focused by sharing manifest indexing and repeated test setup

Why

Incremental routing originally compared stored file hashes and the discovered-file threshold, but not the capability level of the stored index. An unchanged fast -> moderate request could therefore take the no-op route without creating the stronger similarity data.

The inverse transition also needs two distinct concepts. The caller's requested mode determines which files are discovered and reported, while the stored stronger mode determines which extraction passes and capabilities must be preserved. Promoting one mutable mode before discovery widened a full -> fast request back to full discovery. If a stored-only file has changed or disappeared, the pipeline now performs an effective-scope full rediscovery and rebuild so stale stronger-mode graph data cannot survive.

Exact no-op downgrade eligibility compares the requested manifest subset with the stored manifest. Stored extras are accepted only when they remain safe relative, regular, non-symlink files. Duplicate, malformed, or allocation-incomplete comparisons fail closed to a rebuild.

The no-op persistence path also exported the artifact inside staging and then again after publication. Export now has one boundary: after the final database rename.

Mode contract

Stored index	Requested mode	Behavior
weaker	stronger	full rebuild and record the stronger effective mode
same	same	existing incremental/no-op route
stronger	weaker, requested subset unchanged	requested discovery/reporting with stored stronger extraction capabilities; no database rewrite
stronger	weaker, stored-only input changed or missing	effective-scope full rebuild, while preserving requested discovery reporting
missing / unknown / malformed / non-exact	any	one conservative full rebuild

The public MCP API and SQLite schema are unchanged. Existing threshold routing and ADR preservation behavior remain unchanged.

Regression coverage

• unchanged fast -> moderate rebuilds, creates similarity data, and records index_mode=moderate
• changed moderate -> fast preserves SIMILAR_TO and the stronger stored mode
• changed full -> moderate preserves full-only Macro nodes and the stronger stored mode
• full -> fast no-op discovers/reports only the requested scope while retaining stronger graph capabilities
• deleting a full-only file during a fast request forces an effective-full rebuild and removes its stale nodes
• missing legacy metadata and escaped-NUL/non-exact metadata rebuild once and normalize safely
• explicit and implicit no-op persistence each export exactly once, after the real database rename
• persistence failure still propagates as a pipeline failure

Verification

Exact branch head: 3959fdf5 on base 49d928be.

Fail-before on the current-main rebase:

• requested/effective discovery regression: 25 existing focused tests passed; the new contract test failed on the widened discovery report
• artifact publication regression: 25 existing focused tests passed; the new observer independently measured two exports and one export before publication

Pass-after on the exact published tree:

• focused semantic-manifest suite: 27 passed
• broad pipeline suite: 257 passed (one redundant test was removed after its contract was fully subsumed by a stricter test)
• complete plain local matrix: 7500 passed, 1 failed, 5 skipped across 138 suites; the sole daemon-runtime fixture failure reproduces unchanged with the previous published-head runner and is therefore unrelated to this cleanup
• production O2 build, parent/worker watchdogs, worker error response, security string allowlist, destructive-ordering gate, lint-no-suppress, lint-format, no-forbidden-skips, and git diff --check: passed
• Desloppify rescan found no new detector regressions in the changed files; the next reported cleanup target is outside this PR

Exact-head GitHub CI is running. Its clean runners are authoritative for sanitizer and cppcheck coverage: local macOS ASan blocked inside runtime initialization before main, and local cppcheck is not installed.

[Copilot]

Pull request overview

This PR fixes incremental routing decisions across index mode transitions by persisting the effective index capability level in Project metadata (properties.index_mode), forcing a full rebuild when the requested mode requires capabilities not present in the stored index, and preserving stronger stored capabilities during downgrades (while still honoring the requested mode for discovery/exclusions). It also ensures persistent artifacts are refreshed after reindexing and adds regression coverage for mode upgrades/downgrades and malformed/legacy metadata handling.

Changes:

• Persist effective capability mode in the Project node (properties.index_mode) and use exact JSON string comparisons when reading it.
• Route mode upgrades to full rebuilds; keep stored effective mode for incremental changed-file re-extraction on downgrades.
• Refresh existing artifacts after reindexing (FAST quality unless persistence is explicitly requested) and add end-to-end tests for these transitions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/test_pipeline.c	Adds regression tests covering upgrade/downgrade routing, capability preservation, malformed metadata, and artifact refresh behavior.
src/store/store.h	Clarifies that index_mode in coverage metadata is the requested discovery mode; effective capability mode is stored on the Project node.
src/pipeline/pipeline.c	Persists Project index_mode, reads stored mode safely via yyjson, routes upgrades to rebuilds, and adjusts post-publish artifact export behavior.
src/pipeline/pipeline_internal.h	Exposes cbm_pipeline_mode_name for consistent internal serialization and updates incremental API contract docs/signature.
src/pipeline/pipeline_incremental.c	Accepts an effective_mode for changed-file re-extraction, preserves macro extraction capability on downgrades, and propagates persistence failures when required.
src/mcp/mcp.c	Updates MCP tool schema description to document artifact refresh behavior when an artifact already exists.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DeusData]

Thank you for the contribution and for covering both index mode upgrades and downgrades, including malformed metadata. This is now triaged as a high-priority graph correctness bug for 0.9.1-rc. Our community PR queue is currently quite full, so it may take a little time before we can complete the review and, if approved, merge it. We are doing our best to support community contributions and will return with code-grounded feedback as capacity opens.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/pipeline/pipeline.c:1527

• Assigning the stored mode to p->mode here also changes discovery, because cbm_pipeline_run_staged later passes p->mode to cbm_discover_ex2. A full → fast downgrade therefore still scans full-only directories/files instead of using the requested discovery scope, contradicting the stated requested/effective-mode separation and defeating the fast-mode workaround for large repositories. Keep the requested discovery mode separate and use the stored mode only for effective extraction/post-passes and persisted capability metadata.

        p->mode = stored_mode;

src/pipeline/pipeline_incremental.c:2426

• This exports the artifact during the inner staged run, but the successful return is then marked incremental and export_after_publish exports it again (BEST quality for explicit persistence, FAST for an existing artifact). Thus every exact no-op that needs an artifact performs compression/VACUUM twice, with the first export occurring before final publication. Let the existing post-publication path perform the single export.

            return cbm_pipeline_refresh_artifact(p, db_path);