diff --git a/README.md b/README.md index 99b0007..de9455a 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,18 @@ python app.py --base-dir /path/to/claude/projects > The server **refuses to start** if `--debug` is combined with a non-loopback `--host` (e.g. `0.0.0.0`). > That check runs only when you start the app with **`python app.py`** (not via `flask run` or other WSGI entrypoints). +#### Deployment: WSGI workers + +The FTS search index (`utils/search_index.py`) assumes **a single Python process** serves the app: + +- One daemon thread rebuilds the index and publishes it by atomically swapping `search_index.active` (see [Search index — locks and threading](docs/architecture.md#search-index-fts5--locks-and-threading)). +- Module-level locks and the in-memory usability cache are **per process**, not shared across workers. +- A cross-process advisory lock (`search_index.background.lock`) ensures at most **one** background refresher per index directory on a host, but **does not** coordinate search reads or caches across workers. + +**Do not run multiple Gunicorn/uWSGI workers** (`gunicorn -w 2`, etc.) against one operator-facing deployment. Requests will hit different processes with divergent cache state; only one worker runs background refresh; index freshness and `index_locked` / live-scan fallback become nondeterministic. Use **`--workers 1`** (or run `python app.py`) for supported behavior. For horizontal scale, run separate single-worker instances with disjoint `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR` and workspace paths — not multiple workers behind one load balancer on the same paths. + +Set `CLAUDE_CODE_CHAT_BROWSER_NO_SEARCH_INDEX=1` to skip building and querying the FTS index (live JSONL scan only; slower). Existing files under the search index directory are left on disk. + ### CLI Export ```bash diff --git a/docs/architecture.md b/docs/architecture.md index 5d37558..fbe1c72 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -66,7 +66,7 @@ 3. User opens a project → `GET /api/projects//sessions` → full `parse_session()` per file, exclusion filter, summary rows. 4. User opens a session → `GET /api/sessions//` → full session JSON for the message panel. 5. Optional: `GET /api/sessions/.../stats` for sidebar metrics without loading all messages. -6. Search: `GET /api/search?q=...` scans all projects (brute force). +6. Search: `GET /api/search?q=...` queries the local FTS index when usable, with live JSONL scan fallback (see [Search index](#search-index-fts5--locks-and-threading)). 7. Export: `POST /api/export` or `GET /api/export/session/...` → Markdown/zip via exporters; state file updated on successful bulk export. ## Dispatch table @@ -143,6 +143,56 @@ No bundler step — modern browsers load modules directly. Frontend unit tests u - `integration-tests` — API integration subset + coverage artifact - `js-tests` — `npm ci` + vitest +## Search index (FTS5) — locks and threading + +`utils/search_index.py` maintains a local SQLite FTS5 index under `~/.claude-code-chat-browser/` (override with `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR`). Session JSONL on disk remains the source of truth; `GET /api/search` can fall back to a live scan when the index is missing, stale, or when a query returns `index_locked` after `sqlite3.OperationalError`. + +### Threading model + +| Role | Who | What | +|------|-----|------| +| **Readers** | Flask request threads (`GET /api/search` via `query_index_hits`; `index_is_usable` for whether to use the index) | Open the **active** index DB read-only via `_resolve_active_index_db_path()`; `query_index_hits` does not take any module `threading.Lock`. No request handler calls `ensure_search_index` or `build_search_index`. | +| **Writer** | One daemon thread (`search-index-refresh`, started from `start_search_index_background` in `app.py`) | Periodically calls `ensure_search_index` → `build_search_index` when the projects fingerprint changes. | +| **Publish** | Writer, at end of a successful rebuild | `_publish_active_index` writes the new DB basename to `search_index.active.tmp`, then tries `Path.replace` onto `search_index.active`. On `OSError`, it falls back to `pointer.write_text` (not rename-atomic on every platform). Readers resolve the active **SQLite file** by name from the pointer; sidecar DB files are fully committed before publish. After a successful `replace`, readers see the previous or new complete DB, not a half-written SQLite file. The direct-write fallback can briefly expose a partially written pointer file, so treat `replace` as the normal path. | + +Rebuild work happens on a **new** SQLite file while the prior active DB file stays on disk until publish; that isolation does **not** by itself lock the old database or force readers onto the in-progress file. Separately, `query_index_hits` maps `sqlite3.OperationalError` from the FTS query (for example `database is locked` on the DB it opened) to `index_locked=True` so `/api/search` can degrade to live scan — that flag is a **query-time classification** for fallback, not proof that a rebuild is in progress. Concurrency tests should assert both behaviors without conflating them. + +### Lock inventory + +Four module-level synchronization primitives guard index lifecycle and caches (`utils/search_index.py`): + +| Primitive | Type | Guards | +|-----------|------|--------| +| `_index_lock` | `threading.Lock` | `_background_started` and startup/teardown of the background worker (`start_search_index_background`, `reset_background_for_tests`). | +| `_index_build_lock` | `threading.Lock` | Entire `build_search_index` body — at most one rebuild at a time **in this process**. | +| `_background_lock_fd` | OS advisory lock on `search_index.background.lock` | Cross-process **single background owner** for a shared index directory (`_try_acquire_cross_process_background_lock`). | +| `_usability_cache_lock` | `threading.Lock` | In-memory `_usability_cache` used by `index_is_usable` and cleared after rebuild (`_clear_usability_cache`). | + +**Usage map (current code):** + +- `_index_build_lock` — wraps `build_search_index` (serializes rebuild and fingerprint short-circuit reads). +- `_index_lock` — held briefly while deciding whether to start the daemon and while resetting test state; may call `_try_acquire_cross_process_background_lock` while held. +- `_background_lock_fd` — non-blocking exclusive flock/msvcrt lock; failure means another process already owns background refresh for this cache dir. +- `_usability_cache_lock` — `index_is_usable` and `_clear_usability_cache`. +- `_publish_active_index` — pointer update (`replace` with `write_text` fallback) + `_prune_stale_index_files` (no module lock; build serialization via `_index_build_lock`). + +### Lock-acquisition contract (must stay so) + +The four primitives are **not nested arbitrarily**. New code must preserve this contract; the planned Week 30 concurrency suite (`tests/test_search_index_concurrency.py`, PR after this doc) will assert it and fail if lock order regresses. + +1. **`_index_lock`** may be taken alone, or together with acquiring the advisory background lock during `start_search_index_background`. Do **not** call `build_search_index` while holding `_index_lock`. +2. **`_index_build_lock`** is taken only inside `build_search_index`. While held, code may open the active DB read-only and write a **new** sidecar SQLite file. The only permitted nested module lock is **`_usability_cache_lock`** via `_clear_usability_cache` at the end of a successful rebuild (`_index_build_lock` → `_usability_cache_lock`). +3. **`_usability_cache_lock`** must never be held while waiting on `_index_build_lock` or `_index_lock`. Typical pattern: short cache read/update in `index_is_usable` without holding the build lock. +4. **`_background_lock_fd`** is independent of the three `threading.Lock` instances except at startup: acquire the advisory lock only from `_try_acquire_cross_process_background_lock`, which is called under `_index_lock` once per process. + +**Forbidden:** holding `_usability_cache_lock` and then acquiring `_index_build_lock` or `_index_lock`; holding `_index_lock` across `build_search_index`; taking `_index_build_lock` from multiple threads without going through the existing `build_search_index` entry point. + +### Deployment assumptions + +The design targets **one WSGI worker process** (for example `python app.py` or `gunicorn --workers 1`). That gives one in-process writer thread, one advisory-lock owner, and consistent in-memory usability cache per operator session. + +See [Deployment: WSGI workers](../README.md#deployment-wsgi-workers) in the README for multi-worker warnings. + ## What this codebase is not - **Not multi-user** — no authn/authz; single local operator.