From 10e6b86a5fa5f16e24d2637857006335ebe7c0ba Mon Sep 17 00:00:00 2001 From: chen Date: Wed, 22 Jul 2026 00:37:09 +0800 Subject: [PATCH 1/5] docs: search-index lock-acquisition order and threading model --- README.md | 12 ++++++++++ docs/architecture.md | 52 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 63 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 99b0007..0ed055e 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,18 @@ python app.py --base-dir /path/to/claude/projects > The server **refuses to start** if `--debug` is combined with a non-loopback `--host` (e.g. `0.0.0.0`). > That check runs only when you start the app with **`python app.py`** (not via `flask run` or other WSGI entrypoints). +#### Deployment — WSGI workers + +The FTS search index (`utils/search_index.py`) assumes **a single Python process** serves the app: + +- One daemon thread rebuilds the index and publishes it by atomically swapping `search_index.active` (see [Search index — locks and threading](docs/architecture.md#search-index-fts5--locks-and-threading)). +- Module-level locks and the in-memory usability cache are **per process**, not shared across workers. +- A cross-process advisory lock (`search_index.background.lock`) ensures at most **one** background refresher per index directory on a host, but **does not** coordinate search reads or caches across workers. + +**Do not run multiple Gunicorn/uWSGI workers** (`gunicorn -w 2`, etc.) against one operator-facing deployment. Requests will hit different processes with divergent cache state; only one worker runs background refresh; index freshness and `index_locked` / live-scan fallback become nondeterministic. Use **`--workers 1`** (or run `python app.py`) for supported behavior. For horizontal scale, run separate single-worker instances with disjoint `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR` and workspace paths — not multiple workers behind one load balancer on the same paths. + +Disable the index entirely with `CLAUDE_CODE_CHAT_BROWSER_NO_SEARCH_INDEX=1` if you only need live JSONL scan (slower, but avoids index files). + ### CLI Export ```bash diff --git a/docs/architecture.md b/docs/architecture.md index 5d37558..41149c8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -66,7 +66,7 @@ 3. User opens a project → `GET /api/projects//sessions` → full `parse_session()` per file, exclusion filter, summary rows. 4. User opens a session → `GET /api/sessions//` → full session JSON for the message panel. 5. Optional: `GET /api/sessions/.../stats` for sidebar metrics without loading all messages. -6. Search: `GET /api/search?q=...` scans all projects (brute force). +6. Search: `GET /api/search?q=...` queries the local FTS index when usable, with live JSONL scan fallback (see [Search index](#search-index-fts5--locks-and-threading)). 7. Export: `POST /api/export` or `GET /api/export/session/...` → Markdown/zip via exporters; state file updated on successful bulk export. ## Dispatch table @@ -143,6 +143,56 @@ No bundler step — modern browsers load modules directly. Frontend unit tests u - `integration-tests` — API integration subset + coverage artifact - `js-tests` — `npm ci` + vitest +## Search index (FTS5) — locks and threading + +`utils/search_index.py` maintains a local SQLite FTS5 index under `~/.claude-code-chat-browser/` (override with `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR`). Session JSONL on disk remains the source of truth; `GET /api/search` can fall back to a live scan when the index is missing, stale, or temporarily locked during rebuild. + +### Threading model + +| Role | Who | What | +|------|-----|------| +| **Readers** | Flask request threads (and synchronous callers such as `ensure_search_index` on the first request path) | Open the **active** index DB read-only via `_resolve_active_index_db_path()`; `query_index_hits` does not take any module `threading.Lock`. | +| **Writer** | One daemon thread (`search-index-refresh`, started from `start_search_index_background` in `app.py`) | Periodically calls `ensure_search_index` → `build_search_index` when the projects fingerprint changes. | +| **Publish** | Writer, at end of a successful rebuild | `_publish_active_index` writes a new `search_index..sqlite`, atomically replaces `search_index.active` (write temp + `Path.replace`), then prunes older DB files. Readers always open whichever DB name the pointer names — they see the **previous** or **new** index, never a half-written DB file. | + +Rebuild work happens on a **new** SQLite file; the active pointer switches only after the new file is committed. Concurrent FTS reads against the old file may fail with `sqlite3.OperationalError` (for example database locked); `query_index_hits` surfaces that as `index_locked=True` so `/api/search` can degrade to live scan instead of returning a silent empty success. + +### Lock inventory + +Four module-level synchronization primitives guard index lifecycle and caches (`utils/search_index.py`): + +| Primitive | Type | Guards | +|-----------|------|--------| +| `_index_lock` | `threading.Lock` | `_background_started` and startup/teardown of the background worker (`start_search_index_background`, `reset_background_for_tests`). | +| `_index_build_lock` | `threading.Lock` | Entire `build_search_index` body — at most one rebuild at a time **in this process**. | +| `_background_lock_fd` | OS advisory lock on `search_index.background.lock` | Cross-process **single background owner** for a shared index directory (`_try_acquire_cross_process_background_lock`). | +| `_usability_cache_lock` | `threading.Lock` | In-memory `_usability_cache` used by `index_is_usable` and cleared after rebuild (`_clear_usability_cache`). | + +**Usage map (current code):** + +- `_index_build_lock` — wraps `build_search_index` (serializes rebuild and fingerprint short-circuit reads). +- `_index_lock` — held briefly while deciding whether to start the daemon and while resetting test state; may call `_try_acquire_cross_process_background_lock` while held. +- `_background_lock_fd` — non-blocking exclusive flock/msvcrt lock; failure means another process already owns background refresh for this cache dir. +- `_usability_cache_lock` — `index_is_usable` and `_clear_usability_cache`. +- `_publish_active_index` — pointer swap + `_prune_stale_index_files` (no module lock; relies on atomic pointer replace and build serialization). + +### Lock-acquisition contract (must stay so) + +The four primitives are **not nested arbitrarily**. New code must preserve this contract; the planned Week 30 concurrency suite (`tests/test_search_index_concurrency.py`, PR after this doc) will assert it and fail if lock order regresses. + +1. **`_index_lock`** may be taken alone, or together with acquiring the advisory background lock during `start_search_index_background`. Do **not** call `build_search_index` while holding `_index_lock`. +2. **`_index_build_lock`** is taken only inside `build_search_index`. While held, code may open the active DB read-only and write a **new** sidecar SQLite file. The only permitted nested module lock is **`_usability_cache_lock`** via `_clear_usability_cache` at the end of a successful rebuild (`_index_build_lock` → `_usability_cache_lock`). +3. **`_usability_cache_lock`** must never be held while waiting on `_index_build_lock` or `_index_lock`. Typical pattern: short cache read/update in `index_is_usable` without holding the build lock. +4. **`_background_lock_fd`** is independent of the three `threading.Lock` instances except at startup: acquire the advisory lock only from `_try_acquire_cross_process_background_lock`, which is called under `_index_lock` once per process. + +**Forbidden:** holding `_usability_cache_lock` and then acquiring `_index_build_lock` or `_index_lock`; holding `_index_lock` across `build_search_index`; taking `_index_build_lock` from multiple threads without going through the existing `build_search_index` entry point. + +### Deployment assumptions + +The design targets **one WSGI worker process** (for example `python app.py` or `gunicorn --workers 1`). That gives one in-process writer thread, one advisory-lock owner, and consistent in-memory usability cache per operator session. + +See [Deployment — WSGI workers](../README.md#deployment-wsgi-workers) in the README for multi-worker warnings. + ## What this codebase is not - **Not multi-user** — no authn/authz; single local operator. From 33d5ea0802d54f0b85fe6931fd65a02e9197fa8a Mon Sep 17 00:00:00 2001 From: chen Date: Wed, 22 Jul 2026 00:39:38 +0800 Subject: [PATCH 2/5] docs: clarify search-index publish and index_locked semantics --- docs/architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 41149c8..9373d68 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -145,7 +145,7 @@ No bundler step — modern browsers load modules directly. Frontend unit tests u ## Search index (FTS5) — locks and threading -`utils/search_index.py` maintains a local SQLite FTS5 index under `~/.claude-code-chat-browser/` (override with `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR`). Session JSONL on disk remains the source of truth; `GET /api/search` can fall back to a live scan when the index is missing, stale, or temporarily locked during rebuild. +`utils/search_index.py` maintains a local SQLite FTS5 index under `~/.claude-code-chat-browser/` (override with `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR`). Session JSONL on disk remains the source of truth; `GET /api/search` can fall back to a live scan when the index is missing, stale, or when a query returns `index_locked` after `sqlite3.OperationalError`. ### Threading model @@ -153,9 +153,9 @@ No bundler step — modern browsers load modules directly. Frontend unit tests u |------|-----|------| | **Readers** | Flask request threads (and synchronous callers such as `ensure_search_index` on the first request path) | Open the **active** index DB read-only via `_resolve_active_index_db_path()`; `query_index_hits` does not take any module `threading.Lock`. | | **Writer** | One daemon thread (`search-index-refresh`, started from `start_search_index_background` in `app.py`) | Periodically calls `ensure_search_index` → `build_search_index` when the projects fingerprint changes. | -| **Publish** | Writer, at end of a successful rebuild | `_publish_active_index` writes a new `search_index..sqlite`, atomically replaces `search_index.active` (write temp + `Path.replace`), then prunes older DB files. Readers always open whichever DB name the pointer names — they see the **previous** or **new** index, never a half-written DB file. | +| **Publish** | Writer, at end of a successful rebuild | `_publish_active_index` writes the new DB basename to `search_index.active.tmp`, then tries `Path.replace` onto `search_index.active`. On `OSError`, it falls back to `pointer.write_text` (not rename-atomic on every platform). Readers resolve the active **SQLite file** by name from the pointer; sidecar DB files are fully committed before publish. After a successful `replace`, readers see the previous or new complete DB, not a half-written SQLite file. The direct-write fallback can briefly expose a partially written pointer file, so treat `replace` as the normal path. | -Rebuild work happens on a **new** SQLite file; the active pointer switches only after the new file is committed. Concurrent FTS reads against the old file may fail with `sqlite3.OperationalError` (for example database locked); `query_index_hits` surfaces that as `index_locked=True` so `/api/search` can degrade to live scan instead of returning a silent empty success. +Rebuild work happens on a **new** SQLite file while the prior active DB file stays on disk until publish; that isolation does **not** by itself lock the old database or force readers onto the in-progress file. Separately, `query_index_hits` maps `sqlite3.OperationalError` from the FTS query (for example `database is locked` on the DB it opened) to `index_locked=True` so `/api/search` can degrade to live scan — that flag is a **query-time classification** for fallback, not proof that a rebuild is in progress. Concurrency tests should assert both behaviors without conflating them. ### Lock inventory @@ -174,7 +174,7 @@ Four module-level synchronization primitives guard index lifecycle and caches (` - `_index_lock` — held briefly while deciding whether to start the daemon and while resetting test state; may call `_try_acquire_cross_process_background_lock` while held. - `_background_lock_fd` — non-blocking exclusive flock/msvcrt lock; failure means another process already owns background refresh for this cache dir. - `_usability_cache_lock` — `index_is_usable` and `_clear_usability_cache`. -- `_publish_active_index` — pointer swap + `_prune_stale_index_files` (no module lock; relies on atomic pointer replace and build serialization). +- `_publish_active_index` — pointer update (`replace` with `write_text` fallback) + `_prune_stale_index_files` (no module lock; build serialization via `_index_build_lock`). ### Lock-acquisition contract (must stay so) From 6f5cd4e097211adf64eeb7259a014dbbea1a5131 Mon Sep 17 00:00:00 2001 From: yu-med Date: Wed, 22 Jul 2026 03:13:34 +0800 Subject: [PATCH 3/5] docs: clarify NO_SEARCH_INDEX leaves index files on disk --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0ed055e..6b895b2 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ The FTS search index (`utils/search_index.py`) assumes **a single Python process **Do not run multiple Gunicorn/uWSGI workers** (`gunicorn -w 2`, etc.) against one operator-facing deployment. Requests will hit different processes with divergent cache state; only one worker runs background refresh; index freshness and `index_locked` / live-scan fallback become nondeterministic. Use **`--workers 1`** (or run `python app.py`) for supported behavior. For horizontal scale, run separate single-worker instances with disjoint `CLAUDE_CODE_CHAT_BROWSER_SEARCH_INDEX_DIR` and workspace paths — not multiple workers behind one load balancer on the same paths. -Disable the index entirely with `CLAUDE_CODE_CHAT_BROWSER_NO_SEARCH_INDEX=1` if you only need live JSONL scan (slower, but avoids index files). +Set `CLAUDE_CODE_CHAT_BROWSER_NO_SEARCH_INDEX=1` to skip building and querying the FTS index (live JSONL scan only; slower). Existing files under the search index directory are left on disk. ### CLI Export From e8d8b207c5b52b1b4912a8369e64a2db53238926 Mon Sep 17 00:00:00 2001 From: chen Date: Thu, 23 Jul 2026 01:10:44 +0800 Subject: [PATCH 4/5] Address timon's feedback --- README.md | 2 +- docs/architecture.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6b895b2..de9455a 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ python app.py --base-dir /path/to/claude/projects > The server **refuses to start** if `--debug` is combined with a non-loopback `--host` (e.g. `0.0.0.0`). > That check runs only when you start the app with **`python app.py`** (not via `flask run` or other WSGI entrypoints). -#### Deployment — WSGI workers +#### Deployment: WSGI workers The FTS search index (`utils/search_index.py`) assumes **a single Python process** serves the app: diff --git a/docs/architecture.md b/docs/architecture.md index 9373d68..21a2828 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -191,7 +191,7 @@ The four primitives are **not nested arbitrarily**. New code must preserve this The design targets **one WSGI worker process** (for example `python app.py` or `gunicorn --workers 1`). That gives one in-process writer thread, one advisory-lock owner, and consistent in-memory usability cache per operator session. -See [Deployment — WSGI workers](../README.md#deployment-wsgi-workers) in the README for multi-worker warnings. +See [Deployment: WSGI workers](../README.md#deployment-wsgi-workers) in the README for multi-worker warnings. ## What this codebase is not From 4fe10b857c241713470239ae35acdc82ce28c187 Mon Sep 17 00:00:00 2001 From: chen Date: Thu, 23 Jul 2026 02:53:49 +0800 Subject: [PATCH 5/5] Address Will's feedback --- docs/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture.md b/docs/architecture.md index 21a2828..fbe1c72 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -151,7 +151,7 @@ No bundler step — modern browsers load modules directly. Frontend unit tests u | Role | Who | What | |------|-----|------| -| **Readers** | Flask request threads (and synchronous callers such as `ensure_search_index` on the first request path) | Open the **active** index DB read-only via `_resolve_active_index_db_path()`; `query_index_hits` does not take any module `threading.Lock`. | +| **Readers** | Flask request threads (`GET /api/search` via `query_index_hits`; `index_is_usable` for whether to use the index) | Open the **active** index DB read-only via `_resolve_active_index_db_path()`; `query_index_hits` does not take any module `threading.Lock`. No request handler calls `ensure_search_index` or `build_search_index`. | | **Writer** | One daemon thread (`search-index-refresh`, started from `start_search_index_background` in `app.py`) | Periodically calls `ensure_search_index` → `build_search_index` when the projects fingerprint changes. | | **Publish** | Writer, at end of a successful rebuild | `_publish_active_index` writes the new DB basename to `search_index.active.tmp`, then tries `Path.replace` onto `search_index.active`. On `OSError`, it falls back to `pointer.write_text` (not rename-atomic on every platform). Readers resolve the active **SQLite file** by name from the pointer; sidecar DB files are fully committed before publish. After a successful `replace`, readers see the previous or new complete DB, not a half-written SQLite file. The direct-write fallback can briefly expose a partially written pointer file, so treat `replace` as the normal path. |