feat: jrag watch — warm, self-refreshing index daemon#424
Merged
Conversation
This was referenced Jul 11, 2026
HumanBean17
force-pushed
the
worktree-watch-mode
branch
from
248549a to
3325344
Compare
…warm model reuse Three heavy-gated characterization tests (JAVA_CODEBASE_RAG_RUN_HEAVY=1) that lock in the concurrency pillars the `jrag watch` daemon assumes (design §4.7/§9): - test_lance_atomic_versions: a fresh lancedb.connect after a cocoindex catch-up sees a strictly newer atomic version with the new file's chunks fully present; a handle opened before the write keeps a consistent (pre-write) snapshot. - test_graph_cow_snapshot (load-bearing): a read-only reader on a file COPY of code_graph.lbug is isolated from run_incremental_graph writing the original (returncode 0, no single-writer conflict); the sidecar keeps the pre-write node count while the original grows. - test_warm_model_reuse: _get_sentence_transformer returns one process-wide singleton, reused across calls and by search_v2. Key characterization findings recorded in docstrings: - Lance version delta per catch-up is ~+4 (catch-up + serialized optimize), not +1; the pillar only needs monotonic atomic growth. - LadybugGraph.meta()["counts"] is NOT a reliable node count across an incremental rebuild (phantom resolution offsets new nodes: summed total 88->88 while the real :Symbol count grew 88->93); tests use a direct MATCH query instead. - LadybugGraph.reset_for_path does not exist yet (Task 6 adds it); singleton is reset manually as the existing heavy e2e tests do. Co-Authored-By: Claude <noreply@anthropic.com>
ProjectLock: exclusive per-project lock keyed on paths.pid_path(index_dir). acquire opens the pid file r+/w+, takes flock(LOCK_EX|LOCK_NB), translates BlockingIOError to LockHeldError(stored_pid, path), writes our pid on success. release closes the handle (drops the flock) and unlinks the pid file iff it still names us. is_holder + classmethod read_holder (stale-pid aware). WatchUnsupportedPlatform raised on construction when fcntl is unavailable. Co-Authored-By: Claude <noreply@anthropic.com>
…serving) Extract payload-returning cores for the callers/callees/flow/search/find/inspect read commands into a new read_payloads.py, so the upcoming jrag watch daemon can serve them over a socket without touching rendering. Each handler becomes 'payload = <cmd>_payload(args, cfg, graph); render(payload, args)' — rendering byte-identical to before. - callers_payload/callees_payload/flow_payload carry the full assembly verbatim, including the three ad-hoc folds: callers EXPOSES-inbound, callees CLIENT-role HTTP_CALLS, and flow inbound/outbound merge. - search_payload/find_payload/inspect_payload wrap the *_v2 (or find_by_name_or_fqn) backend call. - Resolve/guard/validation failures raise PayloadError(env, rc); the handler renders the carried Envelope, so error output is byte-identical too. New golden-output equivalence tests (tests/jrag/test_read_payloads.py + tests/jrag/golden/) prove byte-identity for 9/10 commands; inspect is pinned by canonicalized JSON because its edge_summary dict key order is pre-existing non-deterministic in describe_v2 (verified on unmodified HEAD). All three folds are exercised by the golden fixtures. Co-Authored-By: Claude <noreply@anthropic.com>
WarmResources holds the warm embedding model and read-only LadybugGraph reader for the jrag watch daemon, and owns the copy-on-write graph snapshot lifecycle: begin copies code_graph.lbug to a .lbug.snapshot sidecar and serves reads from it while a reindex subprocess writes the original; commit drops the sidecar reader and reopens the updated original. Adds LadybugGraph.reset_for_path classmethod (clears the cached singleton on None or matching path) so WarmResources can force the next get() to reopen. Co-Authored-By: Claude <noreply@anthropic.com>
…shot cleanup Co-Authored-By: Claude <noreply@anthropic.com>
WatchServer binds an AF_UNIX socket (chmod 0o600), runs an inline accept loop over newline-delimited JSON requests (Task 4 protocol), dispatches each to its Task-5 payload core, and writes the encoded response. dispatch maps errors per the pinned IPC contract (_IndexStale -> stale_index, ProtocolMismatch/ValueError -> bad_args, any other Exception incl. PayloadError -> backend_error, unknown cmd -> unknown_command). serialize() uses model_dump(mode="json") + recursion (the cold path's --format json Envelope emit is tangled with handler-specific projection; the daemon ships the serialized payload and the client renders). 15 transport/dispatch tests (stub warm + stub PAYLOAD_FNS). Co-Authored-By: Claude <noreply@anthropic.com>
watch/client.py implements the try-daemon-then-cold-fallback seam:
is_daemon_alive, request (AF_UNIX, 2s timeout, ProtocolMismatch->cold),
DaemonUnavailable/DaemonError, and get_payload (the single seam each read
handler calls). On daemon success the payload is reconstructed into the same
object the cold core returns (model_validate for the pydantic *Output models;
SymbolHit(**row) rebuild for find query-mode rows); on DaemonUnavailable /
DaemonError it cold-falls-back to the identical <cmd>_payload core via
_load_graph (a cache hit on the already-loaded singleton).
The 6 jrag read handlers (search/find*/inspect/callers/callees/flow) acquire
their payload via get_payload("<cmd>", vars(args), cfg, cold_core=<cmd>_payload);
only the payload-acquisition line + a lazy import per handler changed -- render
and rc logic are untouched, so the cold path stays byte-identical (Task 5 golden
suite 15/15 unchanged).
Co-Authored-By: Claude <noreply@anthropic.com>
SourceWatcher observes the source tree via a watchdog backend (native Observer with a polling fallback for 'auto'; explicit 'watchdog'/'polling') and, after a single-thread debounce window, runs the minimal reindex for the changed types: java -> vectors THEN graph (graph subprocess under WarmResources' COW snapshot, begin before / commit always in a finally); sql/yaml -> vectors only. Status flows only through on_event callbacks. Adds watchdog>=6,<7. Co-Authored-By: Claude <noreply@anthropic.com>
…iring + pair begin/commit
…val_ms)
Parse the new optional `watch:` block in resolve_operator_config into three
ResolvedOperatorConfig knobs (debounce_ms, backend, poll_interval_ms) plus
companion *_source fields. Precedence: CLI flag > YAML > built-in default (no
env vars introduced). Inline floors/validation: debounce_ms >= 100,
backend in {auto,watchdog,polling}, poll_interval_ms >= 200 — out-of-range
values fall back to the default with a one-line stderr warning.
_pick_int gains an optional cli_val kwarg (default None) mirroring _pick_str,
so the CLI-override tier flows through the helper; existing callers untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 11 (capstone): assemble the watch components into a daemon process and expose the `jrag watch` subcommand with its four lifecycle verbs. - watch/daemon.py `WatchDaemon`: acquires the project lock, eagerly warms the model (fail-fast), installs SIGINT/SIGTERM handlers, starts server+watcher, writes a state file for cross-process status, renders a rich Live panel (TTY) or a one-line startup (non-TTY), and tears down watcher->server->lock on stop. The serving-path shutdown ends with os._exit(0) (mirrors _console_script_main) to dodge the pyarrow/lance worker-thread SIGABRT once a search has loaded lancedb in-process. Early-failure paths (lock held / model load) return 2 normally (no lance threads yet). - jrag.py: `watch` subparser (parents=_core_parser, no auto-scope) with --detach/--stop/--status/--debounce-ms/--backend; `_cmd_watch` + helpers. --status/--stop read ProjectLock.read_holder only (never acquire the lock). --detach spawns `python -m java_codebase_rag.jrag watch` detached (setsid + redirected stdio) and waits for is_daemon_alive. `--debounce-ms`/`--backend` plumbed through _resolve_cfg. - tests/watch/test_daemon.py: 10 lifecycle tests (stubbed warm/watcher in a subprocess so the real os._exit shutdown runs outside pytest) + a heavy-gated golden IPC test proving `jrag search` over the socket is BYTE-IDENTICAL to the cold path, with a _load_graph spy proving the hot path was actually served. Two integration fixes found + resolved by the tests: 1. Stale socket before bind: the daemon acquires the lock BEFORE server.start, so the server's `read_holder is None` stale-socket guard is inert (it sees its own pid). The daemon now clears its own stale socket before bind (it holds the exclusive lock, so it owns the path). 2. `python jrag.py` as a script shadows stdlib `ast` with java_codebase_rag.ast (inspect -> ast.NodeVisitor crashes). Detach now invokes the daemon via `python -m java_codebase_rag.jrag`. queries_served left at 0 for v1 (wiring a count out of WatchServer is out of this task's commit surface and the brief marks it optional). Co-Authored-By: Claude <noreply@anthropic.com>
…nel + detach fail-fast
…TION/ARCHITECTURE/DESIGN + skill tip - JAVA-CODEBASE-RAG-CLI.md: new `### jrag watch` subsection (four modes, --debounce-ms/--backend flags, cold-fallback guarantee, Unix-only note) - CONFIGURATION.md: `watch:` YAML block (debounce_ms/backend/poll_interval_ms with defaults + floors) in the §2 reference - ARCHITECTURE.md: watch daemon in the read path + a watch-path note (warm resources, subprocessed reindex, graph COW snapshot, Lance atomic versions, pidfile/flock); watch/ in the layout table + extension points - DESIGN.md: `jrag watch` as a first-class surface + the watch non-goals (Unix-only; not a boot service; not multi-project; cold path unchanged; not for the MCP surface) - skills/explore-codebase-cli/SKILL.md + agents/explorer-rag-cli.md: tip to run `jrag watch` once per session for fast, fresh queries - tests/test_docs_watch.py: docs-accuracy test pinning the operator surface Co-Authored-By: Claude <noreply@anthropic.com>
…ot watchdog-only)
…for all read cmds + doc limitations Co-Authored-By: Claude <noreply@anthropic.com>
Rebase onto master pulled in PR #413 (find --fuzzy exact→prefix→contains), which added ``matched_mode``/``identifier_matched`` to the find query-mode payload (read_payloads.find_payload). The server serializes the whole dict, but the watch IPC client reconstructs find query mode with a hardcoded key set that predates --fuzzy — so the daemon path would KeyError in ``_render_find_query``. Thread the two keys through the reconstruction and the client unit-test mock. Cold path was already correct; proven by the find_query golden byte-identity test over the real socket. Co-Authored-By: Claude <noreply@anthropic.com>
…goldens Rebase onto master (#422 --count/--exists/--fields; #430 find/rendering QA) left the 4 refactored read handlers (inspect/callers/callees/flow) routing their resolve-miss through ``print(render(pe.env)); return pe.rc``, bypassing master's new ``_emit`` funnel — so ``<cmd> Missing --exists`` returned rc 0 instead of 2 and ignored the flag. Switch them to ``_emit(pe.env, args)``, mirroring master's ``_resolve_traversal_node`` / inspect resolve-miss path, and drop the now-unused ``render`` imports. find (kind-guard) and search (NodeFilter validation) keep ``print(render)`` — master intentionally bypasses ``_emit`` for usage errors so the actionable message isn't hidden by a count/exists shape. Master #430 also enriched route/symbol node rendering (name/method/path/ framework/resolved/file on routes; name/file on symbols), so regenerate the 3 affected byte-identity goldens (callers_route, flow_merge, find_filter); the other 8 goldens are byte-identical. Cold==daemon byte-identity re-verified (tests/watch 116, incl. all 6 IPC goldens). Co-Authored-By: Claude <noreply@anthropic.com>
HumanBean17
force-pushed
the
worktree-watch-mode
branch
from
3325344 to
688c3ed
Compare
…dows
Three CI failure classes after the rebases onto master:
1. install_data out of sync (all platforms). The deployed skill/agent copies
(src/.../install_data/{skills,agents}/explorer-rag-cli) lacked the
``jrag watch`` tip added to the dev source — test_install_data_sync caught
the drift. Re-run scripts/sync_agent_artifacts.py (never hand-patch
deployed copies).
2. test_golden_output_byte_identical cross-platform fragility (callees_symbol /
flow_merge on linux+windows; search on intel-mac). The e2e golden builds a
real index and byte-compares CLI stdout captured on one platform, but
graph-traversal edge order follows indexing file-traversal order (OS-
dependent) and the search golden relies on the Lance table being absent
(index-dir nondeterminism). Gate it behind JAVA_CODEBASE_RAG_RUN_HEAVY=1
(same idiom as test_lancedb_e2e) — it's a LOCAL regression guard, not a
portable CI check. CI still covers the cold read path via tests/package/*;
warm==cold byte-identity is covered (heavy) by tests/watch/test_daemon.py.
The in-process Layer 2 tests stay in CI (only the e2e golden is gated).
3. tests/watch on Windows. jrag watch is Unix-only by design (AF_UNIX sockets
+ fcntl flock; watch/lock.py raises WatchUnsupportedPlatform). The tests
exercise those primitives directly, so add tests/watch/conftest.py that
skips the whole tree when fcntl is unavailable (the product's own gate).
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
jrag watch— a single long-running process per project that (a) keeps the index fresh on file change and (b) serves everyjragread command warm over a Unix socket, eliminating the per-call cold start (torch + embedding-model load + Lance connect). With no daemon running, every read command behaves byte-identically to today — the daemon is a pure accelerator + freshness layer, never a dependency.Solves the two coupled problems (stale-index friction + per-search latency) with one warm process.
Surface
jrag watch(foreground),--detach,--stop,--status,--debounce-ms,--backend {auto,watchdog,polling}.search/find/inspect/callers/callees/flow) use the daemon if up, else cold-fall-back (unchanged output).watch:YAML block:debounce_ms(1500),backend(auto),poll_interval_ms(2000).Design highlights
_st_model,LadybugGraph) served to the CLI.build_ast_graph.py --incremental).ladybughas no transactions, single-writer).flock) — the codebase had none. Unix-only (macOS/Linux).Tests
tests/watch116,tests/jrag15, config 65, docs 9._load_graphspy proves the daemon served, not cold-fall-back).Known limitations / follow-ups (spec §10)
JAVA_CODEBASE_RAG_RUN_HEAVY=0) — the 9 load-bearing tests run only with the env var; consider a heavy CI job/nightly._load_graphmay transiently error during a graph reindex (kùzu single-writer); v1-acceptable, fix path documented.Heads-up: pre-existing full-suite failures (unrelated to this branch)
The full heavy suite has 4 failures that are not regressions from this work:
test_lancedb_ignore_file_reduces_indexed_java_files—assert 2 > 2in the cocoindex/ignore flow (this branch does not touchpath_filtering/index/cocoindex).jraggoldens + one mcp-hints test) that pass in isolation and persist with--ignore=tests/watch.Files
New
java_codebase_rag/watch/subpackage (lock, paths, protocol, warm, server, client, watcher, daemon) +read_payloads.py; modifiedjrag.py,config.py,ladybug_queries.py(additivereset_for_path),pyproject.toml(+watchdog); docs (CLI / CONFIGURATION / ARCHITECTURE / DESIGN) +explorer-rag-cliskill/agent tip. Design spec committed atdocs/specs/2026-07-11-watch-mode-design.md.🤖 Generated with Claude Code