v0.1.0: honest dashboard, remote MCP daemon, migration, settings, one-line install#5
Merged
Merged
Conversation
Add Dim field to VoyageEmbeddingClient struct and update NewVoyageEmbeddingClient to accept dim int parameter (3rd arg). VectorSize() now returns c.Dim (defaulting to 1024 when 0) instead of hardcoded 1024. voyageRequest struct gets OutputDimension field sent as output_dimension in API requests. main.go buildProvider() passes cfg.VectorDim as the 3rd argument. Includes unit tests with httptest.Server verifying: - Dim field stored correctly (dim=512, dim=0 defaults to 1024) - VectorSize returns configured dimension - output_dimension sent in API request body - EmbedQuery also sends output_dimension - Auth header and input_type correctness Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…oviders Move QueryEmbedder interface from qdrant.go to provider.go so it is shared across all providers. Update pgvector and chroma SemanticSearch to check for QueryEmbedder via type assertion and use EmbedQuery (input_type=query) when available, falling back to Embed() when not. Qdrant already had this logic and continues to work unchanged. Add per-provider unit tests with mock embedders implementing and not implementing QueryEmbedder, verifying correct call dispatch and fallback behavior. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…mbed_model/embed_dim injection Add content_hash (SHA-256 first 8 bytes as 16 hex chars) and chunk_version='v2' to indexer document metadata. Implement incremental sync skip: chunks whose content_hash matches an existing point are not re-embedded. Add ModelNamer interface and ModelName() to VoyageEmbeddingClient. Update all provider Index methods (pgvector, qdrant, chroma) to inject embed_model and embed_dim into stored metadata. Extend PointInfo with ContentHash and DocID fields, and update ListPoints in all providers to return these fields. Add Skipped count to SyncResult. All 34 tests pass including hash format, version string, skip behavior, and per-provider metadata injection. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…CP refactor Create pkg/core/service.go with Service struct wrapping provider + reranker + indexer. Add SearchOpts (K, Recall, Hybrid, Rerank) with defaults K=5, Recall=40. Implement Search, IndexProject, ListProjects, CreateProject, DeleteProject, ListPoints, DeletePoints, IndexDocuments, RetrieveContext methods. Add EventBus for SSE events (Subscribe/Unsubscribe/Publish). Add Reranker interface and RerankHit struct to provider.go. Refactor main.go MCP tool handlers to be thin wrappers over core.Service. 26 unit tests with mock provider and mock reranker, all passing. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Create pkg/config/config.go with Config struct (YAML tags), Path() returning ~/.enowx-rag/config.yaml, Load() reading YAML (error if file missing), Save() writing YAML with chmod 0600 and mkdir. Implement config priority: env var > config file > default via Resolve(). Write unit tests for load, save (with permission check), round-trip, env var override priority, and defaults. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…o indexer (VAL-FND-015) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…earch (VAL-RAG-001..007,017) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…L-RAG-008..016) Add content_tsv tsvector GENERATED ALWAYS AS column and GIN index to project_memory table in ensureTable(). Implement SemanticSearchHybrid with Reciprocal Rank Fusion (k=60): FULL OUTER JOIN between dense (cosine) and lexical (ts_rank) rankings, score = COALESCE(1.0/(60+d.rank),0) + COALESCE(1.0/(60+l.rank),0). Add HybridSearcher optional interface to provider.go. Wire into core.Service.Search via retrieveCandidates helper that uses hybrid path when opts.Hybrid=true and provider implements HybridSearcher, falling back to dense-only otherwise. Use EmbedQuery for query embedding in hybrid path. ALTER TABLE migration handles existing tables without content_tsv. Integration tests with real PostgreSQL verify tsvector column, GIN index, RRF scoring, EmbedQuery usage, hybrid vs dense result differences, metadata completeness, and hybrid+rerank end-to-end. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…A serving
- Add --serve and --addr flags to main.go, split into runStdio()/runHTTP()
- Integrate pkg/config into main.go (config.Resolve for env > file > default)
- Create pkg/httpapi/ with chi router, REST handlers (GET /api/projects,
GET /api/projects/{id}, GET /api/projects/{id}/points, POST /api/projects/{id}/reindex,
DELETE /api/projects/{id}, POST /api/search, GET /api/stats, GET /api/events SSE),
SPA fallback handler, and JSON error handling
- Handle 404 for unknown /api/ routes (JSON, not SPA HTML)
- Implement ListProjectIDs in pgvector provider for core.Service.ListProjects
- Fix config.Save() to call os.Chmod(path, 0600) after os.WriteFile
- Replace joinStrings in pkg/core/service.go with strings.Join
- Add JSON tags to rag.Result and rag.PointInfo structs
- Add chunk_version to pgvector ListPoints SELECT
- Write handler tests with httptest (17 test cases)
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
The indexer generates string IDs like 'dir/file.go#chunk0' but the pgvector table used UUID PRIMARY KEY, causing syntax errors on insert. Changed ensureTable() DDL to use TEXT PRIMARY KEY, removed ::uuid cast in test query and id::text cast in ListPoints. Dropped and recreated existing project_memory tables. Added tests verifying string IDs work for Index, ListPointIDs, ListPoints, DeletePoints, and SemanticSearch. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
POST /api/setup/test: tests vector store + embedder connectivity,
returns per-component ok/message/latency_ms. Supports pgvector (TCP
dial), qdrant (/healthz), chroma (/api/v1/heartbeat), voyage
embeddings API, and TEI /embed endpoint.
POST /api/setup/apply: saves config to ~/.enowx-rag/config.yaml with
chmod 0600 via config.Save().
GET /api/setup/status: returns {configured: bool} based on whether
config file exists.
Includes 19 handler tests covering status (before/after apply),
apply (0600 permissions, valid YAML, overwrite, missing fields,
invalid JSON), test (qdrant success/fail, pgvector success/fail,
latency, unsupported store, missing fields), and parsePGDSN unit tests.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…vent cross-project data collision Doc IDs are based on source_dir + filename (e.g. 'config/config.go#chunk0'), so they collide across projects when using a single-column PK on id alone. Changed PRIMARY KEY to (project_id, id), updated ON CONFLICT clause, and added automatic migration for tables with the old single-column PK schema. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…L-API-015) Add ProjectExists check in the search handler before executing SemanticSearch. pgvector silently returns 0 rows for non-existent projects, so the handler was returning HTTP 200 with empty results instead of HTTP 404. The new ProjectExists method on core.Service checks project existence via ListProjectIDs (fast path) or ListPoints (fallback). Search errors on existing projects now correctly return 500 instead of 404. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Scaffold Vite + React + TypeScript + Tailwind + lucide-react SPA. Create true-black flat design tokens (light/dark via data-theme attr). Build app shell: sidebar (brand, nav, project list), topbar (breadcrumb, search, theme toggle), content area. Build Overview page with bento grid: KPI row (4 cards), retrieval playground (query, Run, toggles, chips, scored results with highlights), index status, retrieval breakdown, chunk distribution, recent files, activity log. Create API client (lib/api.ts) and SSE hook (lib/sse.ts). No gradients, no shadows, hairline borders, violet accent, Inter/JetBrains Mono fonts. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…..033,035,036)
Add Content and ChunkIndex fields to PointInfo, update ListPoints in
pgvector/qdrant/chroma to return chunk previews and indices. Add DELETE
/api/projects/{id}/points/{pointId} endpoint for individual chunk deletion.
Enhance Playground page with full-page retrieval layout, SSE activity panel
showing realtime events, and loading spinner. Rewrite Chunks page with content
previews, chunk indices, source_file filter, delete buttons, and pagination.
Update api.ts with deletePoint method and extended PointInfo interface.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Static HTML mockup of 6-step onboarding wizard (Welcome, VectorStore, Embedding, Test, AutoSetup, Done) following true-black flat design. Matches dashboard.html style with zero gradient, #000000 dark bg, hairline borders, violet accent, monospace for technical data. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
6-step wizard with stepper state machine: Welcome (env detection), VectorStore (pgvector/qdrant/chroma cards, local/cloud, mono endpoint), Embedding (voyage/tei cards, API key reveal, model selector, dim input, re-index warning), Test (POST /api/setup/test, per-component green/red, latency, proceed gate), AutoSetup (docker-compose generation, commands, auto-run with disclaimer, SSE progress), Done (POST /api/setup/apply, redirect to dashboard, config overwrite confirmation). First-run detection: App.tsx checks GET /api/setup/status, shows wizard when config missing. Draft state persisted in localStorage across step navigation and page navigation. Setup page accessible via sidebar nav after config exists. True-black flat design matching mockup. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
The wizard already persisted the full DraftConfig to localStorage but not the step position. When the wizard unmounted and remounted, it always restarted at step 1 (Welcome) even though form values were preserved. Add a wizard-step localStorage key that is saved on every step change and loaded on mount via loadStep(). Both wizard-draft and wizard-step are cleaned up when the user finishes the wizard. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…9,020) Add Makefile with targets: web (npm ci && npm run build), build (depends on web, go build -o enowx-rag), dev-web (npm run dev), dev-api (go run --serve), mcp (MCP-only build with placeholder dist), test, vet, clean, placeholder. Update .gitignore for new binary name. Verified: make web builds SPA to web/dist, make build produces self-contained binary, binary works from /tmp without web/ on disk, go build works without web/dist (placeholder), all 6 MCP tools work in stdio mode, env-only config works without config file. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
… (VAL-CROSS-014..018) - Update README with --serve/--addr flag docs, REST API endpoint table, one-command deploy instructions, comprehensive env var table (all RAG_* vars), and dual-mode (stdio + HTTP) documentation. - Add docker-compose.all-in-one.yml: enowx-rag server + Qdrant with optional PostgreSQL/pgvector and TEI services (commented out). - Add RAG_ADMIN_TOKEN middleware: protects /api/* endpoints with Bearer token auth when set, no-op when unset. Uses constant-time comparison to prevent timing attacks. - Update Dockerfile to multi-stage build: Node stage builds SPA, Go stage embeds it, runtime stage is minimal Alpine. - Add auth_test.go with 6 test cases covering all auth scenarios. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
… on SSE events (VAL-CROSS-009..013, VAL-WIZ-028) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…hcheck, clean mcp-server binary - Remove binary name (./enowx-rag) from docker-compose command field since Dockerfile ENTRYPOINT already provides it. Including both caused Docker to pass ./enowx-rag as the first arg, making the container run in stdio MCP mode instead of HTTP serve mode. - Add Authorization header to healthcheck using CMD-SHELL so the healthcheck passes when RAG_ADMIN_TOKEN is set (otherwise /api/stats returns 401). - Add mcp-server/mcp-server to Makefile clean target (built by 'make mcp'). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Two hardening fixes surfaced by a code audit against docs/PLANNING.md: - core.Service.Search now defensively truncates reranked results to k instead of trusting the rerank API to honor top_k. Adds TestSearchRerankClampsToK. - SSE (/api/events) no longer sends a wildcard Access-Control-Allow-Origin. The header is only emitted when RAG_CORS_ORIGIN is set, so the event stream stays same-origin only by default (important when exposed publicly alongside RAG_ADMIN_TOKEN). Adds TestSSE_NoCORSByDefault and TestSSE_CORSWhenConfigured, plus a README env-var row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implementation spec (PLANNING.md), agent handoff guide (HANDOFF.md), and the approved true-black dashboard mockup that guided the SPA build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Establishes the baseline required for the project to be adopted and contributed to: - LICENSE (Apache-2.0) + NOTICE — the project was previously unlicensed, which legally blocks reuse and contribution. - CONTRIBUTING.md, CODE_OF_CONDUCT.md (Contributor Covenant 2.1), SECURITY.md (private vuln reporting + operator hardening notes). - .github/: bug/feature issue templates, PR template, and a CI workflow (GitHub Actions) that runs go vet/build/test and the frontend build on every push and PR. - CHANGELOG.md documenting the v0.1.0 surface. - README: CI, license, and Go-version badges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the dashboard's mock/hardcoded data with real backend metrics and remove dead controls, so the UI never shows a number that didn't come from the backend. Surfaced by a UI audit after a dead "Settings" button was found. Backend (capability-based, local-first): - Token counter: VoyageEmbeddingClient/VoyageReranker parse usage.total_tokens and expose it via a new rag.TokenCounter interface; providers forward their embedder's count. No change to Embed/Rerank signatures. TEI reports 0 honestly. - core.Metrics: in-memory ring-buffer latency (avg/p50/p95) + query count, always on for every backend. Search records latency + QueryComposition. - GET /api/metrics returns the snapshot with honest capability flags (backend name, persistent=false unless the backend implements MetricsStore). - Compress search option: deterministic near-duplicate dedup (content_hash / identical content), no LLM. Wired through SearchOpts + SearchRequest. - `enowx-rag setup [--run]` CLI subcommand generates the docker-compose and optionally runs `docker compose up -d` in the terminal — never over HTTP, so there is no remote-command-execution surface on the server. - Fix: IndexProject now ensures the collection exists first, so first-time reindex from the HTTP UI no longer 404s. Frontend (honest UI): - Overview: drop all mock results/stats/KPIs; wire chunks, file count, chunk distribution, recent files to real listPoints; latency/tokens/retrieval breakdown to /api/metrics with honest empty states; Re-index prompts for a directory and reports success/failure; Compress toggle now functional. - Sidebar: remove dead "Settings" button and fabricated project list. - Topbar: remove non-functional ⌘K search box. - Onboarding: remove fake setTimeout auto-run simulation (pointed to `enowx-rag setup --run`); make Docker/Postgres checks honest. Tests: token accumulation, metrics ring/percentiles/concurrency, compress dedup, metrics-recording in Search, and the /api/metrics endpoint. Deferred (pgvector-only, follow-up): durable metric persistence + real dense/lexical retrieval composition via RRF projection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up audit found several features silently dead or unreachable
depending on backend/entry-point. Fixes:
- Qdrant & Chroma now implement ListProjectIDs, so /api/projects, /api/stats
and the whole dashboard work on those backends — previously they returned
empty unless the backend was pgvector. (Verified live on Qdrant: 10 real
projects / 14,786 chunks now surface; was 0/0 before.)
- MCP tools (rag_semantic_search, rag_retrieve_context) now accept and pass
Hybrid/Rerank/Recall/Compress. Previously they hardcoded SearchOpts{K,
Recall:K}, so the reranker and hybrid RRF were never used from MCP clients
(the primary interface) and the recall→rerank funnel was collapsed.
Hybrid/Rerank default to true; recall defaults to 40.
- Security: /api/setup/apply and /api/setup/test now require localhost or a
valid admin token (LocalOrAdminMiddleware), and /api/setup/apply no longer
echoes the saved config (which contained the Voyage API key) back in its
response. Removed the deprecated middleware.RealIP, which rewrote
RemoteAddr from client X-Forwarded-For and would have let a remote caller
spoof localhost to bypass the setup gate.
- pgvector SemanticSearchHybrid now projects in_dense/in_lexical, so the
retrieval-composition breakdown shows real dense/lexical counts instead of
always zero.
- Onboarding: the deployment=cloud toggle now actually skips local Docker
setup (previously it only changed a label); StepDone hides the reranker row
for TEI (no Voyage reranker); StepTest gate condition simplified; Playground
compress added to the search useCallback deps (toggle took effect one run
late).
- Removed dead joinStrings; corrected the misleading MetricsStore comment
(no provider can implement it without an import cycle — durable persistence
needs a separate injected persister, noted as future work).
Tests: Qdrant/Chroma ListProjectIDs, setup auth (remote rejected without
token, allowed with token, no key leak).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ListProjects (which powers /api/projects, /api/stats, and GetProject) counted a project's chunks by calling ListPoints, which scrolls every point WITH its full payload just to take len(). On Qdrant with 9 large projects this made the dashboard take >10s and time out. Add an optional ProjectCounter interface (CountPoints) implemented by all three providers with native counts — Qdrant points/count, pgvector COUNT(*), Chroma /count. ListProjects prefers it and only falls back to the ListPoints scroll when a provider doesn't implement it. Verified live on Qdrant: /api/projects and /api/stats dropped from >10s (timeout) to ~0.6s, same data (9 projects / 14,770 chunks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two follow-ups from the backend audit. Durable metrics persistence: - Add SQLiteMetricsStore (core), backed by ~/.enowx-rag/metrics.db via the pure-Go modernc.org/sqlite driver — no cgo, so the single-static-binary goal holds. It works with ANY vector-store backend (metrics live independently of the store), not just pgvector. - Redesign MetricsStore as a dependency injected into Service (SetMetricsStore) rather than a provider method. A provider lives in package rag, which cannot import core without an import cycle — the old design could never be implemented, so Persistent was always false. Now Search persists each query asynchronously and MetricsSnapshot reads durable aggregates, so counts and latency percentiles survive restarts. - Wired in main.go; falls back to in-memory if the DB can't be opened. - Verified live on Qdrant: persistent=true, and query_count/latency survive a server restart. Chroma honesty: - Documented the Chroma provider as experimental (legacy /api/v1, mock-tested only; Chroma >= 0.6 uses /api/v2) in the README provider table, a per-backend feature-support matrix, and a doc comment on ChromaProvider. Qdrant and pgvector are the supported/tested backends. Also documented /api/metrics, the compress search field, and the setup-auth requirement in the README, and updated CHANGELOG. Tests: SQLite store empty/aggregate/durability-across-reopen, and Service persisting to an injected store with Persistent=true. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the plain CSS letter "e" brand mark and the browser-default favicon with a proper icon: a stylized "e" monogram wired to two retrieval nodes, symbolizing per-project RAG memory + retrieval. Monochrome and theme-adaptive: black mark on light UI, white mark on dark UI. - Favicon swaps via <link media="(prefers-color-scheme)"> in index.html. - Sidebar and wizard brand marks swap via data-theme CSS (with a prefers-color-scheme fallback when no theme is stamped). - Icons live in web/public and are embedded into the single binary through the existing dist embed.FS. Sizes: 32 (favicon), 192 (apple-touch), 512. Generated from a flat solid-color source, background removed to transparency, then recolored to near-black/near-white keeping antialiased edges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llbar The dashboard scrolled the whole window when results were long. Lock the shell to one screen (app/main = 100vh, overflow hidden) and let content and the Playground result/activity panels scroll internally instead. Query box and toolbar stay pinned above the scrolling results. Also style the scrollbar to match the theme (thin pill using --border-strong) so it's no longer a glaring white bar in dark mode, and fix the Playground compress toggle being one run stale (added to the search useCallback deps). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a wizard "Install" step (6/7) that connects enowx-rag to the user's AI
tool, so onboarding ends with a working install rather than just a saved
config.yaml.
Backend (pkg/httpapi):
- mcpclients.go: registry of 8 clients (Claude Code, Claude Desktop, Cursor,
Cline, Windsurf, Codex, Zed, Continue) with per-format serializers. Three
format groups handled with merge-not-replace: JSON mcpServers (+ Cursor
type:stdio, Cline flags), Zed context_servers with args, Codex TOML section,
Continue YAML list (replace-by-name). Existing servers and top-level keys are
preserved; the original file is backed up to .bak before writing.
- setup_install.go: mcpServerEntry() builds command (os.Executable) + env from
the saved config. Endpoints: POST /setup/install-mcp (writes/merges, gated by
LocalOrAdminMiddleware), GET /setup/mcp-snippet (manual paste), GET
/setup/clients, GET /setup/skill-guide (manual skill instructions — skills
are only auto-installed by some clients, so we show commands instead).
Frontend:
- StepInstall.tsx: pick a client, install automatically or view a copy-paste
snippet ("Other" -> snippet only), global/project scope, plus a manual skill
install section. Wired into the wizard; step badges updated to /7.
- api.ts: mcpClients/installMcp/mcpSnippet/skillGuide.
Verified live: snippet correct per format; install merges into an existing
Cursor config (existing server + top-level keys preserved), writes type:stdio,
and creates a .bak. Tests cover every serializer, merge/backup, and the
loopback gate on install-mcp.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cker The "Auto Setup" step was misleading: it didn't auto-anything and it listed a docker-compose even when nothing needed to run locally (e.g. a hosted Qdrant + the Voyage API). Rework it: - Add localServices(cfg): a component needs Docker only if it's self-hosted at a localhost URL. A cloud API embedder (Voyage) and a remote vector store are omitted. generateDockerCompose/generateCommands now emit only those services. - StepAutoSetup: when localServices is empty, the step says "nothing to run locally" and can be skipped; otherwise it names the exact services and shows the compose + commands to run manually. - Rename the step "Auto Setup" -> "Local Backend" and clarify the copy. - Remove the now-dead "Deployment" (Local/Cloud) toggle and its DeploymentMode field — nothing read it; the localhost-URL check is the real signal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a third embedder alongside Voyage and TEI: RAG_EMBEDDER=openai, targeting any OpenAI-compatible /v1/embeddings API. One provider covers OpenAI, Together, Jina, Mistral, DeepInfra, LiteLLM, a local Ollama, etc. — set base URL + model, API key optional (empty for local/no-auth endpoints). - pkg/rag/openai.go: OpenAIEmbeddingClient (EmbeddingClient + ModelNamer + TokenCounter). URL normalization (root / /v1 / full path), optional "dimensions", token accounting, and VectorSize auto-probe when dim is unknown. - config + main.go + RuntimeConfig: OpenAIConfig and RAG_OPENAI_* env vars, wired into buildProvider and the install-mcp env map. - setup.go: /api/setup/test supports the openai embedder (reuses the provider, so URL handling matches production) and reports the detected dimension. - Wizard: third "OpenAI-compatible" card (base URL / key / model / dim) and the TEI card now explains it serves any local model (BGE, GTE, E5, nomic…). - README + CHANGELOG documented. Verified live: /api/setup/test against a mock OpenAI-compatible server connects and auto-detects the dimension. Tests cover URL normalization, request shape (dimensions sent/omitted), token counting, dimension probe, and no-auth mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend for migrating a project into enowx-rag, changing embedding
dimension/model, or moving between vector stores. Raw vectors are
model-specific and non-portable, so migration re-embeds from the stored text.
- rag.Exporter interface + ExportPoints on qdrant/chroma/pgvector: return every
point with FULL content and metadata (ListPoints truncates to 200 chars for
previews; export must not). Preserves doc_id so identity survives migration.
core.Service.ExportProject surfaces it.
- pkg/ragbuild: build a provider+embedder from an explicit Spec (no env), so a
*destination* provider can differ in store/model/dimension/pgvector-table.
main.go's buildProvider now delegates here (no duplicate logic).
- pkg/migrate.Migrator{Src,Dst}: export from source -> CreateCollection on dest
-> Index in batches (re-embedded by the destination embedder), with throttled
progress callbacks.
- POST /api/migrate: builds the destination from the request spec, runs the
migration asynchronously (202), streams migration_started/progress/completed/
failed over SSE (throttled to integer-percent changes so the bounded event
bus isn't flooded). Loopback/admin-token gated.
Verified live on Qdrant: migsrc (17 chunks) -> migdst re-embedded via Voyage;
SSE progressed 0->100%; dest has 17 chunks and is searchable. Tests cover
export (full content, doc_id preserved) on qdrant/chroma, the Migrator
(batching, collection creation, metadata preserved, non-exporter error), and
the endpoint gate + validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Migration" page to the dashboard (nav + route + SSE wiring) that drives the migration engine: pick a source project, choose a destination store + embedder (+ connection/model/dimension fields), name the destination, and run. - Live progress bar fed by the migration_* SSE events (registered in sse.ts), showing documents done/total and percent. - On success: prompt to verify, then an explicit "Delete source project" button (with confirm) — the source is never auto-removed. - pgvector destination shows a hint that a NEW table name is required to change dimension (single fixed-dimension table). - App.tsx/Sidebar/Topbar wired for the new 'migration' page; api.ts gains migrate() + MigrateRequest/MigrateResponse. Verified live via the Vite proxy (the page's exact network path): uimig -> uimig-voyage-4 re-embedded, 17 -> 17 chunks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…perimental)
Extend migration to pull from external cloud vector databases and re-embed into
enowx-rag. Migrator.Src is now a rag.Exporter, so any source — a local provider
or a cloud connector — flows through the same re-embed/write path.
- pkg/migrate/cloud: CloudSource + connectors.
- Qdrant Cloud reuses the tested rag.QdrantProvider against a remote URL —
VERIFIED live (imported a Qdrant collection → re-embedded → 17→17 chunks).
- Pinecone (list+fetch), Weaviate (GraphQL), Chroma Cloud (/get) read text
from metadata. EXPERIMENTAL: built from vendor docs, mock-tested only, not
verified against live accounts — flagged in code, UI, README, CHANGELOG
(same honesty policy as the Chroma provider).
- POST /api/migrate accepts an optional cloud_source; the source becomes the
connector instead of the running provider.
- Migration page: source-mode toggle (existing project / import from cloud) with
per-vendor fields and a clear experimental banner for unverified vendors.
- README "Migration" section + CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Let an AI agent set up enowx-rag for a project from one short prompt, doing only
what's missing.
- GET /api/docs/setup: markdown the agent reads — probe, then install the MCP
server, skill, and AGENTS.md block, skipping anything already present.
- GET /api/setup/probe?client=&dir=: reports current state — whether the
enowx-rag MCP server is in each client's config, whether the skill is
installed, and whether the project's AGENTS.md already has the enowx-rag block.
- POST /api/setup/write-agents-md {dir, project_id}: merges an enowx-rag section
into AGENTS.md via <!-- enowx-rag:start/end --> markers — create if missing,
append (preserving the user's content) if unmarked, or update in place if the
markers exist. Idempotent. Loopback/admin-token gated.
- Install step shows a copy-paste prompt that points the agent at the docs URL.
Verified live: probe correctly reported claude-code MCP + skill installed and
AGENTS.md present-without-block on this machine; write appended while preserving
user content, then re-running updated in place (single block, no duplication).
Tests cover create/append/update merge, content preservation, and probe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Docs" page to the dashboard so the agent-setup instructions are readable by humans, not just fetchable by agents. It shows the copy-paste agent prompt at the top and renders the /api/docs/setup markdown below (headings, paragraphs, lists, inline code, endpoint blocks) with a tiny inline renderer — no markdown dependency added. Wired into nav/route/topbar as the 'docs' page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expand Docs from the single agent-setup page into a full, sectioned reference
served from the backend so both the dashboard and agents read one source.
Backend (pkg/httpapi/docs.go):
- Section registry with GET /api/docs (table of contents) and
GET /api/docs/{section} (markdown). /api/docs/setup kept as an alias for the
agent-setup section. Sections: Overview, Quick start, MCP tools, API
reference, Embedders, Vector stores, Search (hybrid/rerank/compress),
Migration, Metrics, Agent setup — each documenting the real, shipped behavior.
Frontend (Docs page):
- Left section nav + content pane; fetches the list and each section's markdown.
- Minimal inline markdown renderer extended to handle tables and code blocks (no
markdown dependency). The agent-setup section keeps the copy-paste prompt.
Tests cover the docs list, a section, the setup alias, and 404. Verified live:
/api/docs returns all 10 sections.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the MCP server over HTTP so agents can use enowx-rag as a centralized
remote daemon (e.g. on a VPS) instead of a local stdio process — the user's
"daemon on a VPS, accessed by API key" use case.
- Factor server construction into newMCPServer(svc), shared by stdio and HTTP.
- runHTTP mounts a StreamableHTTPHandler (stateless) at /mcp using the SDK's
NewStreamableHTTPHandler (go-sdk v0.4.0, no new dependency). One server
instance is shared across all sessions.
- NewRouter gains an mcpHandler param; /mcp is mounted before the SPA catch-all
in a group gated by the existing AdminTokenMiddleware, so RAG_ADMIN_TOKEN
secures /mcp exactly like /api (open when unset — local only).
- Docs "Remote / daemon" section + README + CHANGELOG: how to run the daemon,
secure it, and connect a client via { url, headers: { Authorization } }.
Local stdio mode is unchanged (default, no --serve).
Verified live: with RAG_ADMIN_TOKEN set, POST /mcp without a bearer → 401; with
the bearer, the MCP `initialize` handshake returns capabilities and `tools/list`
returns all six RAG tools. Tests cover the /mcp gate (401 / pass-through / open).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After adding MCP-over-HTTP, the setup flow only produced local stdio config.
Now install-mcp and mcp-snippet can produce a remote entry pointing at a daemon.
- mcpEntry gains RemoteURL + Token; all serializers (JSON mcpServers /
context_servers, Codex TOML, Continue YAML list) emit a { url, headers:
{ Authorization: Bearer <token> } } entry when remote, instead of command+env.
- install-mcp accepts mode/remote_url/token; mcp-snippet accepts
?mode=remote&remote_url=&token= (with a placeholder URL if none given).
- StepInstall UI: a Local (stdio) / Remote daemon toggle with URL + token fields;
applies to both auto-install and the manual snippet.
- Agent-setup docs document the remote option and point to the Remote / daemon
section.
Verified live: remote snippet (JSON + TOML) yields url + Authorization header,
and a remote install writes a { url, headers } Cursor config. Tests cover the
remote form across JSON/TOML/YAML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose more of core.Service to agents. Adds (6 -> 11 tools): - rag_list_projects: list projects + chunk counts (discover available memory) - rag_project_exists: check a project has indexed data - rag_list_points: list chunks (id, source_file, preview), optional file filter - rag_delete_points: delete specific chunks by ID (no full re-index) - rag_stats: projects, total chunks, embed model, latency, token usage All are thin wrappers over existing Service methods and work over both stdio and remote HTTP. Docs "MCP tools" section, README, and CHANGELOG updated. Verified live via /mcp: tools/list returns all 11; rag_list_projects returns the 9 real projects with chunk counts. Note: setup/config (apply config, install MCP) is intentionally NOT an MCP tool — that's the HTTP API + wizard's job; an agent talking over MCP is already installed. MCP tools stay scoped to memory operations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a Settings page (and endpoints) to view and change secrets without editing config.yaml or env by hand. - GET /api/setup/config: current config with secrets masked (pa-a••••viD). - GET /api/setup/config/reveal: full secrets — gated by LocalOrAdminMiddleware (localhost or a valid admin token) so it isn't open on an exposed daemon. - POST /api/setup/config: partial update; only provided keys overwrite, merged into ~/.enowx-rag/config.yaml (0600). - POST /api/setup/gen-token: generate a strong random admin token, save it to config, and return it once to copy. - config.AdminToken + EffectiveAdminToken(): the admin token now comes from RAG_ADMIN_TOKEN (precedence) OR config.yaml. AdminTokenMiddleware / LocalOrAdminMiddleware read it per-request, so a generated token gates requests immediately without a restart. - Settings UI: masked keys with reveal, per-key update, and token generation with an env-precedence warning. Verified live: config returns a masked Voyage key; reveal (localhost) returns the full key. Tests cover masking and that a generated token then gates a remote request (401 without it, 200 with it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Settings page listed every provider key (Voyage, OpenAI, Qdrant) even when unused, which was confusing — e.g. an OpenAI key field with a Voyage setup. Now it shows only the keys relevant to the active config: the embedder's key (Voyage / OpenAI-compatible, or a "no key needed" note for TEI) and the vector store's key (Qdrant). A hint states the active embedder + vector store and points to the Setup wizard to change them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Auth middleware now reads config.EffectiveAdminToken(), which loads ~/.enowx-rag/config.yaml. That made httpapi tests non-deterministic: a developer's saved admin_token (or a RAG_ADMIN_TOKEN in the environment) would turn expected-200 responses into 401s, and host config could leak into setup/config assertions. This is a release/CI prerequisite. - newTestServer points HOME at an empty temp dir and clears RAG_ADMIN_TOKEN, but skips the HOME override when a test has already isolated it (HOME != realHome captured at init), so config-writing tests keep their own dir. - Tests that exercise auth set RAG_ADMIN_TOKEN *after* newTestServer (read per-request, so it still takes effect). - Direct NewRouter callers that need host isolation (TestSearch_ BadProject_NoLister, TestMCPMount_OpenWhenNoToken) isolate HOME. Verified deterministic: suite passes uncached with both a host config admin_token present AND RAG_ADMIN_TOKEN=hostile-leak-token in the env, and on a clean empty HOME. go vet clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The only way to install was clone + `make build` (needs Go + Node). Add simple, toolchain-free install paths for v0.1.0. The binary is CGO-free (pure-Go modernc.org/sqlite, no `import "C"`), so it cross-compiles to all targets; web/dist is committed, so `go install` embeds the real dashboard without npm. Verified builds for linux, darwin (amd64/arm64) and windows/amd64. - .goreleaser.yaml: cross-compile matrix, -trimpath, version via ldflags, tar.gz/zip archives + checksums, Homebrew tap (brews:). - .github/workflows/release.yml: on tag v*, run GoReleaser then publish the npm wrapper. - install.sh: curl|sh — detect OS/arch, download the matching release archive, verify checksum, install to /usr/local/bin (or ~/.local/bin). - npm/: thin wrapper (postinstall downloads the native binary; bin shim execs it and forwards args/exit). No runtime deps. - main.go: `enowx-rag version`/`--version`; version stamped via -X main.version, with a runtime/debug build-info fallback for `go install ...@vX.Y.Z`. - README Install section; CHANGELOG; gitignore dist/ + npm/bin. Archive naming verified identical across GoReleaser, install.sh, and the npm wrapper. Full Go suite green; go vet clean. GoReleaser config will be exercised by CI on the release tag (couldn't build locally: disk full). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TestSetupTest_PGVectorSuccess dialed a real local PostgreSQL
("per the mission setup"), so it passed on the dev machine but failed
in CI (connection refused, no DB there). The pgvector check only dials
TCP (setup.go: "we don't import pgx here"), so a bare net.Listen stand-in
proves reachability without any database — deterministic in CI.
Verified: full suite green with an empty HOME and no Postgres running.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a hero Overview screenshot, a "Why enowx-rag?" benefits section, and a 6-shot dashboard preview gallery (Overview, Playground, Chunks, Migration, Docs, Settings) to the README. Screenshots are captured from a live instance with real data (16k+ chunks across 9 projects, real retrieval results with scores/snippets), not mockups. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepares enowx-rag for its first tagged release (v0.1.0). 47 commits; highlights:
Install & distribution
install.sh(curl | sh),Homebrew tap, npm wrapper, and
go install. GoReleaser cross-compiles CGO-freebinaries for macOS/Linux (amd64/arm64) and Windows (amd64);
web/distis committedso
go installembeds the real dashboard. Release workflow runs on tagv*.enowx-rag version/--version, stamped via ldflags with a build-info fallback.Features
--servealso exposes/mcp(Streamable HTTP,stateless), gated by the same
RAG_ADMIN_TOKENbearer — run enowx-rag on a VPS.gated reveal, per-request token from env or config).
between vector stores; cloud import (Qdrant Cloud verified; others experimental).
/api/docs/setup, idempotent probe, AGENTS.md merge.persisted to SQLite (removed all mock/hardcoded data).
Test hygiene (release prerequisite)
~/.enowx-rag/config.yamland env vars so thesuite is deterministic in CI. Verified green uncached even with a host config
admin_tokenpresent and a hostileRAG_ADMIN_TOKENin the environment.Security
Follow-up (needs repo secrets, not blocking binaries/install.sh/go install)
HOMEBREW_TAP_GITHUB_TOKEN(+enowdev/homebrew-taprepo) for the brew step.NPM_TOKENfor the npm publish step.🤖 Generated with Claude Code