diff --git a/docs/design/byok-confluence-import/byok-confluence-import-spike.md b/docs/design/byok-confluence-import/byok-confluence-import-spike.md new file mode 100644 index 000000000..8aea4478d --- /dev/null +++ b/docs/design/byok-confluence-import/byok-confluence-import-spike.md @@ -0,0 +1,651 @@ +# Spike for LCORE-788: Auto import of content for BYOK from Confluence + +Spike ticket: LCORE-2664. Feature ticket: LCORE-788 (scope reduced 2026-06 to +Confluence only). Related epic: LCORE-256 (Alternative BYOK — frequent KB +updates without rebuild pain). Product context: OCPSTRAT-2278. + +## Overview + +**The problem**: Customers don't want to hand-build BYOK RAG databases. They +want to point Lightspeed at their Confluence and have the content imported — +and kept up to date — automatically. Today the BYOK workflow is entirely +manual and offline: an admin prepares Markdown/HTML/text files, runs the +`lightspeed-rag-content` tooling to build a faiss/pgvector store, and ships +the artifact to the service by hand. Nothing fetches from a remote source, +nothing refreshes, and the service cannot pick up a changed DB without a +restart. + +**The recommendation**: Add a **Confluence fetch stage** to +`lightspeed-rag-content` (rendered-HTML export via the Confluence REST API → +existing docling/HTMLReader pipeline), plus a **documented Kubernetes +CronJob** that periodically re-crawls, rebuilds the vector-DB artifact, and +triggers a service rollout. Keep lightspeed-stack unchanged (existing +`byok_rag` config consumes the artifact as before). Incremental sync ships +as change *detection* (skip re-fetching/re-embedding unchanged pages) while +always producing a full rebuilt artifact; hot-reload of a live store is +explicitly out of scope. + +**PoC validation**: A moderate PoC crawled a real space on +redhat.atlassian.net (Confluence Cloud, 20 pages, read-only API token), +built a `llamastack-faiss` store through the unmodified rag-content +pipeline, retrieved correct chunks with per-page Confluence URLs as +citations, and demonstrated incremental change detection (second run: +2 API calls instead of 21, zero re-embeds). See [PoC results](#poc-results). + +## Settled decisions + +Decided during the spike; recorded for context, **no reviewer +confirmation needed**. + +### Decision S1: Where the importer lives — `rag-content` (settled) + +The Confluence importer is build-time content tooling. The +[BYOK PDF spike](../byok-pdf/byok-pdf-spike.md) (Decision 3) already +established `lightspeed-core/rag-content` as the home for import tooling; +lightspeed-stack never opens vector DBs directly (all access goes through +the llama-stack client) and its config direction keeps ingestion out of the +serving path. The PoC needed zero changes to rag-content library code — +only a fetch script and a `MetadataProcessor` subclass. Alternatives +considered and rejected: lightspeed-stack (ingestion + crawler/docling +deps in the serving codebase) and a new repository (duplicated pipeline +internals, extra repo/release overhead). See +[Current BYOK state](#current-byok-state). + +## Strategic decisions — for @sbunciak + +High-level decisions that determine scope, approach, and cost. Each has a +recommendation — please confirm or override. + +### Decision S2: Refresh architecture + +LCORE-256's goal is frequent KB updates without pain. Today the service has +no hot-reload: a rebuilt DB requires a restart (no file-watching exists; +see [Current BYOK state](#current-byok-state)). See +[Refresh patterns on Kubernetes](#refresh-patterns-on-kubernetes). + +| Option | Description | +|--------|-------------| +| A. Periodic full re-crawl → full rebuild → new artifact → rollout | Immutable artifact, deletions handled for free, zero service changes; staleness = cron period | +| B. Incremental sync into a live store | Fresher; requires net-new hot-reload plumbing in the service and mutates the artifact in place | +| C. Runtime ingestion inside lightspeed-stack | Service crawls Confluence itself; crawl+embed load in the serving path | + +**Recommendation**: A as the epic baseline. Ship the cheap half of B as an +optional optimization (skip re-fetching/re-embedding unchanged pages via +CQL `lastmodified` + page-version cache — output is still a full rebuilt +artifact; validated in the PoC). Hot-reload stays out of scope as a +possible future epic. + +**Confidence**: 85% + +### Decision S3: Primary delivery artifact + +Upstream Lightspeed Core Stack consumes a **file** (`byok_rag[].db_path`, +volume-mounted). Downstream OpenShift Lightspeed consumes a **BYOK OCI +image** (`/rag/vector_db`; see [Prior art](#prior-art-s2i-poc-and-ols-byok)). +rag-content already produces both (`--output-image`). + +| Option | Description | +|--------|-------------| +| A. DB file primary, image optional | Filesystem artifact is the documented happy path; `--output-image` remains for image-based products | +| B. OCI image primary | Matches OLS contract; forces registry infra into the minimal upstream path | +| C. Both first-class | Double docs/test surface in this epic | + +**Recommendation**: A. + +**Confidence**: 80% + +### Decision S4: Form of the deployable automation + +LCORE-788 asks for "an automation Lightspeed admins can deploy". + +| Option | Description | +|--------|-------------| +| A. Documented Kubernetes CronJob reference manifest | Wraps the importer container; portable to any k8s/OpenShift cluster | +| B. Shipwright BuildStrategy | Extends the s2i PoC; OCP-specific, assumes image contract + registry | +| C. Docs only | Ship the importer CLI/container; scheduling left entirely to admins | +| D. A + podman/systemd-timer reference | A, plus a Quadlet (`.container` + `.timer` systemd units) reference for podman-only hosts | + +**Recommendation**: D — CronJob reference manifest + admin guide in +rag-content, documenting the DB-file flow (shared volume + rollout +trigger), **plus a podman-only reference**: Quadlet systemd units +(`.container` + `.timer`) running the same importer container on a +schedule, with a plain-cron one-liner noted as the minimal fallback. +Both references drive the identical container image and flags — the +automation layer is thin by design. Note B as the natural OLS +product-layer variant built on the same importer container. + +*2026-07-21: extended from A to D per PM review on the spike PR +(non-Kubernetes deployments must be covered).* + +**Confidence**: 88% (PM-confirmed with the podman amendment) + +### Decision S5: Lightspeed-stack config surface + +LCORE-788's scope list literally includes "provide configuration options in +Lightspeed stack config to enable this auto import". Under the recommended +design the importer is an offline tool: the existing `byok_rag` / +`rag.byok.stores` config already covers consumption of the produced +artifact, and importer configuration (Confluence URL, spaces, schedule, +credentials) belongs to the importer/CronJob, not the serving config. + +| Option | Description | +|--------|-------------| +| A. No new lightspeed-stack config | Importer config lives with the importer; LCS docs point at it | +| B. Mirror importer settings in lightspeed-stack.yaml | Serving config describes an import the service itself never performs | + +**Recommendation**: A. This deliberately reinterprets the ticket letter — +flagging for explicit PM confirmation. + +**Confidence**: 82% + +## Technical decisions — for @tisnik + +### Decision T1: Fetch client library + +See [Existing loaders](#existing-confluence-loaders). The PoC used raw +stdlib HTTP successfully; production needs pagination, retry/backoff, and +Cloud/DC parity without hand-rolling. + +| Option | Description | +|--------|-------------| +| A. atlassian-python-api (Apache-2.0) | Thin, mature wrapper over Cloud+DC REST; we keep control of body format, CQL, state | +| B. llama-index-readers-confluence (MIT) | Highest-level; brings heavy attachment deps (pytesseract, pdf2image, docx2txt…); hides pagination/state | +| C. Raw REST (requests/httpx) | No new dep; we own retry/backoff/pagination/Cloud-DC differences | + +**Recommendation**: A. B's attachment machinery is out of scope (T5) and +its abstraction hides the sync-state control we need; C re-implements what +A already does. A is also what B uses underneath. + +**Confidence**: 70% + +### Decision T2: Page body format + +| Option | Description | +|--------|-------------| +| A. `body.export_view` (rendered HTML, v1 API) | Macros expanded; clean input for docling HTML→Markdown | +| B. `body.storage` (XHTML + `ac:`/`ri:` macros) | v2-native but macro tags mangle generic HTML→MD conversion | +| C. `atlas_doc_format` (ADF JSON) | Needs a dedicated renderer | + +**Recommendation**: A. PoC evidence: headings, bold, links, nested lists, +and expand-macros all converted cleanly; the docling letter-spacing weakness +that affects Confluence "Export to PDF" does not apply to the HTML path. +Risk: `export_view` is not in the v2 API — Cloud requires v1 endpoints +(deprecation risk over the feature lifetime; DC keeps this surface). + +**Confidence**: 88% + +### Decision T3: Authentication modes + +| Option | Description | +|--------|-------------| +| A. Cloud email+API-token (Basic) and DC PAT (Bearer) | Two single-secret headless modes, from env/K8s Secret | +| B. Also OAuth 2.0 3LO | Interactive consent + rotating refresh tokens; built for marketplace apps, poor fit for a CronJob | + +**Recommendation**: A. Documentation must recommend a **service/bot +account** over personal tokens (Atlassian policies commonly require it — +Red Hat's own instance does; PoC finding 4) and must surface auth failures +loudly (Cloud tokens expire ≤ 1 year). + +**Confidence**: 90% + +### Decision T4: Content selection configuration + +**Recommendation**: Space key list (required) + optional CQL filter and/or +label filter per space. CQL is the native server-side selection mechanism +and doubles as the incremental-sync primitive. + +**Confidence**: 85% + +### Decision T5: Attachments + +**Recommendation**: Out of scope for the baseline epic — pages only. +PDF/Office attachments can later route through the byok-pdf pipeline +(LCORE-2090 line of work) as a follow-up; images/diagrams are dropped by +the HTML→Markdown conversion anyway. + +**Confidence**: 75% + +### Decision T6: Citation metadata plumbing + +**Recommendation**: The fetch stage writes a `manifest.json` +(file → page id, title, canonical URL, version); a +`ConfluenceMetadataProcessor(MetadataProcessor)` resolves `url_function` +and `get_file_title` from it, with `hermetic_build=True` (page URLs are +auth-gated; reachability pings would mark every URL unreachable). Proven in +the PoC — every retrieved chunk carried title + canonical page URL. + +**Confidence**: 90% + +### Decision T7: Embedding-model pinning + +A rebuilt DB is only queryable with the exact embedding model configured in +the consuming service; same-dimension model drift fails *silently*. + +**Recommendation**: The importer requires an explicit model name; default to +HF-id resolution (`-md '' -mn ` convention) so the artifact is +path-portable; the admin guide states the runtime-match requirement in a +warning box. (PoC confirmed the absolute-model-path bake-in live.) + +**Confidence**: 88% + +### Decision T8: Change-detection mechanics + +**Recommendation**: Sync state file (per-page version numbers + watermark) +kept next to the output. Delta = CQL `lastmodified >= watermark − overlap` +∪ pages absent from state; **deletions** = full ID enumeration diff (cheap, +no bodies; CQL alone cannot see deletions). Unchanged pages are neither +re-fetched nor re-embedded. Validated in the PoC for the no-change and +new-state cases; live update/delete exercised by the e2e tickets against a +mock Confluence. + +**Confidence**: 85% + +## Out of scope + +- **Hot-reload of vector stores in lightspeed-stack** — no file-watching or + re-registration path exists today; a restart/rollout picks up the new + artifact. Deliberately deferred; candidate future epic under LCORE-256. +- **SharePoint, Git, generic web sources** — OCPSTRAT-2278 mentions them; + LCORE-788 scope was explicitly reduced to Confluence (PM comment, + 2026-06-15). The fetch-stage/manifest seam is source-agnostic by design. +- **Attachments** (T5) — follow-up via the byok-pdf pipeline. +- **OAuth 2.0 3LO** (T3). +- **Operator/OLSConfig/Shipwright integration** — product-layer (OLS) work + on top of the importer container; see + [Prior art](#prior-art-s2i-poc-and-ols-byok). +- **Confluence permission mirroring** — imported content is served to all + LCS users regardless of per-page Confluence restrictions; the admin guide + must warn to import only spaces whose audience matches the assistant's. + +## Proposed JIRAs + +### Epic: BYOK auto-import from Confluence + +Automation that builds and refreshes a BYOK vector store from Confluence, +per `docs/design/byok-confluence-import/byok-confluence-import.md`. + +**Goals**: +- An admin can point the importer at Confluence spaces and get a BYOK + vector-store artifact with per-page citations, with no manual document + preparation. +- A documented CronJob keeps the artifact fresh (re-crawl, skip unchanged + pages, rebuild, trigger rollout). +- Lightspeed Core Stack consumes the artifact with existing `byok_rag` + configuration — no service changes. + +**Scope**: +- In: rag-content fetch stage (Cloud + Data Center), incremental change + detection, CronJob reference manifest, docs, unit/integration/e2e tests. +- Out: hot-reload, attachments, non-Confluence sources, operator + integration (see spike doc Out-of-scope). + +**Success criteria**: +- E2E: mock-Confluence → importer → artifact → lightspeed-stack answers + with page-URL citations; a page edit followed by re-import updates + answers after rollout. + + + +#### LCORE-???? E2E feature files for BYOK Confluence auto-import (no step implementation) + +**User story**: As a Lightspeed Core e2e engineer, I want the behave +feature files for the Confluence auto-import scenarios written before the +implementation lands, so that the test shape reflects intended behavior +rather than the chosen implementation. + +**Description**: Author behave `.feature` files under `tests/e2e/features/` +describing: import from a (mock) Confluence produces a queryable BYOK +store with page-URL citations; incremental re-import skips unchanged pages; +an edited page's content is served after re-import + restart; a deleted +page's content disappears after re-import + restart. Step definitions are +explicitly **not** part of this ticket (covered by LCORE-????). + +**Scope**: +- `.feature` files covering R1..Rn from the spec doc +- Additions to `tests/e2e/test_list.txt` +- Author from spec doc requirements only; do not read implementation code + +**Acceptance criteria**: +- behave parses every new `.feature` file without syntax errors +- behave marks all new scenario steps as `undefined` +- `uv run make test-e2e` remains green (new scenarios undefined, not failing) + +**Blocks**: LCORE-???? (step-definitions counterpart) + +**Agentic tool instruction**: + +```text +Read "Requirements" and "Acceptance test surface" in +docs/design/byok-confluence-import/byok-confluence-import.md. +Do NOT read other JIRAs' scope sections or implementation code while +authoring. Key files to create: +tests/e2e/features/byok_confluence_import*.feature plus additions to +tests/e2e/test_list.txt. Do NOT create step definitions. +``` + + + +#### LCORE-???? Implement behave step definitions for BYOK Confluence auto-import + +**Description**: Implement Python step definitions under +`tests/e2e/features/steps/` for the feature files authored in the kickoff +ticket, including a mock Confluence REST fixture (v1 content listing, +`body.export_view`, CQL search) so scenarios run hermetically. Take the +Gherkin as-is; if a scenario cannot be implemented faithfully, raise it +against the spec doc rather than weakening the test. + +**Blocked by**: +- LCORE-???? (E2E feature files kickoff) +- LCORE-???? (Confluence fetch stage) +- LCORE-???? (pipeline integration + packaging) + +**Agentic tool instruction**: + +```text +Read "Architecture" and "Requirements" in +docs/design/byok-confluence-import/byok-confluence-import.md. +Take feature files from tests/e2e/features/byok_confluence_import*.feature +as-is. To verify: `uv run make test-e2e` runs every new scenario green. +``` + + + +#### LCORE-???? Implement Confluence fetch stage in rag-content + +**User story**: As a Lightspeed admin, I want to fetch Confluence spaces as +pipeline-ready documents with one command, so that I don't prepare BYOK +content by hand. + +**Description**: New `lightspeed_rag_content.confluence` module + CLI +(mirroring the `html`/`pdf` module pattern): crawl configured spaces via +the REST API (Cloud API-token and DC PAT auth), write per-page rendered +HTML + `manifest.json` + sync state; incremental mode per spike Decision +T8 (CQL delta, version skip, deletion diff). Includes +`ConfluenceMetadataProcessor` (manifest-driven URL/title, +`hermetic_build`). Unit tests against recorded API fixtures. + +**Scope**: +- Full + incremental crawl, deletion handling, pagination, retry/backoff + honoring 429 `Retry-After` +- Auth from env vars; no secrets in CLI args or logs +- Loud, actionable failure on auth errors (expired token) and permission + denials + +**Acceptance criteria**: +- One command produces a directory consumable by the existing pipeline + with correct per-page metadata +- Second run against unchanged spaces performs no body fetches and no + re-embeds +- Deleting a page removes it from the next build +- Works against Confluence Cloud and Data Center API shapes (fixtures) + +**Agentic tool instruction**: + +```text +Read "Architecture" and "Implementation Suggestions" in +docs/design/byok-confluence-import/byok-confluence-import.md. +Key files: src/lightspeed_rag_content/confluence/ (new), +tests/confluence/ (new), following the html/ and pdf/ module patterns +in lightspeed-core/rag-content. +``` + + + +#### LCORE-???? End-to-end importer command and container packaging + +**Description**: A single `import-confluence` entrypoint that runs +fetch → build → (optional) image packaging in one invocation, wired into +the rag-content tool image; integration test covering +fetch-dir → `llamastack-faiss` artifact with citation metadata. + +**Blocked by**: LCORE-???? (Confluence fetch stage) + +**Scope**: +- CLI orchestration (fetch stage + DocumentProcessor + optional + `--output-image`) +- Embedding-model pinning per spike Decision T7 (explicit model, HF-id + resolution default, mismatch warning in output) +- Tool-image (Containerfile) inclusion + smoke test + +**Acceptance criteria**: +- `podman run … import-confluence …` with env-provided credentials + produces the artifact end-to-end +- Integration test builds from fixture HTML and asserts chunk metadata + (title, canonical URL) + +**Agentic tool instruction**: + +```text +Read "Architecture" and "Implementation Suggestions" in +docs/design/byok-confluence-import/byok-confluence-import.md. +Key files: scripts/, Containerfile, src/lightspeed_rag_content/confluence/ +in lightspeed-core/rag-content. +``` + + + +#### LCORE-???? Scheduled-refresh references (CronJob, podman/systemd) and admin automation guide + +**Description**: Reference automation for scheduled refresh in two +deployment shapes, both driving the same importer container: a Kubernetes +CronJob manifest (shared volume for the artifact + state; rollout trigger +for lightspeed-stack) and a podman-only Quadlet reference (`.container` + +`.timer` systemd units, plain-cron fallback noted). Plus the admin guide +covering credentials (Secret / podman secret; service-account +recommendation), scheduling, staleness expectations, and the security +warning about permission flattening. + +**Blocked by**: LCORE-???? (end-to-end importer command) + +**Acceptance criteria**: +- Manifest applies cleanly on a stock OpenShift/k8s cluster +- Quadlet units run the same refresh on a podman-only RHEL-family host + (systemd timer fires, artifact refreshed, restart documented) +- Guide walks an admin from zero to scheduled refresh with the DB-file + contract in both shapes (image variant referenced) + +**Agentic tool instruction**: + +```text +Read "Architecture" and "Rollout / deployment plan" in +docs/design/byok-confluence-import/byok-confluence-import.md. +Key files: examples/ and docs/ in lightspeed-core/rag-content. +``` + + + +#### LCORE-???? Update lightspeed-stack BYOK documentation for Confluence auto-import + +**Description**: Extend `docs/byok_guide.md` with the Confluence source: +point at the rag-content importer + CronJob guide, document the +embedding-model-match requirement, the restart-to-pick-up-changes +behavior, and the permission-flattening warning. Per spike Decision S5, no +lightspeed-stack config changes. + +**Blocked by**: LCORE-???? (CronJob reference manifest and admin guide) + +**Acceptance criteria**: +- byok_guide has an end-to-end "from Confluence" walkthrough consistent + with the rag-content docs + +**Agentic tool instruction**: + +```text +Read "What" and "Architecture" in +docs/design/byok-confluence-import/byok-confluence-import.md. +Key files: docs/byok_guide.md, docs/rag_guide.md in +lightspeed-core/lightspeed-stack. +``` + +## Proposed incidental JIRAs + + + +### LCORE-???? rag-content: generated llama-stack.yaml conflicts with registration persisted in faiss_store.db + +**Description**: A freshly built `llamastack-faiss` store cannot be opened +with its own generated `llama-stack.yaml`: the +`registered_resources.vector_stores` entry re-registers the vector store +with fewer fields than the registration already persisted inside +`faiss_store.db`, and llama-stack raises +`ValueError: Object of type 'vector_store' … already exists with +conflicting field values: {'provider_resource_id': (None, 'vs_…'), +'vector_store_name': (None, '')}`. This breaks +`scripts/query_rag.py` out of the box (observed during the LCORE-2664 PoC; +worked around by dropping the `registered_resources.vector_stores` entry +and querying the persisted registration). Likely a llama-stack +version-bump regression: either the generated yaml should carry the full +field set, or query_rag should not re-register. + +## PoC results + +### What the PoC does + +Moderate-ambition PoC (three small scripts + +`ConfluenceMetadataProcessor`), run against **real Confluence Cloud** +(redhat.atlassian.net, space RHAT, 20 pages, regular user's read-only API +token): + +1. Full crawl: v1 REST `body.export_view` → 20 HTML files + + `manifest.json` + `state.json`. +2. Build: unmodified rag-content pipeline (docling `HTMLReader`, + `MarkdownNodeParser`, `all-mpnet-base-v2`, `llamastack-faiss`) → + `faiss_store.db` (3.8 MB). +3. Verify: `vector_io.query` via the llama-stack library client. +4. Incremental: second crawl with CQL `lastmodified` + version comparison. + +**Important**: The PoC diverges from the production design in these ways: +- Raw stdlib HTTP instead of atlassian-python-api; no retry/backoff. +- Credentials read from the developer's Jira CLI credentials file. +- No CLI/config surface, no container, no CronJob. +- Update/delete sync paths implemented but not exercised live (no writable + space available — see finding 3); covered logically and deferred to the + e2e tickets (mock Confluence). + +### Results + +| Step | Outcome | +|---|---| +| Full crawl | 20/20 pages, ~1 API call/page + 1 listing call | +| Build | All pages converted; no letter-spacing artifacts (HTML path) | +| Retrieval | Top-3 chunks for a policy question: correct pages, scores 2.25/1.65/1.60 | +| Citations | Title + canonical page URL on every chunk | +| Incremental | `fetched=0 unchanged=20 deleted=0`; 2 API calls vs 21 | + +Evidence: `poc-results/` (sanitized — crawled content is Red Hat-internal; +removed before merge). + +### Findings discovered during the PoC + +- **Zero rag-content changes needed** for the mechanism — the + `file_extractor`/`MetadataProcessor` seams suffice. Implication: the + implementation is fetch-stage + glue, low architectural risk. +- **`hermetic_build=True` is mandatory** for auth-gated sources, else every + URL is flagged unreachable (or worse, `drop` mode empties the corpus). +- **Read-only tokens are the realistic baseline**: even space creation was + 403-forbidden for a regular corporate user. The importer must require + nothing beyond read. +- **Governance**: Atlassian-instance policies (including Red Hat's own) + require registered bot/service accounts for team integrations — + the admin guide must say so (T3). +- **`doc_type="html"` does not auto-wire the HTMLReader** — the caller must + pass `file_extractor={".html": HTMLReader()}`; `required_exts` is also + needed to keep `manifest.json`/`state.json` out of the corpus. +- **Incidental bug**: generated `llama-stack.yaml` + `query_rag.py` + registration conflict (see Proposed incidental JIRAs). +- **Absolute embedding-model path** is baked into the generated config and + kv registry unless HF-id resolution is used (T7). +- **Incomplete git-LFS model checkout** fails late with an obscure + `model.safetensors not found` — a preflight check in the importer command + would save admins a full crawl. + +## External input needed + +- Whether OpenShift Lightspeed plans to consume this importer for its BYOK + image flow (affects how much weight the image-packaging path gets) — ask + the OLS PM via @sbunciak. + +## Background sections + +### Current BYOK state + +lightspeed-stack: operators declare stores under `byok_rag:` +(`src/models/config.py` `ByokRag`; faiss `db_path` or pgvector); +`src/llama_stack_configuration.py` enriches them into llama-stack +`run.yaml` (`VECTOR_IO_TEMPLATES` supports `inline::faiss` and +`remote::pgvector` only); retrieval fans out in +`src/utils/vector_search.py`. All vector access is mediated by the +llama-stack client — the service never opens DBs directly, and the +config-merge design (LCORE-836) keeps operator config backend-agnostic. +**No hot-reload**: nothing watches `db_path`; a changed DB needs a +restart. The customer workflow today is fully manual +(`docs/byok_guide.md`): prepare files → run rag-content → mount the +artifact. + +rag-content: a local-files framework — `SimpleDirectoryReader` + +per-extension readers (docling `HTMLReader` on main, `PDFReader` on the +LCORE-2091 branch) → `MarkdownNodeParser` (380/0) → embed → faiss/pgvector +in llama-index or llama-stack flavor → optional OCI packaging +(`--output-image`, artifact at `/rag/vector_db`). Chunk metadata carries +`docs_url`/`title` via `MetadataProcessor` (frontmatter `url` or +`url_function`). **No remote-source concept, no crawler, no scheduler +anywhere** — the fetch stage is greenfield, but everything downstream of +local files is reusable as-is (proven by the PoC). + +### Confluence API landscape + +- **Cloud**: v2 API for fast cursor-paginated listing; **CQL search and + `body.export_view` are v1-only** — a Cloud importer necessarily uses + both. DC: the v1-shaped surface + PATs. +- **CQL**: `space in (…) and type=page and lastmodified >= "yyyy/MM/dd + HH:mm"` is the delta primitive; minute granularity and user-timezone + interpretation ⇒ use an overlap buffer + version comparison + (deduplicates re-fetches). `content/search` caps result windows (50 with + body expand) ⇒ search for IDs, fetch bodies individually. +- **Deletions are invisible to CQL** ⇒ full ID enumeration diff each run + (cheap: no body expands). +- **Rate limits**: Cloud is points-based per tenant with 429 + + `Retry-After` (API-token Basic traffic currently exempt but unpublished + burst limits apply); DC limits are admin-configurable. Importer must + honor `Retry-After`. + +### Existing Confluence loaders + +| Tool | License | Fit | +|---|---|---| +| atlassian-python-api 4.x | Apache-2.0 | Recommended (T1): thin Cloud+DC client, CQL, attachment APIs | +| llama-index-readers-confluence | MIT | Full loader; heavy attachment deps; hides sync state | +| langchain-community ConfluenceLoader | MIT | Same underlying client; no incremental state; langchain dep | +| unstructured-ingest / Airbyte / RAGFlow / Elastic connectors | various | Platform-scale connectors; useful as sync-pattern references (Elastic's version-skip + ID-diff informs T8), not as dependencies | + +### Prior art: s2i PoC and OLS BYOK + +- **syedriko s2i PoC** (lightspeed-rag-content fork, `syedriko-s2i` + branch): Git-hosted Markdown → Shipwright BuildRun on OCP runs + `generate_embeddings_tool.py` → UBI9 image containing `/rag/vector_db` → + registry → OLS picks it up. The Confluence importer slots in as the + *document producer* upstream of exactly this build — which is why S4 + keeps Shipwright as the OLS-layer variant. +- **OLS BYOK today**: customer Markdown → `lightspeed-rag-tool` container → + `byok-image.tar` → registry → `OLSConfig.spec.ols.rag` + (`indexPath: /rag/vector_db`). Tech Preview. + +### Refresh patterns on Kubernetes + +| Pattern | Pros | Cons | +|---|---|---| +| CronJob → rebuild artifact → rollout (**recommended**) | Immutable, auditable; matches both LCS file and OLS image contracts; no service changes | Staleness = cron period; full-rebuild embed cost (mitigated by version-skip) | +| CronJob → live store on shared PVC | No restart | Needs hot-reload the service doesn't have; RWX storage; mutable artifact | +| Sidecar sync | Freshest | Embedding workload inside serving pods; per-replica duplication | +| Init container | Simple | Refresh only on restart; slow pod starts | + +## Glossary + +- **BYOK** — Bring Your Own Knowledge: customer-supplied RAG content + served alongside product docs. +- **export_view** — Confluence REST body representation: fully rendered + HTML with macros expanded. +- **CQL** — Confluence Query Language; server-side content search + (selection + delta detection). +- **Sync state** — per-page version numbers + last-sync watermark kept + between importer runs. diff --git a/docs/design/byok-confluence-import/byok-confluence-import.md b/docs/design/byok-confluence-import/byok-confluence-import.md new file mode 100644 index 000000000..52b6d3149 --- /dev/null +++ b/docs/design/byok-confluence-import/byok-confluence-import.md @@ -0,0 +1,284 @@ +# Feature design for LCORE-788: BYOK auto-import from Confluence + +| | | +|--------------------|-------------------------------------------| +| **Date** | 2026-07-09 | +| **Component** | lightspeed-core/rag-content (primary), lightspeed-core/lightspeed-stack (docs only) | +| **Authors** | Maxim Svistunov | +| **Feature** | [LCORE-788](https://redhat.atlassian.net/browse/LCORE-788) | +| **Spike** | [LCORE-2664](https://redhat.atlassian.net/browse/LCORE-2664) — [spike doc](byok-confluence-import-spike.md) | +| **Links** | [LCORE-256](https://redhat.atlassian.net/browse/LCORE-256) (Alternative BYOK), OCPSTRAT-2278 | + +## What + +An importer in `lightspeed-rag-content` that builds — and periodically +refreshes — a BYOK vector-store artifact directly from Confluence: it +crawls configured spaces via the Confluence REST API (Cloud and Data +Center), converts rendered page HTML to Markdown through the existing +docling pipeline, embeds and packages the store exactly like today's +manual flow, and preserves per-page Confluence URLs as citation metadata. +A documented Kubernetes CronJob wraps the importer for scheduled refresh; +Lightspeed Core Stack consumes the artifact through its existing +`byok_rag` configuration, unchanged. + +## Why + +Customers store their private documentation predominantly in Confluence. +Today's BYOK flow requires them to export/prepare files by hand, run the +rag-content tooling manually, and re-do all of it whenever content +changes. That is enough friction that BYOK content goes stale or never +gets built. This feature removes the manual preparation (point at spaces, +get a store) and the manual refresh (CronJob keeps it current, skipping +unchanged pages). + +## Requirements + +- **R1:** The importer fetches all pages of one or more configured + Confluence spaces and produces a directory of documents consumable by + the existing rag-content pipeline, with no manual preparation. +- **R2:** Confluence Cloud (email + API token, Basic) and Data Center + (Personal Access Token, Bearer) are both supported; credentials come + from environment variables (Kubernetes Secret), never CLI args or logs. +- **R3:** Selection is configurable: space keys (required), optional CQL + and/or label filter. +- **R4:** Every chunk in the built store carries the source page's title + and canonical Confluence URL, so answers can cite the page. +- **R5:** A single command performs fetch → build → artifact + (`llamastack-faiss` file by default; optional OCI image via the existing + `--output-image`). +- **R6:** Incremental refresh: a re-run against unchanged spaces performs + no page-body fetches and no re-embeddings; changed pages are re-imported; + deleted pages disappear from the rebuilt artifact. +- **R7:** The importer requires an explicit embedding model and produces a + path-portable artifact by default (HF-id resolution); the docs state the + build/runtime model-match requirement. +- **R8:** Failures are loud and actionable: auth errors (expired token), + permission denials, and rate limiting (429 honoring `Retry-After`) are + reported distinctly; a partial crawl never silently produces a truncated + store. +- **R9:** Reference automation and an admin guide document the + scheduled-refresh deployment in two shapes driving the same importer + container: a Kubernetes CronJob manifest (shared volume + rollout + trigger) and a podman-only Quadlet reference (`.container` + `.timer` + systemd units) for non-Kubernetes hosts. +- **R10:** lightspeed-stack requires no code or config-schema changes; its + BYOK guide documents the Confluence flow (spike Decision S5). + +## Use Cases + +- **U1:** As a Lightspeed admin, I want to point the importer at my + Confluence spaces and get a ready BYOK store, so that I don't prepare + documents by hand. +- **U2:** As a Lightspeed admin, I want a scheduled job to keep the store + current with Confluence edits and deletions, so that answers don't go + stale. +- **U3:** As a Lightspeed user, I want answers grounded in my company's + Confluence with links to the exact pages, so that I can verify and read + more. +- **U4:** As an OpenShift Lightspeed engineer, I want the importer usable + as a container step, so that the product's BYOK-image build (Shipwright) + can adopt it. + +## Architecture + +### Overview + +```text + lightspeed-rag-content (importer, offline) +┌──────────────────────────────────────────────────────────────────┐ +│ fetch stage (new) build stage (existing) │ +│ Confluence REST ──► pages/*.html ──► docling HTMLReader │ +│ Cloud v1+v2 / DC manifest.json │ Markdown │ +│ CQL delta, version state.json ▼ │ +│ skip, deletion diff MarkdownNodeParser (380/0) │ +│ │ embed (pinned model) │ +│ ▼ │ +│ faiss_store.db (+ llama-stack.yaml) │ +│ [optional --output-image OCI tar] │ +└──────────────────────────────────────────────────────────────────┘ + ▲ CronJob (scheduled) │ artifact on shared + │ credentials from Secret ▼ volume / registry + Confluence lightspeed-stack (unchanged) + (customer) byok_rag: db_path → rollout restart + → answers with page-URL citations +``` + +### Trigger mechanism + +Manual invocation (one-shot CLI / `podman run`) or the reference CronJob +on a schedule. There is no in-service trigger: the service picks up a new +artifact on restart/rollout (spike Decision S2; hot-reload deferred). + +### Storage / data model changes + +None to lightspeed-stack. New importer-side files kept next to the +output artifact: + +- `manifest.json` — per fetched file: page id, title, canonical URL, + version, space. +- `state.json` — last-sync watermark (UTC) + page-id → version map, + enabling incremental runs and deletion detection. + +### Configuration + +Importer configuration (CLI flags / env; no lightspeed-stack schema): + +```yaml +# conceptual shape; concrete form is CLI flags + env vars +confluence: + url: https://example.atlassian.net # or DC base URL + auth: cloud_token | dc_pat # secrets via env: CONFLUENCE_EMAIL / CONFLUENCE_TOKEN + spaces: [DOCS, RUNBOOKS] + cql_filter: 'label = "public-docs"' # optional +embedding: + model_name: sentence-transformers/all-mpnet-base-v2 # required, HF-id resolved +output: + dir: /data/vector_db + index: byok-confluence + image: null # optional OCI tar path +``` + +### Error handling + +- Auth failure (401/403): exit non-zero with a message naming the auth + mode and the likely cause (expired Cloud token ≤ 1 yr, revoked PAT). +- Rate limiting (429): honor `Retry-After`, bounded retries, then fail + loudly. +- Per-page conversion failure: log and continue (page skipped, listed in + the run summary); abort only if a configurable failure ratio is + exceeded — a partial crawl must never silently ship a truncated store + (R8). +- Preflight: verify the embedding model directory is complete + (`model.safetensors` present) *before* crawling (PoC finding). + +### Security considerations + +- Read-only Confluence access is sufficient and is the required posture; + docs recommend a registered service/bot account (instance policies — + including Red Hat's own — commonly mandate it). +- Secrets only via env/Secret; never logged, never in CLI args (R2). +- **Permission flattening**: imported content is served to every LCS user + regardless of per-page Confluence restrictions. The admin guide carries + a prominent warning to import only spaces whose audience matches the + assistant's audience. +- Crawled corporate content lands in artifacts (store, optionally image): + treat them with the same confidentiality as the source space. + +## Acceptance test surface + +| Req | Observable behavior | Verified by | +|-----|---------------------|-------------| +| R1 | One command against a (mock) Confluence yields a queryable store containing all space pages | e2e (mock Confluence), integration | +| R2 | Cloud-token and DC-PAT auth shapes both work; secrets absent from logs/argv | unit (fixtures), e2e | +| R3 | Space/CQL/label selection imports exactly the matching pages | unit, e2e | +| R4 | Query responses cite page title + canonical URL | e2e through lightspeed-stack | +| R5 | Fetch→build→artifact in one invocation; optional image tar loadable | integration | +| R6 | Re-run, no changes: 0 body fetches, 0 re-embeds. Edit: page re-imported. Delete: gone after rebuild | e2e (mock) | +| R7 | Missing/incomplete model fails preflight; artifact loads from a different absolute path | integration | +| R8 | 401/403/429 produce distinct, actionable, non-zero outcomes | unit (fixtures) | +| R9 | Reference CronJob applies cleanly; scheduled run refreshes the artifact; Quadlet timer does the same on a podman-only host | e2e (kind/CRC) or documented manual verification | +| R10 | lightspeed-stack serves the artifact with existing `byok_rag` config | e2e | + +## Aspect-specific concerns + +### Latency and Cost + +No serving-path impact (offline importer). Importer cost per run ≈ +1 listing call per ~100 pages + 1 body call per changed page + embedding +of changed pages only (R6). Full first crawl of a large space is +API-bound; Cloud rate limits make very large spaces a multi-minute, not +multi-hour, operation at default limits. + +### Failure modes + +- Confluence v1 API deprecation on Cloud would break `export_view` + fetching (spike Decision T2 risk) — isolated inside the fetch stage. +- Macro-heavy pages (Jira tables, drawio) degrade to whatever + `export_view` renders; content inside unrendered macros is lost + silently — documented limitation. +- Embedding-model drift between importer and service returns silently + irrelevant results (R7 warning; not detectable at runtime today). +- CronJob rebuilds while the service reads the mounted file: the reference + manifest writes to a versioned path and switches atomically + (symlink/rename), never in-place. + +### Rollout / deployment plan + +Admin-driven: deploy the scheduler (k8s CronJob, or Quadlet systemd timer +on podman-only hosts) → first full crawl → point `byok_rag` at the +artifact → restart. Refresh = scheduled rebuild + restart trigger +(documented `kubectl rollout restart`, `systemctl restart`, or +operator-specific equivalent). Rollback = repoint to the previous +versioned artifact and restart. + +## Implementation Suggestions + +### Key files and insertion points + +| File | What to do | +|------|------------| +| `src/lightspeed_rag_content/confluence/__init__.py` (rag-content, new) | Package docstring | +| `src/lightspeed_rag_content/confluence/fetcher.py` (new) | Crawl/CQL/state logic on `atlassian-python-api`; writes pages + manifest + state | +| `src/lightspeed_rag_content/confluence/metadata.py` (new) | `ConfluenceMetadataProcessor(MetadataProcessor)`: `url_function`/`get_file_title` from manifest; `hermetic_build=True` | +| `src/lightspeed_rag_content/confluence/__main__.py` (new) | CLI (`fetch`, `import` subcommands), mirroring `html/__main__.py` | +| `scripts/import_confluence.py` (new) | One-shot orchestration: fetch → `DocumentProcessor` → optional `--output-image` | +| `Containerfile` (modify) | Include the importer entrypoint in the tool image | +| `examples/confluence-cronjob.yaml` (new) | Reference CronJob manifest | +| `examples/confluence-import.container` + `.timer` (new) | Quadlet reference for podman-only hosts | +| `docs/` (rag-content, new page) | Admin automation guide | +| `docs/byok_guide.md` (lightspeed-stack, modify) | Confluence walkthrough, model-match + restart + permission warnings | +| `tests/confluence/` (rag-content, new) | Unit tests on recorded Cloud/DC fixtures; integration test fixture-HTML → artifact | + +### Insertion point detail + +The pipeline is consumed as-is: +`DocumentProcessor.process(folder, metadata=ConfluenceMetadataProcessor(...), +required_exts=[".html"], file_extractor={".html": HTMLReader()})` then +`save(index, output)`. Two PoC-verified gotchas: `doc_type="html"` sets +only the node parser — the caller must wire `HTMLReader` via +`file_extractor`; and `required_exts` must exclude `manifest.json` / +`state.json` from the corpus. + +### Config pattern + +Importer flags extend `utils.get_common_arg_parser()` (rag-content +convention) rather than introducing a config file; secrets exclusively via +env vars. No lightspeed-stack Pydantic changes (R10). + +### Test patterns + +- Unit: recorded REST fixtures for Cloud and DC shapes (pagination, + CQL windows, 429 with `Retry-After`, 401/403). +- Integration: fixture HTML dir → artifact; assert chunk metadata + (title/URL), version-skip behavior, deletion diff. +- e2e (lightspeed-stack): mock Confluence REST fixture (v1 listing, + `export_view`, CQL) driving import → serve → cite; edit and delete + scenarios per R6. Live Confluence is explicitly not required by CI. + +## Open Questions for Future Work + +- Hot-reload of a refreshed store without restart — deferred from spike + Decision S2; candidate epic under LCORE-256. +- Attachments (PDF/Office via byok-pdf pipeline) — deferred from spike + Decision T5. +- Non-Confluence sources (SharePoint/Git/web) reusing the fetch-stage + seam — out of scope per LCORE-788 PM scope reduction (2026-06-15). +- OLS/Shipwright adoption of the importer container — pending External + input (spike doc); affects image-path priority. +- Confluence v1 API longevity on Cloud (export_view/CQL) — monitor; + isolated in `fetcher.py` (spike Decision T2). + +## Changelog + +| Date | Change | Reason | +|------|--------|--------| +| 2026-07-09 | Initial version | LCORE-2664 spike | +| 2026-07-21 | R9 + rollout + key files: added podman/Quadlet reference alongside the k8s CronJob | PM review on spike PR (spike Decision S4 amended A→D) | + +## Appendix A: PoC evidence summary + +Validated on Confluence Cloud (redhat.atlassian.net, 20-page space, +read-only token): full crawl → unmodified pipeline → 3.8 MB +`llamastack-faiss` store; correct top-3 retrieval with page-URL citations; +incremental re-run touched 2 API endpoints and re-embedded nothing. +Details and findings: [spike doc, PoC results](byok-confluence-import-spike.md#poc-results). diff --git a/docs/design/byok-confluence-import/poc-results/README.md b/docs/design/byok-confluence-import/poc-results/README.md new file mode 100644 index 000000000..fefb4f693 --- /dev/null +++ b/docs/design/byok-confluence-import/poc-results/README.md @@ -0,0 +1,66 @@ +# PoC results: BYOK auto-import from Confluence (LCORE-2664) + +Evidence for the spike doc's PoC findings. Raw crawled content is **not** +included: the PoC ran against the RHAT space on redhat.atlassian.net +(Red Hat-internal content); this directory carries sanitized stats and +excerpts only. + +## Design + +- **Target**: Confluence Cloud (redhat.atlassian.net), space `RHAT` + (20 pages), read-only API token of a regular user. +- **Fetch**: v1 REST `body.export_view` (rendered HTML), one HTML file per + page + `manifest.json` (id → title, canonical URL, version) + + `state.json` (sync watermark, per-page versions). +- **Build**: existing rag-content pipeline — docling `HTMLReader`, + `MarkdownNodeParser`, chunk 380/0, `all-mpnet-base-v2` (768-dim), + `llamastack-faiss` output. +- **Verify**: `vector_io.query` via llama-stack library client, top-3. +- **Incremental**: run 2 with `--incremental` — CQL + `lastmodified >= watermark − 10min` + full ID enumeration (deletion diff) + + version comparison to skip unchanged pages. + +## Results + +| Step | Outcome | +|---|---| +| Full crawl (run 1) | 20/20 pages fetched, ~1 API call/page + 1 listing call | +| DB build | `faiss_store.db` 3.8 MB, no page failed conversion | +| Retrieval | Top-3 chunks on a policy question: correct pages, scores 2.25/1.65/1.60 | +| Citations | `title` + canonical page URL present on every chunk (see `retrieval-evidence.txt`) | +| Incremental (run 2) | `fetched=0 unchanged=20 deleted=0` — 2 API calls total instead of 21 | +| Conversion quality | Headings/bold/nested lists/expand-macros clean (see `conversion-sample.md` excerpt) | + +## Findings (carried into the spike doc) + +1. **The mechanism works end-to-end** with zero changes to rag-content + library code — only a fetch stage and a `MetadataProcessor` subclass. +2. **Citations flow**: per-page Confluence URLs land in chunk metadata via + `url_function`; `hermetic_build=True` is required (page URLs are + auth-gated, reachability pings would mark all unreachable). +3. **Read-only is all you get**: space creation 403-forbidden for regular + users on redhat.atlassian.net; the importer must not assume any write + permission. Update/delete sync paths are therefore implemented but + validated logically, not live (no writable page available). +4. **Governance**: Red Hat's own Atlassian policy (retrieved by the PoC + itself) requires team integrations to use registered bot/service + accounts, not personal tokens — auth docs must say so. +5. **Incidental bug**: the generated `llama-stack.yaml` + + `scripts/query_rag.py` fail out-of-the-box — re-registering the + vector store conflicts with the registration persisted inside + `faiss_store.db` (`provider_resource_id`/`vector_store_name`: None vs + set). Worked around by dropping `registered_resources.vector_stores` + and querying the persisted registration. +6. **Portability gotcha confirmed live**: the built DB bakes the absolute + embedding-model path into `llama-stack.yaml` and the kv registry. +7. **Incomplete-LFS gotcha**: an embeddings-model directory without + `model.safetensors` fails only at build time with an obscure error. + +## Implications + +- Recommended design (fetch stage + existing pipeline) is validated; + no design change needed. +- Incremental sync as "skip unchanged re-embeds, full rebuild output" + is cheap and works; live update/delete validation moves to the e2e + test ticket (needs a writable test instance/mock). +- Finding 5 becomes a proposed incidental JIRA. diff --git a/docs/design/byok-confluence-import/poc-results/conversion-sample.md b/docs/design/byok-confluence-import/poc-results/conversion-sample.md new file mode 100644 index 000000000..f00b5c747 --- /dev/null +++ b/docs/design/byok-confluence-import/poc-results/conversion-sample.md @@ -0,0 +1,31 @@ +# Conversion quality evidence (sanitized) + +Largest crawled page ("Atlassian Cloud FAQ", export_view HTML → docling → +Markdown, 0.14 s). Body text redacted (internal); structure shown verbatim: + +``` +# Atlassian Cloud FAQ + +## Review the answers to common questions you may have about our Cloud environment. + +##### **What is Atlassian Cloud?** + +[paragraph — clean prose, no markup artifacts] + +##### **What is the timeline and when is the migration?** + +**[bold lead sentence]** + +- [nested list, 2 levels, rendered correctly] + - [timezone conversion sub-items] +``` + +Observations: + +- Headings, bold, links, and 2-level nested lists all convert cleanly. +- Confluence expand-macros arrive already expanded in `export_view` + (their content is present as regular headings/paragraphs). +- No letter-spaced-font artifacts (the known docling weakness with + Confluence "Export to PDF" does not apply to the HTML path). +- Minor: one stray list item rendered as an empty `-` bullet + (trailing empty list node in the source HTML) — harmless for chunking. diff --git a/docs/design/byok-confluence-import/poc-results/crawl-runs.txt b/docs/design/byok-confluence-import/poc-results/crawl-runs.txt new file mode 100644 index 000000000..27326c174 --- /dev/null +++ b/docs/design/byok-confluence-import/poc-results/crawl-runs.txt @@ -0,0 +1,5 @@ +# Run 1 (full crawl) summary line: +[full] spaces=['RHAT'] fetched=20 unchanged=0 deleted=0 total_on_disk=20 + +# Run 2 (incremental, zero changes): +[incremental] spaces=['RHAT'] fetched=0 unchanged=20 deleted=0 total_on_disk=20 diff --git a/docs/design/byok-confluence-import/poc-results/retrieval-evidence.txt b/docs/design/byok-confluence-import/poc-results/retrieval-evidence.txt new file mode 100644 index 000000000..2c6bd2dd7 --- /dev/null +++ b/docs/design/byok-confluence-import/poc-results/retrieval-evidence.txt @@ -0,0 +1,20 @@ +Query: "What is the policy for using API scripts and bots in the Atlassian Cloud environment?" +Store: llamastack-faiss, top_k=3, mode=vector + +chunk 1 | score=2.2482 + title: APIs, Scripts, and Bots: Policy for Cloud Environment + docs_url: https://redhat.atlassian.net/wiki/spaces/RHAT/pages/303600805/APIs+Scripts+and+Bots+Policy+for+Cloud+Environment + text: "## Policy Statement\n\n- Personal experimentation using Atlassian REST APIs may use API tokens in a user's own account.\n- Team-supporting integrations must use a registered bot user or approved service account, not a personal token. [...]" + +chunk 2 | score=1.6468 + title: APIs, Scripts, and Bots: Policy for Cloud Environment + docs_url: (same page as chunk 1) + text: [Purpose section — redacted, internal] + +chunk 3 | score=1.5955 + title: API, Script, and Bot Policy - public + docs_url: https://redhat.atlassian.net/wiki/spaces/RHAT/pages/297468192/API+Script+and+Bot+Policy+-+public + text: "## Policy Statement\n\n- Users using Atlassian API, bots, or scripts against Jira or Confluence must have a bot user. All bots need to be registered with the PME team. [...]" + +All three chunks: correct pages for the query; every chunk carries title + +canonical per-page Confluence URL in metadata (citation path proven). diff --git a/docs/design/byok-confluence-import/poc/README.md b/docs/design/byok-confluence-import/poc/README.md new file mode 100644 index 000000000..61b3d95bb --- /dev/null +++ b/docs/design/byok-confluence-import/poc/README.md @@ -0,0 +1,51 @@ +# PoC: BYOK auto-import from Confluence (LCORE-2664 spike) + +Validates the core mechanism recommended by the spike doc: fetch Confluence +pages as rendered HTML via the REST API, feed them to the existing +`lightspeed-rag-content` pipeline (docling `HTMLReader` → Markdown → +`MarkdownNodeParser` → embeddings), produce a `llamastack-faiss` vector DB, +and query it with correct per-page citations. Also validates incremental +change detection (CQL `lastmodified` + page-version comparison). + +Not production code: credentials come from the developer's Jira CLI +credentials file, there is no retry/backoff, no attachments, no config +surface. + +## Files + +| File | Purpose | +|---|---| +| `confluence_fetch.py` | Fetch stage: full + incremental crawl, manifest, state | +| `confluence_processor.py` | `custom_processor.py`-pattern build driver with `ConfluenceMetadataProcessor` | +| `confluence_test_page.py` | Test-space/page helper (unused in the end: space creation is 403-forbidden on redhat.atlassian.net; kept as evidence of the read-only constraint) | + +## Repro + +Prereqs: `~/.config/jira/credentials.json` (email + API token for +redhat.atlassian.net), a checkout of `lightspeed-rag-content` with its uv +venv, and a *complete* local copy of the embedding model +(`model.safetensors` present — an LFS-less clone will fail). + +```bash +RUN_DIR=/tmp/poc-run # fetched content is Red Hat internal; keep out of the repo +POC_DIR=docs/design/byok-confluence-import/poc + +# 1. Full crawl (run 1) +python3 $POC_DIR/confluence_fetch.py --spaces RHAT --out $RUN_DIR/pages + +# 2. Build the vector DB (from the rag-content repo root) +cd ../rag-content +uv run python $POC_DIR/confluence_processor.py \ + -f $RUN_DIR/pages -o $RUN_DIR/vector_db -i byok-confluence-poc \ + -md embeddings_model -mn sentence-transformers/all-mpnet-base-v2 \ + --vector-store-type llamastack-faiss + +# 3. Query (workaround script; scripts/query_rag.py hits a +# registered_resources conflict — see incidental findings in the spike doc) +uv run python query_workaround.py "your question here" 3 + +# 4. Incremental crawl (run 2) — re-uses state.json, fetches only changes +python3 $POC_DIR/confluence_fetch.py --spaces RHAT --out $RUN_DIR/pages --incremental +``` + +Results and evidence: see `../poc-results/`. diff --git a/docs/design/byok-confluence-import/poc/confluence_fetch.py b/docs/design/byok-confluence-import/poc/confluence_fetch.py new file mode 100644 index 000000000..4be7ed4b2 --- /dev/null +++ b/docs/design/byok-confluence-import/poc/confluence_fetch.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""PoC: fetch Confluence pages as rendered HTML for the BYOK RAG pipeline. + +Fetches all pages from the given Confluence spaces via the v1 REST API +(`body.export_view` = rendered HTML), writes one HTML file per page plus a +`manifest.json` (page id -> title/url/version) consumed by the metadata +processor, and a `state.json` sync watermark. + +Modes: + - Full crawl (default, or no state file): enumerate every page in each space. + - Incremental (`--incremental`, requires state file): CQL `lastmodified >=` + watermark to find changed pages, full ID enumeration to detect deletions. + +PoC shortcuts (not production shape): credentials read from the Jira CLI +credentials file; no retry/backoff beyond honoring nothing; no attachments. +""" + +import argparse +import base64 +import datetime +import json +import pathlib +import sys +import urllib.parse +import urllib.request + +CRED_FILE = pathlib.Path.home() / ".config/jira/credentials.json" +BASE = "https://redhat.atlassian.net" +EXPAND = "body.export_view,version,space" +# Overlap buffer so CQL minute granularity / timezone skew can't miss edits; +# version comparison deduplicates re-fetches. +WATERMARK_OVERLAP_MIN = 10 + + +def _auth_header() -> str: + creds = json.loads(CRED_FILE.read_text()) + token = base64.b64encode(f"{creds['email']}:{creds['token']}".encode()).decode() + return f"Basic {token}" + + +def api_get(path: str, params: dict | None = None) -> dict: + url = f"{BASE}{path}" + if params: + url += "?" + urllib.parse.urlencode(params) + req = urllib.request.Request( + url, headers={"Authorization": _auth_header(), "Accept": "application/json"} + ) + with urllib.request.urlopen(req, timeout=60) as resp: + return json.load(resp) + + +def list_space_page_ids(space: str) -> dict[str, int]: + """Enumerate all page ids (id -> version) in a space, no bodies.""" + ids: dict[str, int] = {} + start = 0 + while True: + res = api_get( + "/wiki/rest/api/content", + {"spaceKey": space, "type": "page", "limit": 100, "start": start, + "expand": "version"}, + ) + for page in res.get("results", []): + ids[page["id"]] = page["version"]["number"] + if len(res.get("results", [])) < 100: + return ids + start += 100 + + +def fetch_page(page_id: str) -> dict: + return api_get(f"/wiki/rest/api/content/{page_id}", {"expand": EXPAND}) + + +def changed_page_ids(spaces: list[str], since_utc: datetime.datetime) -> set[str]: + """CQL delta: page ids modified since the watermark (minus overlap).""" + since = since_utc - datetime.timedelta(minutes=WATERMARK_OVERLAP_MIN) + space_list = ",".join(f'"{s}"' for s in spaces) + cql = ( + f"space in ({space_list}) and type=page" + f' and lastmodified >= "{since.strftime("%Y/%m/%d %H:%M")}"' + ) + ids: set[str] = set() + start = 0 + while True: + res = api_get( + "/wiki/rest/api/content/search", + {"cql": cql, "limit": 100, "start": start}, + ) + ids.update(r["id"] for r in res.get("results", [])) + if len(res.get("results", [])) < 100: + return ids + start += 100 + + +def write_page(page: dict, out_dir: pathlib.Path) -> dict: + """Write one page as HTML; return its manifest entry.""" + title = page["title"] + html_body = page["body"]["export_view"]["value"] + doc = ( + "\n\n\n\n" + f"{title}\n\n\n" + f"

{title}

\n{html_body}\n\n\n" + ) + filename = f"{page['id']}.html" + (out_dir / filename).write_text(doc, encoding="utf-8") + return { + "id": page["id"], + "title": title, + "url": BASE + "/wiki" + page["_links"]["webui"], + "version": page["version"]["number"], + "when": page["version"]["when"], + "space": page["space"]["key"], + "file": filename, + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--spaces", nargs="+", required=True, help="Space keys to crawl") + parser.add_argument("--out", required=True, help="Output directory for HTML files") + parser.add_argument("--incremental", action="store_true", + help="Use CQL delta + deletion diff against state file") + args = parser.parse_args() + + out_dir = pathlib.Path(args.out) + out_dir.mkdir(parents=True, exist_ok=True) + manifest_path = out_dir / "manifest.json" + state_path = out_dir / "state.json" + + manifest: dict[str, dict] = ( + json.loads(manifest_path.read_text()) if manifest_path.exists() else {} + ) + state: dict = json.loads(state_path.read_text()) if state_path.exists() else {} + run_started = datetime.datetime.now(datetime.timezone.utc) + + known_versions: dict[str, int] = state.get("pages", {}) + stats = {"fetched": 0, "unchanged": 0, "deleted": 0, "api_calls_saved": 0} + + # Current inventory (id -> version) across all spaces: needed for deletion + # detection in both modes, and as the fetch list in full mode. + inventory: dict[str, int] = {} + for space in args.spaces: + inventory.update(list_space_page_ids(space)) + + if args.incremental and state.get("last_sync"): + watermark = datetime.datetime.fromisoformat(state["last_sync"]) + candidates = changed_page_ids(args.spaces, watermark) + # Also anything present in inventory but unknown to state (new pages + # CQL may have missed due to timezone skew). + candidates.update(pid for pid in inventory if pid not in known_versions) + to_fetch = { + pid for pid in candidates + if pid in inventory and inventory[pid] != known_versions.get(pid) + } + stats["unchanged"] = len(inventory) - len(to_fetch) + stats["api_calls_saved"] = stats["unchanged"] + else: + to_fetch = set(inventory) + + for pid in sorted(to_fetch): + entry = write_page(fetch_page(pid), out_dir) + manifest[entry["file"]] = entry + stats["fetched"] += 1 + print(f" fetched [{entry['space']}] v{entry['version']} {entry['title']}") + + # Deletion diff: anything on disk whose page id vanished from inventory. + for filename in list(manifest): + pid = manifest[filename]["id"] + if pid not in inventory: + (out_dir / filename).unlink(missing_ok=True) + del manifest[filename] + stats["deleted"] += 1 + print(f" deleted {filename}") + + manifest_path.write_text(json.dumps(manifest, indent=2)) + state_path.write_text(json.dumps({ + "last_sync": run_started.isoformat(), + "spaces": args.spaces, + "pages": inventory, + }, indent=2)) + + mode = "incremental" if args.incremental and state.get("last_sync") else "full" + print(f"[{mode}] spaces={args.spaces} fetched={stats['fetched']} " + f"unchanged={stats['unchanged']} deleted={stats['deleted']} " + f"total_on_disk={len(manifest)}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/design/byok-confluence-import/poc/confluence_processor.py b/docs/design/byok-confluence-import/poc/confluence_processor.py new file mode 100644 index 000000000..a6c10feea --- /dev/null +++ b/docs/design/byok-confluence-import/poc/confluence_processor.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +"""PoC: build a BYOK vector DB from Confluence-fetched HTML pages. + +Follows the documented `custom_processor.py` pattern from +lightspeed-rag-content: a Confluence-aware ``MetadataProcessor`` that +resolves each page's canonical URL and title from the ``manifest.json`` +written by ``confluence_fetch.py``, driving the existing +``DocumentProcessor`` with ``doc_type="html"`` (docling HTMLReader -> +Markdown -> MarkdownNodeParser chunks). + +Run from the rag-content repo root so its venv and embeddings model resolve: + + uv run python -f -o -i \ + -md embeddings_model -mn sentence-transformers/all-mpnet-base-v2 +""" + +import json +import os + +from lightspeed_rag_content import utils +from lightspeed_rag_content.document_processor import DocumentProcessor +from lightspeed_rag_content.html.html_reader import HTMLReader +from lightspeed_rag_content.metadata_processor import MetadataProcessor + + +class ConfluenceMetadataProcessor(MetadataProcessor): + """Resolve per-page URL and title from the fetch-stage manifest. + + ``hermetic_build=True`` because Confluence page URLs are auth-gated: + an unauthenticated reachability ping would mark every URL unreachable. + """ + + def __init__(self, manifest_path: str): + super().__init__(hermetic_build=True) + with open(manifest_path, encoding="utf-8") as fp: + self.manifest = json.load(fp) + + def _entry(self, file_path: str) -> dict: + return self.manifest.get(os.path.basename(file_path), {}) + + def url_function(self, file_path: str) -> str: + return self._entry(file_path).get("url", os.path.basename(file_path)) + + def get_file_title(self, file_path: str) -> str: + # HTML files have no frontmatter; first-line extraction would return + # raw markup. The manifest carries the real Confluence page title. + return self._entry(file_path).get("title", os.path.basename(file_path)) + + +if __name__ == "__main__": + parser = utils.get_common_arg_parser() + args = parser.parse_args() + + metadata_processor = ConfluenceMetadataProcessor( + os.path.join(args.folder, "manifest.json") + ) + document_processor = DocumentProcessor( + chunk_size=args.chunk, + chunk_overlap=args.overlap, + model_name=args.model_name, + embeddings_model_dir=args.model_dir, + num_workers=args.workers, + vector_store_type=args.vector_store_type, + doc_type="html", + ) + document_processor.process( + args.folder, + metadata=metadata_processor, + required_exts=[".html"], + file_extractor={".html": HTMLReader()}, + ) + document_processor.save(args.index, args.output) diff --git a/docs/design/byok-confluence-import/poc/confluence_test_page.py b/docs/design/byok-confluence-import/poc/confluence_test_page.py new file mode 100644 index 000000000..451d929c6 --- /dev/null +++ b/docs/design/byok-confluence-import/poc/confluence_test_page.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""PoC helper: manage the editable test page for the incremental-sync demo. + +Subcommands: + create - ensure the private PoC space exists and create the test page + containing a distinctive retrieval probe ("codeword"). + edit - update the test page in place, changing the codeword, so the + next incremental crawl sees exactly one modified page. + delete-page / delete-space - cleanup after the PoC. + +The space is created via `POST /wiki/rest/api/space/_private`, which is +visible only to the creating user. +""" + +import argparse +import base64 +import json +import pathlib +import sys +import urllib.error +import urllib.request + +CRED_FILE = pathlib.Path.home() / ".config/jira/credentials.json" +BASE = "https://redhat.atlassian.net" +SPACE_KEY = "LCORE2664POC" +SPACE_NAME = "LCORE-2664 Confluence import PoC (temporary)" +PAGE_TITLE = "LCORE-2664 incremental sync test page" + +CODEWORD_V1 = "AMBER-FALCON" +CODEWORD_V2 = "COBALT-HERON" + + +def _auth_header() -> str: + creds = json.loads(CRED_FILE.read_text()) + token = base64.b64encode(f"{creds['email']}:{creds['token']}".encode()).decode() + return f"Basic {token}" + + +def api(method: str, path: str, body: dict | None = None) -> dict: + req = urllib.request.Request( + f"{BASE}{path}", + method=method, + headers={ + "Authorization": _auth_header(), + "Accept": "application/json", + "Content-Type": "application/json", + }, + data=json.dumps(body).encode() if body is not None else None, + ) + with urllib.request.urlopen(req, timeout=60) as resp: + raw = resp.read() + return json.loads(raw) if raw else {} + + +def page_body(codeword: str, note: str) -> str: + return ( + f"

This page exists solely for the LCORE-2664 spike PoC " + f"(BYOK auto-import from Confluence). {note}

" + f"

The current secret codeword for retrieval testing is " + f"{codeword}. Remember the codeword {codeword}.

" + f"

Purpose

Validates that an incremental crawl picks up " + f"page edits via CQL lastmodified and page version numbers.

" + ) + + +def ensure_space() -> None: + try: + api("GET", f"/wiki/rest/api/space/{SPACE_KEY}") + print(f"space {SPACE_KEY} already exists") + except urllib.error.HTTPError as err: + if err.code != 404: + raise + api("POST", "/wiki/rest/api/space/_private", + {"key": SPACE_KEY, "name": SPACE_NAME}) + print(f"created private space {SPACE_KEY}") + + +def find_page() -> dict | None: + res = api("GET", f"/wiki/rest/api/content?spaceKey={SPACE_KEY}" + f"&type=page&expand=version&limit=25") + for page in res.get("results", []): + if page["title"] == PAGE_TITLE: + return page + return None + + +def cmd_create() -> None: + ensure_space() + if find_page(): + print("test page already exists") + return + page = api("POST", "/wiki/rest/api/content", { + "type": "page", + "title": PAGE_TITLE, + "space": {"key": SPACE_KEY}, + "body": {"storage": { + "value": page_body(CODEWORD_V1, "Initial revision."), + "representation": "storage", + }}, + }) + print(f"created page id={page['id']} codeword={CODEWORD_V1}") + + +def cmd_edit() -> None: + page = find_page() + if not page: + sys.exit("test page not found; run create first") + new_version = page["version"]["number"] + 1 + api("PUT", f"/wiki/rest/api/content/{page['id']}", { + "type": "page", + "title": PAGE_TITLE, + "version": {"number": new_version}, + "body": {"storage": { + "value": page_body(CODEWORD_V2, "EDITED revision for the delta demo."), + "representation": "storage", + }}, + }) + print(f"edited page id={page['id']} -> v{new_version} codeword={CODEWORD_V2}") + + +def cmd_delete_page() -> None: + page = find_page() + if not page: + print("test page already gone") + return + api("DELETE", f"/wiki/rest/api/content/{page['id']}") + print(f"deleted page id={page['id']}") + + +def cmd_delete_space() -> None: + api("DELETE", f"/wiki/rest/api/space/{SPACE_KEY}") + print(f"deletion of space {SPACE_KEY} requested (async long task)") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("command", + choices=["create", "edit", "delete-page", "delete-space"]) + args = parser.parse_args() + {"create": cmd_create, "edit": cmd_edit, + "delete-page": cmd_delete_page, "delete-space": cmd_delete_space}[args.command]() + + +if __name__ == "__main__": + main() diff --git a/docs/design/byok-confluence-import/poc/query_workaround.py b/docs/design/byok-confluence-import/poc/query_workaround.py new file mode 100644 index 000000000..14263e87e --- /dev/null +++ b/docs/design/byok-confluence-import/poc/query_workaround.py @@ -0,0 +1,46 @@ +"""Query the PoC vector DB, working around the registered_resources conflict. + +The store is already registered inside faiss_store.db's kv registry; we drop +the run.yaml re-registration and query the persisted registration directly. +""" + +import os +import sys +import tempfile + +import yaml + +DB_DIR = os.path.dirname(os.path.abspath(__file__)) + "/vector_db" +QUERY = sys.argv[1] +TOP_K = int(sys.argv[2]) if len(sys.argv) > 2 else 3 + +tmp_dir = tempfile.TemporaryDirectory(prefix="ls-rag-") +os.environ["LLAMA_STACK_CONFIG_DIR"] = tmp_dir.name + +with open(f"{DB_DIR}/llama-stack.yaml", encoding="utf-8") as fp: + cfg = yaml.safe_load(fp) + +vs_entries = cfg["registered_resources"].pop("vector_stores") +vector_store_id = vs_entries[0]["vector_store_id"] + +cfg_file = os.path.join(tmp_dir.name, "llama-stack.yaml") +with open(cfg_file, "w", encoding="utf-8") as fp: + yaml.safe_dump(cfg, fp) + +from llama_stack.core.library_client import LlamaStackAsLibraryClient # noqa: E402 + +with LlamaStackAsLibraryClient(cfg_file) as client: + res = client.vector_io.query( + vector_store_id=vector_store_id, + query=QUERY, + params={"max_chunks": TOP_K, "mode": "vector", "score_threshold": 0}, + ) + print(f"\nQUERY: {QUERY}") + print(f"chunks retrieved: {len(res.chunks)}") + for i, (chunk, score) in enumerate(zip(res.chunks, res.scores)): + meta = chunk.metadata or {} + text = chunk.content if isinstance(chunk.content, str) else str(chunk.content) + print(f"\n--- chunk {i + 1} | score={score:.4f}") + print(f" title: {meta.get('title')}") + print(f" docs_url: {meta.get('docs_url')}") + print(f" text: {text[:300]!r}")