Pull docs from the website repo on a schedule

[dividedmind]

Replaces the push-based docs sync from getappmap/applandinc.github.io with a scheduled pull, and cleans up the corpus drift the old mechanism accumulated.

Started from a CI failure on a docs sync PR — sequenceDiagram.spec.ts dying with No usable sandbox! from Chromium. That turned out to be a symptom, not a bug: the PNG test has been skipped and --no-sandbox has been passed since 2025-01-08. The sync PR was running a tree frozen at 2024-12-20.

Why

The website repo owns the content of docs/, which is not repo documentation — it is the help corpus that appmap navie help indexes (packages/cli/src/cmds/navie/help.ts) and that packages/cli bundles into its published npm tarball and native binaries. Until now the website pushed updates here: a workflow there rsynced _docs and opened a PR against the next-doc staging branch on every push to master.

Three things were wrong with that.

The base was never refreshed. next-doc's code is frozen at 82d7ef9f1 (2024-12-20, chore(release): 3.180.1); the only 12 commits since are doc syncs and main has never been merged in — 477 commits behind. create-pull-request cuts its branch from the base, not from the checked-out ref, so every sync PR carried that Dec-2024 tree and ran the full CI suite against it. (Proof it used the base: had the head branch been main-based, merging #2370 into next-doc in July 2026 would have pulled main's 477 commits in. It didn't.)

Push-triggered and fire-and-forget. Real sync events: several in Dec 2024, then 2025-01-03, 01-24, 02-07 — then nothing until 2026-07-02. A 17-month gap during which every shipped CLI carried a stale help corpus. dfa7ef0f7 ("Manually sync the docs since the PR didn't open") shows it failed silently, and structurally it had to: the job ran in the website repo, so failures surfaced to people who don't own the consumer.

The rsync was not a mirror. rsync -av --delete _docs/* docs — the shell expands the glob, so docs/ itself is never a synced directory, only its children are. --delete therefore pruned inside subdirectories that still existed upstream but never removed top-level entries the website had retired. docs/ had grown to 95 markdown files against 57 upstream.

That last one was a product bug, not hygiene. docs/navie/{using-navie,bring-your-own-model,how-navie-works,how-to-open-navie}.md are the superseded ancestors of upstream using-navie-ai/{navie-workflow,choose-llm-provider,navie-architecture,navie-opening}.md, and docs/guides/ and docs/integrations/ moved under docs/reference/ — both generations sat in the lunr index. Since help.ts stamps every snippet with a https://appmap.io/docs/<path> permalink, Navie could answer from documentation retired ~18 months ago and cite URLs that no longer resolve.

What changed

Commit
c7c59f353	ci: modernize deprecated workflow syntax and action versions
02a0b1d16	feat: pull docs from the website repo on a schedule
decb4bd8b	chore: sync docs from applandinc.github.io
829e80e48	ci: drop the vestigial yarn_install job

scripts/sync-docs.mjs — the mirror, with no git and no network in it, so it is unit testable and runnable by hand onto any branch:

yarn sync-docs --source ../applandinc.github.io/_docs   # mirror
yarn sync-docs --source <dir> --dry-run                 # report only
yarn sync-docs --source <dir> --check                   # exit 1 if stale

Implemented in Node rather than shelling to rsync: the bug being fixed was rsync flag semantics, so mirror behavior is now explicit and asserted — and it works on Windows. Mirrors only the *.md files the indexer consumes, deletes what the website no longer publishes, prunes emptied directories, and takes an exclude list — which gives build:doc's post-hoc rm -f of the retired appmap-agent-js docs a proper home. 13 tests via node --test (no new deps), wired as yarn test:scripts and run in CI from test_the_rest.

.github/workflows/sync-docs.yml — weekly (Mon 06:00 UTC) + workflow_dispatch + a repository_dispatch: [sync-docs] hook, never pull_request. Sparse-checks-out _docs only, and opens a single long-lived chore/sync-docs PR with add-paths: docs. Records the upstream SHA in the commit and PR body for provenance.

Pulling beats fixing the push in place: a schedule is self-healing (a missed run is fixed by the next), the branch is cut from main each run so the base cannot go stale, failures surface in the repo that owns the consumer, and the credential direction inverts — this repo reads the website, where the website previously held a cross-repo write PAT. The PR gate is preserved, so a human still reviews what ships in the CLI.

Two credentials, deliberately

The website is read with a read-only deploy key (DOCS_REPO_SSH_KEY), not with an appmap-releasebot token. A GitHub App declares its permissions app-wide, not per repository, so installing releasebot on the website would let every holder of its private key — appmap-ruby, appmap-node, appmap-intellij-plugin — mint write access to it. Requesting a downscoped token via permission-contents: read would not fix that either: it is a request made by the same job that holds the key, so it guards against accidents, not against anything adversarial. A deploy key is scoped to one repository and read-only by construction, so nothing in this job can write to the website.

The pull request is opened with an app token, scoped to this repository and minted immediately before the step that uses it. That part is load-bearing rather than hygiene: GitHub suppresses workflow runs for GITHUB_TOKEN-authored events, so a sync PR created with it would arrive with zero checks — and test_cli building the help index over docs/ is the whole reason for opening one.

packages/cli build:doc now reuses the same script, so there is one mirror implementation instead of two and the build no longer needs rsync.

CI docs_only gate — a checkout-free changes job uses the PR files API to detect docs-only PRs and skips test_components, test_chromatic, test_scanner, test_native and test_the_rest for them. test_cli always runs, because HelpIndex.spec.ts builds the index over the real docs/ and asserts a ['java', 'maven'] search returns snippets containing both terms — an upstream reorganization can legitimately break search, and that is exactly the check worth keeping on a sync PR. The filter applies only to pull_request, so release's needs: are never skipped.

yarn_install removed. It saved caches under ${{ runner.os }}-{yarn,node-modules}-<hash>, but the only consumer of a dependency cache in the repo is .github/actions/setup-node, keyed ${{ github.job }}-… — and inside a composite
action github.job resolves to the calling job, never to yarn_install. Nothing has been reading what it wrote: it ran a full yarn install every CI run into orphaned cache entries while serializing six jobs behind itself. Trade-off: a cold
cache now costs one install per job rather than one per run, but jobs start in parallel. Sharing one cache across jobs would mean unifying setup-node's keys — deliberately left as a separate change.

Reviewing the docs commit

decb4bd8b is 44 files, −4519 lines. It is mechanical, and you can confirm it:

yarn sync-docs --source ../applandinc.github.io/_docs --check

docs/ is now 57 markdown files, exactly matching upstream. Also dropped: the non-Markdown assets the indexer never reads (two PDFs, two stray HTML files), and TODO-review2.md, which was added here rather than upstream — preserved in history.

Verification

• All 7 workflows parse; 13 script tests pass; yarn verify OK
• HelpIndex.spec.ts passes against the trimmed corpus
• build:doc produces 57 files and is idempotent
• --check reports the mirror clean

CI on this PR runs everything (it touches workflows and scripts, so it is not docs-only) — which exercises both the new changes job and the gate's default path. The app-token path and add-paths: docs can only be exercised by a real run; see
below.

Before merge

Three secrets, none of which exist here yet:

• RELEASE_BOT_CLIENT_ID and RELEASE_BOT_PRIVATE_KEY. The app is installed with the needed permissions, but secrets are per-repo.

  Note the Client ID, not the App ID — create-github-app-token deprecated app-id in favour of client-id in v3.1.0, and new code shouldn't start on a deprecated input. It's a different value (Iv23li…), on the app's settings page just under the App ID. The other repos using this app (appmap-ruby, appmap-node, appmap-intellij-plugin) are still on app-id with @v1; these are org-level secrets, added the client ID there to let them migrate later without another secret.

• DOCS_REPO_SSH_KEY — the private half of a read-only deploy key:

  ssh-keygen -t ed25519 -N '' -C 'appmap-js docs sync' -f docs-sync-key

  Add docs-sync-key.pub as a deploy key on applandinc.github.io with write access unchecked, and docs-sync-key here as DOCS_REPO_SSH_KEY.

Coordination

Pairs with https://github.com/getappmap/applandinc.github.io/pull/1581, which removes the push side.

Merge this one first, so there is never a window without a working sync path. If someone pushes to the website's master in between, the old workflow may open one more stale feat/sync-docs-* PR against next-doc — just close it.

After merge

1. workflow_dispatch the sync once rather than waiting for Monday. Expect 0 added, 0 updated, 0 deleted and no PR — a clean signal that both the mirror and the app token work.
2. Delete the next-doc branch. Nothing targets it now, and its only value was the staging hop that caused the stale-base bug.
3. Merge the website PR, then delete DOC_SYNC_TOKEN from the website repo's secrets — the cross-repo write credential is no longer used by anything.

Not in scope

• The PNG test that started this. sequenceDiagram.spec.ts stays skipped; re-enabling it needs a browser on the runner, since PUPPETEER_SKIP_DOWNLOAD is set globally. Verified locally that it passes both with the bundled Chromium 110 and with a system Chromium 150 via PUPPETEER_EXECUTABLE_PATH, so puppeteer 19's CDP usage here tolerates the version skew — the gap is browser provisioning, not the renderer.
• Unifying setup-node's cache keys so jobs share one entry.
• release still authenticating with the GH_TOKEN PAT — another appmap-releasebot candidate.
• SHA-pinning actions repo-wide. The two third-party actions introduced here are pinned, since they share a job with the app's private key. The rest of the repo still uses mutable tags; worth doing, but as its own change.

[Copilot]

Pull request overview

This PR replaces the old push-based documentation sync with a scheduled pull-based mirror from getappmap/applandinc.github.io, ensuring docs/ stays an exact (Markdown-only) help corpus for appmap navie help and preventing stale-base CI failures and corpus drift.

Changes:

• Added a Node-based, unit-tested scripts/sync-docs.mjs mirroring implementation (including delete/prune semantics and an exclude list) and wired it into Yarn scripts.
• Added a scheduled .github/workflows/sync-docs.yml workflow that checks out the website’s _docs via a GitHub App token and opens/updates a long-lived sync PR.
• Updated CI to detect docs-only PRs and skip expensive jobs while always running test_cli; updated packages/cli build:doc to reuse the same mirror logic.

Reviewed changes

Copilot reviewed 48 out of 50 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/sync-docs.test.mjs	Adds unit tests for the docs mirroring behavior (copy/update/delete/prune/exclude/dry-run).
scripts/sync-docs.mjs	Introduces the cross-platform, explicit docs mirroring implementation and CLI wrapper.
packages/cli/package.json	Switches build:doc from rsync to the shared sync-docs script.
package.json	Adds test:scripts and sync-docs entry points at the repo root.
docs/using-navie-ai/navie-user-interface.md	Removed from this repo’s mirrored corpus (upstream no longer publishes it).
docs/unused_for_now_index.html	Removes non-Markdown asset from the mirrored corpus.
docs/TODO-review2.md	Removes local-only TODO from the mirrored corpus (retained in history).
docs/reference/appmap-python.md	Updates mirrored docs content (log file behavior section).
docs/reference/appmap-client-cli.md	Updates mirrored docs content (adds/extends sanitize documentation).
docs/navie/using-navie.md	Removes superseded Navie doc page from corpus.
docs/navie/index.md	Removes superseded Navie index page from corpus.
docs/navie/how-to-open-navie.md	Removes superseded Navie doc page from corpus.
docs/navie/how-navie-works.md	Removes superseded Navie doc page from corpus.
docs/navie/bring-your-own-model.md	Removes superseded Navie doc page from corpus.
docs/navie-reference/navie-user-interface.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-pre-built-libraries-for-appmap-data.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-options.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-openai-key-management.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-how-it-works.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-github-repository.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-commands.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-bring-your-own-model-examples.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/navie-accessing-logs.md	Removes superseded Navie reference page from corpus.
docs/navie-reference/index.md	Removes superseded Navie reference index page from corpus.
docs/navie-reference.md	Removes monolithic Navie reference page from corpus.
docs/integrations/smartbear-swaggerhub.md	Removes superseded integrations doc page from corpus.
docs/integrations/readme.md	Removes superseded integrations doc page from corpus.
docs/integrations/postman.md	Removes superseded integrations doc page from corpus.
docs/integrations/plantuml.md	Removes superseded integrations doc page from corpus.
docs/integrations/mermaid.md	Removes superseded integrations doc page from corpus.
docs/integrations/index.md	Removes superseded integrations index page from corpus.
docs/integrations/docker.md	Removes superseded integrations doc page from corpus.
docs/integrations/circle-ci.md	Removes superseded integrations doc page from corpus.
docs/integrations/atlassian-confluence.md	Removes superseded integrations doc page from corpus.
docs/integrations/atlassian-compass.md	Removes superseded integrations doc page from corpus.
docs/guides/using-appmap-diagrams.md	Removes superseded guide page from corpus.
docs/guides/using-appmap-analysis.md	Removes superseded guide page from corpus.
docs/guides/reverse-engineering.md	Removes superseded guide page from corpus.
docs/guides/refine-appmap-data.md	Removes superseded guide page from corpus.
docs/guides/reading-sql-in-appmap-diagrams.md	Removes superseded guide page from corpus.
docs/guides/openapi.md	Removes superseded guide page from corpus.
docs/guides/navigating-code-objects.md	Removes superseded guide page from corpus.
docs/guides/index.md	Removes superseded guides index page from corpus.
docs/guides/handling-large-appmap-diagrams.md	Removes superseded guide page from corpus.
docs/guides/exporting-appmap-diagrams.md	Removes superseded guide page from corpus.
docs/analysis/rules-reference.html	Removes non-Markdown redirect artifact from corpus.
.github/workflows/sync-docs.yml	Adds scheduled pull-based sync workflow using GitHub App auth and a single long-lived PR.
.github/workflows/main.yml	Adds docs-only gating and runs yarn test:scripts; modernizes job dependencies after removing the old install gate.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 50 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

scripts/sync-docs.mjs:94

• syncDocs is exported, but when called with missing/undefined source or target it will currently throw a Node TypeError from existsSync/path handling rather than a clear usage error. Adding explicit argument validation here makes failures easier to diagnose and prevents confusing stack traces.