diff --git a/docs/telemetry-enhancements.md b/docs/telemetry-enhancements.md new file mode 100644 index 0000000..b039105 --- /dev/null +++ b/docs/telemetry-enhancements.md @@ -0,0 +1,127 @@ +# CLI Telemetry — Enhancements & Roadmap + +Possible improvements to the ChaiBuilder CLI telemetry (CLI events, the +`/in/cli/track` ingestion endpoint, and PostHog dashboards). + +**Priority:** P0 (go-live) · P1 (high) · P2 (nice) · P3 (later). +**Effort:** S (hours) · M (a day) · L (multi-day). + +--- + +## Current state + +- CLI emits `cmd:step:action[:result]` events for `create`, `new`, `switch` + (e.g. `create:db_migration:finish:error`). +- Collected: `os`, `cli_version`, per-run anonymous `distinct_id`/`run_id`, + event properties (`template`, `error_type`, redacted `error`). +- First-run notice + opt-out (`DO_NOT_TRACK`, `CHAIBUILDER_TELEMETRY_DISABLED`). +- Endpoint `POST /in/cli/track` (zod-validated, body-size cap) forwards to PostHog + after the response. Never blocks or crashes the CLI. +- **Not live yet:** endpoint PR unmerged; prod PostHog key not set. + +--- + +## 0. Go-live checklist (P0) + +| Item | Effort | +|---|---| +| Merge the endpoint PR (`cbpl` #104) and deploy | S | +| Set `POSTHOG_API_KEY` + `POSTHOG_HOST` in prod Vercel (cbpl) | S | +| Merge CLI telemetry PR (`cli` #8), publish the CLI | S | +| Merge `new`/`switch` PR (`cli` #9) after #8 | S | +| Delete/exclude seed test events in PostHog (`seed-*` distinct_ids) | S | + +--- + +## 1. Event data & schema + +**Status:** 1.1–1.5 ✅ implemented (see "Implemented — 1.1–1.5" below). 1.6–1.7 pending. + +| # | Enhancement | Why | P / Effort | +|---|---|---|---| +| 1.1 ✅ | **Structured error `code`** on `*:finish:error` (e.g. `MIGRATION_FAILED`, `TEMPLATE_DOWNLOAD_FAILED`) | Break down failures by a clean enum instead of the redacted free-text `error`; no PII risk | P1 / S | +| 1.2 ✅ | **Persistent anonymous id** (write a UUID once to `~/.config/chaibuilder/id`, reuse) | Distinguish unique machines from repeat runs; today `distinct_id` is per-run, so "users" == "runs" | P1 / S | +| 1.3 ✅ | **CI detection** (`is_ci` from `process.env.CI`) | Segment automated/CI runs from real developers | P2 / S | +| 1.4 ✅ | **Command-outcome event** (`create:done` / `create:failed`) | One event for success-rate queries without a multi-step funnel | P2 / S | +| 1.5 ✅ | **Step durations** | The redesign dropped `duration_ms`. Either rely on funnel "time between steps", or re-add a `duration_ms` on finish events | P2 / S | +| 1.6 | **Optional safe dimensions** (`packageManager`, `db: local\|remote`, `node`, `arch`) | Richer segmentation — currently intentionally OS-only; revisit with Suraj | P2 / S | +| 1.7 | **Homepage source** (`remote` vs bundled `fallback`) on seed | Know how often the remote homepage fetch fails | P3 / S | + +### Implemented — 1.1–1.5 + +- **1.1 error codes** — every `*:finish:error` now carries a stable `code` (`errorCode(step)`: `MIGRATION_FAILED`, `TEMPLATE_DOWNLOAD_FAILED`, else `_FAILED`). `check-env` keeps its own `INVALID_ENV` / `MISSING_ENV_FILE`. +- **1.2 persistent id** — `distinct_id` is now a UUID persisted once to `~/.config/chaibuilder/id` (best-effort, never throws); `run_id` stays per-run. Skipped entirely when the user has opted out. +- **1.3 CI detection** — `is_ci` rides in `properties` on every event (`CI`, `GITHUB_ACTIONS`, `GITLAB_CI`, `CIRCLECI`, `TRAVIS`, `BUILDKITE`, `TF_BUILD`, `JENKINS_URL`). +- **1.4 command outcome** — one terminal event per run: `create|new|switch|check_env` × `:done` / `:failed`. Suppressed on user cancel. `switch:done` also carries `changed: true|false`. +- **1.5 durations** — `duration_ms` on every step `*:finish:*` and on the outcome event. + +**No endpoint change required:** `distinct_id`/`run_id` are existing top-level fields (values only changed); `is_ci`, `code`, `duration_ms` ride inside `properties`, which the endpoint forwards to PostHog wholesale. + +--- + +## 2. Privacy & consent + +| # | Enhancement | Why | P / Effort | +|---|---|---|---| +| 2.1 | **`telemetry` subcommand** (`status` / `enable` / `disable`) | Let users manage telemetry without env vars; persist the choice | P1 / S | +| 2.2 | **Persist opt-out** to the config file | Today opt-out is env-only; persist a disabled flag so it sticks | P1 / S | +| 2.3 | **Notify-only first run** (don't collect on the very first run) | Stricter consent — the run where the notice first appears sends nothing | P2 / S | +| 2.4 | **Public "what we collect" page** linked from the notice | Transparency; standard for CLI telemetry | P2 / S | +| 2.5 | **Redaction hardening + fuzz tests** | Expand patterns (short tokens, connection strings), add property-based tests | P2 / M | + +--- + +## 3. Endpoint & infrastructure + +| # | Enhancement | Why | P / Effort | +|---|---|---|---| +| 3.1 | **Batch events per run** (buffer, send once via PostHog `/batch`) | ~13 POSTs/run → 1; less overhead and fewer dropped events at exit | P1 / M | +| 3.2 | **Point CLI at `www.chaibuilder.com`** (or add a non-www route) | Avoid the `chaibuilder.com` → `www` 308 hop on every event | P2 / S | +| 3.3 | **Basic rate limiting / abuse guard** on the endpoint | It's public + unauthenticated; cap per-IP | P2 / M | +| 3.4 | **Geo/country** (forward client `$ip`) | Where are users? — privacy trade-off, gate behind the notice | P2 / S | +| 3.5 | **`posthog-node` SDK** instead of raw fetch | Built-in batching/retries/flush | P3 / S | +| 3.6 | **Retry/queue failed sends** (persist, retry next run) | Recover events lost to transient network failures | P3 / M | + +--- + +## 4. Dashboards & analysis (PostHog) + +| # | Enhancement | Why | P / Effort | +|---|---|---|---| +| 4.1 | **Core tiles** — success funnel, create step funnel, failures-by-step, runs/day, OS, CLI version, new/switch usage | The base dashboard (in progress) | P0 / S | +| 4.2 | **Exclude seed/test distinct_ids** from all tiles | Keep metrics clean once real data flows | P0 / S | +| 4.3 | **Alerts** — success-rate drop / failure spike → Slack/email | Catch regressions in the wild fast | P1 / S | +| 4.4 | **Failure-reason table** (once 1.1 lands) — breakdown by `code` | Actionable "why installs/migrations fail" | P1 / S | +| 4.5 | **CLI-version adoption cohort** | Track migration off broken/old versions | P2 / S | +| 4.6 | **Weekly digest / subscription** | Passive visibility for the team | P2 / S | + +--- + +## 5. Testing & CI + +| # | Enhancement | Why | P / Effort | +|---|---|---|---| +| 5.1 | **`cli-e2e` telemetry assertions** (run against a mock endpoint, assert emitted events) | Guard the event contract end-to-end | P1 / M | +| 5.2 | **`cli-e2e` in CI** (GitHub Actions on PRs) | Auto-run create/new/switch on every change | P1 / M | +| 5.3 | **Error + remote-DB scenarios** in `cli-e2e` | Cover failure paths + libSQL/Turso, not just the happy path | P2 / M | +| 5.4 | **Cross-platform e2e** (Windows/Linux runners) | Catch path/PM issues per-OS | P3 / L | + +--- + +## 6. CLI UX / observability + +| # | Enhancement | Why | P / Effort | +|---|---|---|---| +| 6.1 | **`CHAIBUILDER_TELEMETRY_DEBUG`** — print events to stderr | Debug what's emitted without a server | P2 / S | +| 6.2 | **Telemetry on remaining commands** (`check-env`, future ones) | Full coverage as the CLI grows | P2 / S | +| 6.3 | **Event sampling** (if volume ever gets large) | Cost/volume control | P3 / S | + +--- + +## Suggested first slice (post go-live) + +1. **1.1 structured error codes** + **4.4 failure table** — turns the dashboard from "how many fail" into "why they fail." +2. **1.2 persistent id** — real unique-user counts. +3. **2.1 telemetry subcommand** + **2.2 persist opt-out** — proper user control. +4. **5.1 / 5.2 e2e telemetry + CI** — lock the contract. +5. **3.1 batching** — efficiency once volume is real. diff --git a/src/commands/check-env-run.ts b/src/commands/check-env-run.ts index ee922bc..862aed5 100644 --- a/src/commands/check-env-run.ts +++ b/src/commands/check-env-run.ts @@ -7,12 +7,15 @@ import { errorDetails, stepEvent } from '../lib/telemetry.js' const CMD = 'check_env' export async function runCheckEnv(ports: Ports): Promise<{ ok: boolean; exitCode: number }> { + const startedAt = Date.now() + const dur = () => ({ duration_ms: Date.now() - startedAt }) ports.telemetry.capture(stepEvent(CMD, 'run', 'start')) try { const envPath = ports.fs.join(ports.fs.cwd(), '.env') if (!(await ports.fs.exists(envPath))) { ports.log.error('Missing .env file') - ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'error'), { code: 'MISSING_ENV_FILE' }) + ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'error'), { code: 'MISSING_ENV_FILE', ...dur() }) + ports.telemetry.capture(`${CMD}:failed`, { code: 'MISSING_ENV_FILE', ...dur() }) return { ok: false, exitCode: 1 } } @@ -32,18 +35,18 @@ export async function runCheckEnv(ports: Ports): Promise<{ ok: boolean; exitCode if (result.ok) { ports.log.success('Required env vars look good') - ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'success')) + ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'success'), dur()) + ports.telemetry.capture(`${CMD}:done`, dur()) } else { - ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'error'), { - code: 'INVALID_ENV', - missing_count: result.missing.length, - empty_count: result.empty.length, - }) + const props = { code: 'INVALID_ENV', missing_count: result.missing.length, empty_count: result.empty.length } + ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'error'), { ...props, ...dur() }) + ports.telemetry.capture(`${CMD}:failed`, { ...props, ...dur() }) } return { ok: result.ok, exitCode: result.ok ? 0 : 1 } } catch (err) { - ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'error'), errorDetails(err)) + ports.telemetry.capture(stepEvent(CMD, 'run', 'finish', 'error'), { ...errorDetails(err), ...dur() }) + ports.telemetry.capture(`${CMD}:failed`, { ...errorDetails(err), ...dur() }) throw err } finally { await ports.telemetry.flush() diff --git a/src/commands/create-app-run.ts b/src/commands/create-app-run.ts index 675d67c..593846d 100644 --- a/src/commands/create-app-run.ts +++ b/src/commands/create-app-run.ts @@ -4,12 +4,13 @@ import type { Step, WizardState } from '../core/context.js' import { CliError } from '../core/errors.js' import { runSteps } from '../core/run-steps.js' import { TEMPLATE_SOURCE } from '../constants.js' -import { errorDetails, stepEvent } from '../lib/telemetry.js' +import { errorCode, errorDetails, stepEvent, tagCode } from '../lib/telemetry.js' import { checkUpdate } from '../steps/check-update.js' import { chooseDatabase } from '../steps/choose-database.js' import { detectPm } from '../steps/detect-package-manager.js' import { downloadTemplate } from '../steps/download-template.js' import { finalizeLocalDb } from '../steps/finalize-local-db.js' +import { initGit } from '../steps/init-git.js' import { installDeps } from '../steps/install-deps.js' import { outro } from '../steps/outro.js' import { promptAppName } from '../steps/prompt-app-name.js' @@ -29,15 +30,18 @@ function isCancellation(ports: Ports, err: unknown): boolean { function tracked(step: string, fn: Step, propsFn?: PropsFn): Step { return async (ctx, ports) => { + const startedAt = Date.now() ports.telemetry.capture(stepEvent(CMD, step, 'start'), propsFn?.(ctx)) try { const patch = await fn(ctx, ports) if (ports.prompts.isCancel(patch)) return patch - ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'success'), propsFn?.({ ...ctx, ...patch })) + ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'success'), { ...propsFn?.({ ...ctx, ...patch }), duration_ms: Date.now() - startedAt }) return patch } catch (err) { if (isCancellation(ports, err)) throw err - ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'error'), { ...propsFn?.(ctx), ...errorDetails(err) }) + const code = errorCode(step) + ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'error'), { ...propsFn?.(ctx), ...errorDetails(err), code, duration_ms: Date.now() - startedAt }) + tagCode(err, code) throw err } } @@ -45,6 +49,7 @@ function tracked(step: string, fn: Step, propsFn?: PropsFn): Step { function trackedSequence(step: string, fns: Step[], propsFn?: PropsFn): Step { return async (ctx, ports) => { + const startedAt = Date.now() ports.telemetry.capture(stepEvent(CMD, step, 'start'), propsFn?.(ctx)) let state = ctx try { @@ -53,11 +58,13 @@ function trackedSequence(step: string, fns: Step[], propsFn?: PropsFn): Step { if (ports.prompts.isCancel(patch)) return patch state = { ...state, ...patch } } - ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'success'), propsFn?.(state)) + ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'success'), { ...propsFn?.(state), duration_ms: Date.now() - startedAt }) return state } catch (err) { if (isCancellation(ports, err)) throw err - ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'error'), { ...propsFn?.(ctx), ...errorDetails(err) }) + const code = errorCode(step) + ports.telemetry.capture(stepEvent(CMD, step, 'finish', 'error'), { ...propsFn?.(ctx), ...errorDetails(err), code, duration_ms: Date.now() - startedAt }) + tagCode(err, code) throw err } } @@ -80,6 +87,7 @@ export const createAppSteps = [ tracked('db_migration', runMigrations), trackedSequence('create_admin_user', [phaseHeader('[2/3] Create admin user'), setupAppAndUser, seedDb]), finalizeLocalDb, + initGit, outro, ] @@ -97,7 +105,9 @@ export async function runCreateApp( cwd: ports.fs.cwd(), ...initial, } - const final = await runSteps(createAppSteps, state, ports) + const startedAt = Date.now() + const final = await runSteps(createAppSteps, state, ports, { cmd: CMD, startedAt }) + ports.telemetry.capture(`${CMD}:done`, { duration_ms: Date.now() - startedAt }) await ports.telemetry.flush() return final } diff --git a/src/commands/new-app-run.ts b/src/commands/new-app-run.ts index 20398f9..7ec6655 100644 --- a/src/commands/new-app-run.ts +++ b/src/commands/new-app-run.ts @@ -5,7 +5,7 @@ import { parseEnv, upsertEnv } from '../lib/env-file.js' import { createAppRecord, findUserIdByEmail, openDb } from '../lib/app-record.js' import { runAdminOp } from '../lib/admin-runner.js' import { resolvePackageManagerForDir } from '../lib/package-manager.js' -import { trackStep } from '../lib/telemetry.js' +import { errorDetails, trackStep } from '../lib/telemetry.js' const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/ @@ -18,6 +18,7 @@ const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/ * CHAIBUILDER_APP_KEY at it. */ export async function runNewApp(ports: Ports): Promise { + const startedAt = Date.now() ports.log.intro('Create a new ChaiBuilder app') try { const appDir = ports.fs.cwd() @@ -97,7 +98,11 @@ export async function runNewApp(ports: Ports): Promise { ports.log.message(pc.bold('Next steps')) ports.log.note(`Active app is now ${pc.cyan(appName)} (${appId}).`) ports.log.note('Restart your dev server to load it.') + ports.telemetry.capture('new:done', { duration_ms: Date.now() - startedAt }) ports.log.outro('Done') + } catch (err) { + ports.telemetry.capture('new:failed', { ...errorDetails(err), duration_ms: Date.now() - startedAt }) + throw err } finally { await ports.telemetry.flush() } diff --git a/src/commands/switch-app-run.ts b/src/commands/switch-app-run.ts index c55ce25..1cf7b30 100644 --- a/src/commands/switch-app-run.ts +++ b/src/commands/switch-app-run.ts @@ -3,10 +3,11 @@ import type { Ports } from '../core/adapters.js' import { CliError } from '../core/errors.js' import { parseEnv, upsertEnv } from '../lib/env-file.js' import { listApps, openDb, type AppSummary } from '../lib/app-record.js' -import { trackStep } from '../lib/telemetry.js' +import { errorDetails, trackStep } from '../lib/telemetry.js' export async function runSwitchApp(ports: Ports): Promise { + const startedAt = Date.now() ports.log.intro('Switch ChaiBuilder app') try { const appDir = ports.fs.cwd() @@ -57,7 +58,10 @@ export async function runSwitchApp(ports: Ports): Promise { }) if (ports.prompts.isCancel(selected)) return void ports.log.outro('Cancelled') - if (selected === current) return void ports.log.outro('No change — already the active app.') + if (selected === current) { + ports.telemetry.capture('switch:done', { duration_ms: Date.now() - startedAt, changed: false }) + return void ports.log.outro('No change — already the active app.') + } await trackStep(ports.telemetry, 'switch', 'switch_app', async () => { const content = await ports.fs.readFile(envPath) @@ -67,7 +71,11 @@ export async function runSwitchApp(ports: Ports): Promise { ports.log.message(pc.bold('Switched')) ports.log.note(`Active app: ${pc.cyan(picked?.name ?? String(selected))}`) ports.log.note('Restart your dev server to load it.') + ports.telemetry.capture('switch:done', { duration_ms: Date.now() - startedAt, changed: true }) ports.log.outro('Done') + } catch (err) { + ports.telemetry.capture('switch:failed', { ...errorDetails(err), duration_ms: Date.now() - startedAt }) + throw err } finally { await ports.telemetry.flush() } diff --git a/src/core/real-adapters.ts b/src/core/real-adapters.ts index a0db77f..6b9513e 100644 --- a/src/core/real-adapters.ts +++ b/src/core/real-adapters.ts @@ -8,7 +8,7 @@ import { createRequire } from 'node:module' import path from 'node:path' import { execa } from 'execa' import { downloadTemplate } from 'giget' -import { createTelemetry } from '../lib/telemetry.js' +import { createTelemetry, resolveDistinctId, telemetryDisabled } from '../lib/telemetry.js' import type { ExecPort, FsPort, @@ -233,7 +233,10 @@ export function createRealPorts(): Ports { exec: createExec(), template: createTemplate(), log: createLog(), - telemetry: createTelemetry({ cliVersion: cliVersion() }), + telemetry: createTelemetry({ + cliVersion: cliVersion(), + distinctId: telemetryDisabled() ? undefined : resolveDistinctId(), + }), randomBytes: (size) => nodeRandomBytes(size), } } diff --git a/src/core/run-steps.ts b/src/core/run-steps.ts index 3cb0d78..8db0a50 100644 --- a/src/core/run-steps.ts +++ b/src/core/run-steps.ts @@ -1,12 +1,14 @@ import type { Ports } from './adapters.js' import type { Step, WizardState } from './context.js' import { CliError } from './errors.js' +import { errorDetails, readCode } from '../lib/telemetry.js' import pc from 'picocolors' export async function runSteps( steps: Step[], initial: WizardState, ports: Ports, + opts?: { cmd?: string; startedAt?: number }, ): Promise { let state = { ...initial } @@ -20,13 +22,20 @@ export async function runSteps( } state = { ...state, ...patch } } catch (err: any) { - if (ports.prompts.isCancel(err)) { + const cancelled = ports.prompts.isCancel(err) || (err instanceof CliError && err.message === 'Cancelled') + if (cancelled) { await ports.telemetry.flush() console.error(pc.red('Cancelled')) process.exit(1) } - // Step-level telemetry (finish/error) is emitted by the tracked() wrappers - // in create-app-run before the throw reaches here. + // terminal command-outcome event (step-level finish/error already emitted upstream) + if (opts?.cmd) { + ports.telemetry.capture(`${opts.cmd}:failed`, { + ...errorDetails(err), + code: readCode(err) ?? `${opts.cmd.toUpperCase()}_FAILED`, + ...(opts.startedAt != null ? { duration_ms: Date.now() - opts.startedAt } : {}), + }) + } await ports.telemetry.flush() console.error(pc.red(err.message || String(err))) process.exit(1) diff --git a/src/lib/telemetry.ts b/src/lib/telemetry.ts index d9625a6..6884bff 100644 --- a/src/lib/telemetry.ts +++ b/src/lib/telemetry.ts @@ -1,4 +1,7 @@ import { randomUUID } from 'node:crypto' +import { mkdirSync, readFileSync, writeFileSync } from 'node:fs' +import { homedir } from 'node:os' +import { dirname, join } from 'node:path' import type { TelemetryPort } from '../core/adapters.js' const DEFAULT_ENDPOINT = 'https://chaibuilder.com/in/cli/track' @@ -12,6 +15,8 @@ export type TelemetryOptions = { disabled?: boolean /** Injectable fetch for tests; defaults to global fetch. */ fetchImpl?: typeof fetch + /** Persistent anonymous id (per machine). Falls back to a per-run id if omitted. */ + distinctId?: string } export type StepAction = 'start' | 'finish' @@ -27,7 +32,45 @@ export function stepEvent(cmd: string, step: string, action: StepAction, result? return result ? `${cmd}:${step}:${action}:${result}` : `${cmd}:${step}:${action}` } -/** Emit start then finish:success|error around an async operation. */ +/** True when running under a known CI provider (segments automated runs). */ +export function isCI(): boolean { + const e = process.env + return Boolean( + e.CI || + e.CONTINUOUS_INTEGRATION || + e.GITHUB_ACTIONS || + e.GITLAB_CI || + e.CIRCLECI || + e.TRAVIS || + e.BUILDKITE || + e.TF_BUILD || + e.JENKINS_URL, + ) +} + +const ERROR_CODE_ALIASES: Record = { + template_clone: 'TEMPLATE_DOWNLOAD_FAILED', + db_migration: 'MIGRATION_FAILED', +} + +/** Stable failure enum for a step, e.g. `MIGRATION_FAILED`, `INSTALL_DEPS_FAILED`. */ +export function errorCode(step: string): string { + return ERROR_CODE_ALIASES[step] ?? `${step.toUpperCase()}_FAILED` +} + +const CODE_TAG = Symbol.for('chaibuilder.telemetryCode') + +/** Attach a failure `code` to an error so a later outcome event can reuse it. */ +export function tagCode(err: unknown, code: string): void { + if (err && typeof err === 'object') (err as Record)[CODE_TAG] = code +} + +/** Read a `code` attached by {@link tagCode}, if any. */ +export function readCode(err: unknown): string | undefined { + return err && typeof err === 'object' ? ((err as Record)[CODE_TAG] as string | undefined) : undefined +} + +/** Emit start then finish:success|error (with `duration_ms`, and `code` on error). */ export async function trackStep( telemetry: TelemetryPort, cmd: string, @@ -35,13 +78,19 @@ export async function trackStep( fn: () => Promise, props?: Record, ): Promise { + const startedAt = Date.now() telemetry.capture(stepEvent(cmd, step, 'start'), props) try { const result = await fn() - telemetry.capture(stepEvent(cmd, step, 'finish', 'success'), props) + telemetry.capture(stepEvent(cmd, step, 'finish', 'success'), { ...props, duration_ms: Date.now() - startedAt }) return result } catch (err) { - telemetry.capture(stepEvent(cmd, step, 'finish', 'error'), { ...props, ...errorDetails(err) }) + telemetry.capture(stepEvent(cmd, step, 'finish', 'error'), { + ...props, + ...errorDetails(err), + code: errorCode(step), + duration_ms: Date.now() - startedAt, + }) throw err } } @@ -50,20 +99,25 @@ export function createTelemetry(opts: TelemetryOptions): TelemetryPort { const disabled = opts.disabled ?? telemetryDisabled() const endpoint = opts.endpoint ?? process.env.CHAIBUILDER_TELEMETRY_URL ?? DEFAULT_ENDPOINT const doFetch = opts.fetchImpl ?? (globalThis.fetch as typeof fetch | undefined) - const id = safeUUID() const base = { - distinct_id: id, - run_id: id, + distinct_id: opts.distinctId ?? safeUUID(), + run_id: safeUUID(), cli_version: opts.cliVersion, os: process.platform, } + const baseProps = { is_ci: isCI() } const inflight = new Set>() function capture(event: string, properties: Record = {}): void { if (disabled || typeof doFetch !== 'function') return let body: string try { - body = JSON.stringify({ event, ...base, timestamp: new Date().toISOString(), properties }) + body = JSON.stringify({ + event, + ...base, + timestamp: new Date().toISOString(), + properties: { ...baseProps, ...properties }, + }) } catch { return } @@ -139,3 +193,26 @@ function safeUUID(): string { return `run_${Date.now()}_${Math.random().toString(36).slice(2, 10)}` } } + +/** + * Persistent anonymous machine id: read (or create once) a UUID at + * `~/.config/chaibuilder/id`. Best-effort — never throws; falls back to a + * per-run id if the file can't be read or written. + */ +export function resolveDistinctId(idFile?: string): string { + try { + const file = idFile ?? join(homedir(), '.config', 'chaibuilder', 'id') + try { + const existing = readFileSync(file, 'utf8').trim() + if (existing) return existing + } catch {} + const id = safeUUID() + try { + mkdirSync(dirname(file), { recursive: true }) + writeFileSync(file, id + '\n') + } catch {} + return id + } catch { + return safeUUID() + } +} diff --git a/src/steps/init-git.ts b/src/steps/init-git.ts new file mode 100644 index 0000000..4d154d5 --- /dev/null +++ b/src/steps/init-git.ts @@ -0,0 +1,73 @@ +import type { Ports } from '../core/adapters.js' +import type { Step } from '../core/context.js' +import { errorDetails } from '../lib/telemetry.js' + +const CMD = 'create' +const STEP = 'git_init' +const COMMIT_MESSAGE = 'feat: chaibuilder starter setup' + +async function gitAvailable(ports: Ports, cwd: string): Promise { + try { + return (await ports.exec.run('git', ['--version'], { cwd })).exitCode === 0 + } catch { + return false + } +} + +/** + * Offer to initialize a git repository and make the first commit. Best-effort: + * only prompts when git is installed, treats decline/cancel as a skip, and never + * fails the create flow. + */ +export const initGit: Step = async (ctx, ports) => { + const cwd = ctx.appDir + if (!(await gitAvailable(ports, cwd))) return {} + if (await ports.fs.exists(ports.fs.join(cwd, '.git'))) return {} + + const startedAt = Date.now() + ports.telemetry.capture(`${CMD}:${STEP}:start`) + + let confirmed: boolean + try { + confirmed = await ports.prompts.confirm({ + message: 'Initialize a git repository and make the first commit?', + initialValue: true, + }) + } catch { + confirmed = false + } + if (ports.prompts.isCancel(confirmed) || !confirmed) { + ports.telemetry.capture(`${CMD}:${STEP}:finish:success`, { committed: false, declined: true, duration_ms: Date.now() - startedAt }) + return {} + } + + const spinner = ports.log.spinner() + spinner.start('Initializing git repository') + try { + const run = (args: string[]) => ports.exec.run('git', args, { cwd }) + const init = await run(['init']) + if (init.exitCode !== 0) throw new Error(init.stderr || 'git init failed') + const add = await run(['add', '-A']) + if (add.exitCode !== 0) throw new Error(add.stderr || 'git add failed') + // never stage env secrets, regardless of the template's .gitignore + await run(['reset', '-q', '--', '.env', '.env.local']) + + // supply a fallback identity only for the field(s) git has none configured for + const identity: string[] = [] + if ((await run(['config', 'user.name'])).exitCode !== 0) identity.push('-c', 'user.name=ChaiBuilder') + if ((await run(['config', 'user.email'])).exitCode !== 0) identity.push('-c', 'user.email=noreply@chaibuilder.com') + const commit = await run([...identity, 'commit', '-m', COMMIT_MESSAGE]) + if (commit.exitCode !== 0) throw new Error(commit.stderr || 'git commit failed') + + spinner.stop('Initialized a git repository with the first commit') + ports.telemetry.capture(`${CMD}:${STEP}:finish:success`, { committed: true, duration_ms: Date.now() - startedAt }) + } catch (err) { + spinner.error('Skipped git setup') + ports.log.warn('Could not initialize git — continuing without it.') + try { + await ports.fs.rm(ports.fs.join(cwd, '.git'), { recursive: true, force: true }) + } catch {} + ports.telemetry.capture(`${CMD}:${STEP}:finish:error`, { ...errorDetails(err), code: 'GIT_INIT_FAILED', duration_ms: Date.now() - startedAt }) + } + return {} +} diff --git a/tests/commands/check-env.test.ts b/tests/commands/check-env.test.ts index 7a24bb8..3e4a5b4 100644 --- a/tests/commands/check-env.test.ts +++ b/tests/commands/check-env.test.ts @@ -24,6 +24,8 @@ describe('runCheckEnv', () => { const names = ports.telemetry.events.map((e) => e.event) expect(names).toContain('check_env:run:start') expect(names).toContain('check_env:run:finish:success') + expect(names).toContain('check_env:done') + expect(names).not.toContain('check_env:failed') expect(ports.telemetry.flushed).toBeGreaterThan(0) }) @@ -40,7 +42,11 @@ describe('runCheckEnv', () => { const failed = ports.telemetry.events.find((e) => e.event === 'check_env:run:finish:error') expect(failed).toBeTruthy() expect(failed!.properties.code).toBe('INVALID_ENV') + const outcome = ports.telemetry.events.find((e) => e.event === 'check_env:failed') + expect(outcome).toBeTruthy() + expect(outcome!.properties.code).toBe('INVALID_ENV') expect(ports.telemetry.events.map((e) => e.event)).not.toContain('check_env:run:finish:success') + expect(ports.telemetry.events.map((e) => e.event)).not.toContain('check_env:done') }) it('warns on incomplete s3/smtp', async () => { diff --git a/tests/commands/create-app.test.ts b/tests/commands/create-app.test.ts index 27ead00..b39532d 100644 --- a/tests/commands/create-app.test.ts +++ b/tests/commands/create-app.test.ts @@ -70,7 +70,14 @@ describe('runCreateApp', () => { expect(names).toContain('create:template_clone:start') expect(names).toContain('create:install_deps:finish:success') expect(names).toContain('create:db_migration:finish:success') - expect(names.at(-1)).toBe('create:create_admin_user:finish:success') + expect(names).toContain('create:create_admin_user:finish:success') + // terminal command-outcome event fires last + expect(names.at(-1)).toBe('create:done') + const done = ports.telemetry.events.find((e) => e.event === 'create:done')! + expect(typeof done.properties.duration_ms).toBe('number') + // step finish events carry a duration + const migr = ports.telemetry.events.find((e) => e.event === 'create:db_migration:finish:success')! + expect(typeof migr.properties.duration_ms).toBe('number') // template_clone events carry the template details const clone = ports.telemetry.events.find((e) => e.event === 'create:template_clone:start')! expect(String(clone.properties.template)).toContain('chaibuilder-starter') @@ -130,8 +137,15 @@ describe('runCreateApp', () => { expect(failed).toBeTruthy() expect(failed!.properties.error_type).toBe('CliError') expect(typeof failed!.properties.error).toBe('string') + expect(failed!.properties.code).toBe('MIGRATION_FAILED') // db_migration/start fired before it failed expect(ports.telemetry.events.map((e) => e.event)).toContain('create:db_migration:start') + // terminal command-outcome event fires on failure too + const outcome = ports.telemetry.events.find((e) => e.event === 'create:failed')! + expect(outcome).toBeTruthy() + expect(outcome.properties.error_type).toBe('CliError') + expect(outcome.properties.code).toBe('MIGRATION_FAILED') // propagated from the failed step + expect(ports.telemetry.events.map((e) => e.event)).not.toContain('create:done') expect(ports.telemetry.flushed).toBeGreaterThan(0) }) @@ -173,5 +187,8 @@ describe('runCreateApp', () => { const names = ports.telemetry.events.map((e) => e.event) expect(names).toContain('create:setup:start') expect(names.filter((n) => n.startsWith('create:setup:finish'))).toHaveLength(0) + // a user cancel is not an outcome — neither done nor failed + expect(names).not.toContain('create:done') + expect(names).not.toContain('create:failed') }) }) diff --git a/tests/commands/new-app.test.ts b/tests/commands/new-app.test.ts index 4a7d670..b7ec392 100644 --- a/tests/commands/new-app.test.ts +++ b/tests/commands/new-app.test.ts @@ -48,6 +48,8 @@ describe('runNewApp', () => { const names = ports.telemetry.events.map((e) => e.event) expect(names).toContain('new:create_app:finish:success') expect(names).not.toContain('new:create_user:start') // existing user reused + expect(names).toContain('new:done') + expect(names).not.toContain('new:failed') expect(ports.telemetry.flushed).toBeGreaterThan(0) expect(JSON.stringify(ports.telemetry.events)).not.toMatch(/admin@example|password/) }) diff --git a/tests/commands/switch-app.test.ts b/tests/commands/switch-app.test.ts index 5523841..e015fe1 100644 --- a/tests/commands/switch-app.test.ts +++ b/tests/commands/switch-app.test.ts @@ -40,6 +40,9 @@ describe('runSwitchApp', () => { const names = ports.telemetry.events.map((e) => e.event) expect(names).toContain('switch:load_apps:finish:success') expect(names).toContain('switch:switch_app:finish:success') + const done = ports.telemetry.events.find((e) => e.event === 'switch:done')! + expect(done).toBeTruthy() + expect(done.properties.changed).toBe(true) expect(ports.telemetry.flushed).toBeGreaterThan(0) }) @@ -49,6 +52,9 @@ describe('runSwitchApp', () => { const ports = createFakePorts({ fs, prompts: createScriptedPrompts(['app-a']) }) await runSwitchApp(ports) expect(parseEnv(await fs.readFile('/app/.env')).CHAIBUILDER_APP_KEY).toBe('app-a') + const done = ports.telemetry.events.find((e) => e.event === 'switch:done')! + expect(done).toBeTruthy() + expect(done.properties.changed).toBe(false) }) it('errors when no apps exist', async () => { diff --git a/tests/lib/telemetry.test.ts b/tests/lib/telemetry.test.ts index 38f9c6a..94c83a4 100644 --- a/tests/lib/telemetry.test.ts +++ b/tests/lib/telemetry.test.ts @@ -1,5 +1,19 @@ +import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' -import { createTelemetry, errorDetails, redact, stepEvent, telemetryDisabled } from '../../src/lib/telemetry.js' +import { + createTelemetry, + errorCode, + errorDetails, + isCI, + redact, + resolveDistinctId, + stepEvent, + telemetryDisabled, + trackStep, +} from '../../src/lib/telemetry.js' +import { createFakeTelemetry } from '../fakes.js' type Call = { url: string; body: any } @@ -41,8 +55,10 @@ describe('telemetry', () => { properties: { template: 'x' }, }) expect(calls[0].body.distinct_id).toBeTruthy() - expect(calls[0].body.run_id).toBe(calls[0].body.distinct_id) + expect(calls[0].body.run_id).toBeTruthy() + expect(calls[0].body.run_id).not.toBe(calls[0].body.distinct_id) // persistent id vs per-run id expect(calls[0].body.distinct_id).toBe(calls[1].body.distinct_id) // stable within a run + expect(calls[0].body.run_id).toBe(calls[1].body.run_id) // stable within a run // OS only — no node version or CPU arch expect(calls[0].body).not.toHaveProperty('node') expect(calls[0].body).not.toHaveProperty('arch') @@ -113,4 +129,75 @@ describe('telemetry', () => { expect(() => t.capture('cli_started')).not.toThrow() await expect(t.flush()).resolves.toBeUndefined() }) + + it('uses the provided persistent distinctId with a separate run_id', async () => { + const { calls, fetchImpl } = stubFetch() + const t = createTelemetry({ cliVersion: '1.0.0', fetchImpl, distinctId: 'persist-123' }) + t.capture('x') + await t.flush() + expect(calls[0].body.distinct_id).toBe('persist-123') + expect(calls[0].body.run_id).toBeTruthy() + expect(calls[0].body.run_id).not.toBe('persist-123') + }) + + it('adds is_ci to every event’s properties', async () => { + const { calls, fetchImpl } = stubFetch() + const t = createTelemetry({ cliVersion: '1.0.0', fetchImpl }) + t.capture('x', { template: 'y' }) + await t.flush() + expect(typeof calls[0].body.properties.is_ci).toBe('boolean') + expect(calls[0].body.properties.template).toBe('y') + }) + + it('isCI reflects known CI env vars', () => { + const keys = ['CI', 'CONTINUOUS_INTEGRATION', 'GITHUB_ACTIONS', 'GITLAB_CI', 'CIRCLECI', 'TRAVIS', 'BUILDKITE', 'TF_BUILD', 'JENKINS_URL'] + const saved = keys.map((k) => [k, process.env[k]] as const) + for (const k of keys) delete process.env[k] + expect(isCI()).toBe(false) + process.env.GITHUB_ACTIONS = 'true' + expect(isCI()).toBe(true) + for (const [k, v] of saved) { + if (v === undefined) delete process.env[k] + else process.env[k] = v + } + }) + + it('errorCode maps steps to stable enums', () => { + expect(errorCode('db_migration')).toBe('MIGRATION_FAILED') + expect(errorCode('template_clone')).toBe('TEMPLATE_DOWNLOAD_FAILED') + expect(errorCode('install_deps')).toBe('INSTALL_DEPS_FAILED') + }) + + it('trackStep records duration_ms on success and code+duration on error', async () => { + const t = createFakeTelemetry() + await trackStep(t, 'new', 'create_app', async () => 'ok') + const ok = t.events.find((e) => e.event === 'new:create_app:finish:success')! + expect(typeof ok.properties.duration_ms).toBe('number') + + await expect( + trackStep(t, 'create', 'db_migration', async () => { + throw new Error('boom') + }), + ).rejects.toThrow('boom') + const err = t.events.find((e) => e.event === 'create:db_migration:finish:error')! + expect(err.properties.code).toBe('MIGRATION_FAILED') + expect(typeof err.properties.duration_ms).toBe('number') + }) +}) + +describe('resolveDistinctId', () => { + it('creates an id once, persists it, and reuses it', () => { + const file = join(mkdtempSync(join(tmpdir(), 'chai-id-')), 'id') + const first = resolveDistinctId(file) + expect(first.length).toBeGreaterThan(8) + expect(readFileSync(file, 'utf8').trim()).toBe(first) + expect(resolveDistinctId(file)).toBe(first) // reused, not regenerated + }) + + it('returns an id (best-effort) even when the path is unwritable', () => { + const blocker = join(mkdtempSync(join(tmpdir(), 'chai-id-')), 'blocker') + writeFileSync(blocker, 'x') + // parent of the id file is a regular file → read and mkdir/write both fail + expect(resolveDistinctId(join(blocker, 'id'))).toBeTruthy() + }) }) diff --git a/tests/steps/init-git.test.ts b/tests/steps/init-git.test.ts new file mode 100644 index 0000000..8a870c5 --- /dev/null +++ b/tests/steps/init-git.test.ts @@ -0,0 +1,122 @@ +import { describe, expect, it } from 'vitest' +import type { WizardState } from '../../src/core/context.js' +import { initGit } from '../../src/steps/init-git.js' +import { createFakePorts, createRecordingExec, createScriptedPrompts } from '../fakes.js' +import type { ExecCall } from '../fakes.js' + +const ctx: WizardState = { + appName: 'app', + appDir: '/tmp/app', + packageManager: 'pnpm', + dbChoice: 'sqlite', + databaseUrl: '', +} + +function gitExec(opts: { installed?: boolean; hasName?: boolean; hasEmail?: boolean; commitFails?: boolean } = {}) { + const { installed = true, hasName = true, hasEmail = true, commitFails = false } = opts + return createRecordingExec({ + git: (call: ExecCall) => { + const a = call.args + if (a.includes('--version')) return { stdout: 'git version 2.40', stderr: '', exitCode: installed ? 0 : 1 } + if (a[0] === 'config') { + const present = a[1] === 'user.name' ? hasName : hasEmail + return { stdout: present ? 'x' : '', stderr: '', exitCode: present ? 0 : 1 } + } + if (a.includes('commit')) return { stdout: '', stderr: commitFails ? 'no identity' : '', exitCode: commitFails ? 1 : 0 } + return { stdout: '', stderr: '', exitCode: 0 } // init, add, reset + }, + }) +} + +const gitCalls = (exec: { calls: ExecCall[] }) => exec.calls.filter((c) => c.cmd === 'git') + +describe('initGit', () => { + it('does nothing (no prompt, no telemetry) when git is not installed', async () => { + const exec = gitExec({ installed: false }) + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([]) }) + await initGit(ctx, ports) + expect(ports.telemetry.events).toHaveLength(0) + // only the availability probe ran — never prompted or committed + expect(gitCalls(exec).map((c) => c.args.join(' '))).toEqual(['--version']) + }) + + it('skips when the scaffold already has a .git directory', async () => { + const exec = gitExec() + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([]) }) + await ports.fs.mkdir('/tmp/app/.git', { recursive: true }) + await initGit(ctx, ports) + expect(ports.telemetry.events).toHaveLength(0) + }) + + it('inits + commits when the user confirms', async () => { + const exec = gitExec() + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([true]) }) + await initGit(ctx, ports) + const args = gitCalls(exec).map((c) => c.args.join(' ')) + expect(args).toContain('init') + expect(args).toContain('add -A') + expect(args.some((a) => a.startsWith('reset -q -- .env'))).toBe(true) // secrets never staged + expect(args.some((a) => a.startsWith('commit -m'))).toBe(true) + expect(args.some((a) => a.includes('feat: chaibuilder starter setup'))).toBe(true) + + const names = ports.telemetry.events.map((e) => e.event) + expect(names).toContain('create:git_init:start') + const done = ports.telemetry.events.find((e) => e.event === 'create:git_init:finish:success')! + expect(done.properties.committed).toBe(true) + expect(typeof done.properties.duration_ms).toBe('number') + }) + + it('falls back to an inline identity when git has no user configured', async () => { + const exec = gitExec({ hasName: false, hasEmail: false }) + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([true]) }) + await initGit(ctx, ports) + const commit = gitCalls(exec).find((c) => c.args.includes('commit'))! + expect(commit.args).toContain('user.name=ChaiBuilder') + expect(commit.args).toContain('user.email=noreply@chaibuilder.com') + }) + + it('only fills the missing identity field, keeping the configured one', async () => { + const exec = gitExec({ hasName: true, hasEmail: false }) + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([true]) }) + await initGit(ctx, ports) + const commit = gitCalls(exec).find((c) => c.args.includes('commit'))! + expect(commit.args).not.toContain('user.name=ChaiBuilder') // real name preserved + expect(commit.args).toContain('user.email=noreply@chaibuilder.com') + }) + + it('treats a cancelled prompt as a skip', async () => { + const exec = gitExec() + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([Symbol.for('clack.cancelled')]) }) + await initGit(ctx, ports) + expect(gitCalls(exec).some((c) => c.args.includes('commit'))).toBe(false) + const done = ports.telemetry.events.find((e) => e.event === 'create:git_init:finish:success')! + expect(done.properties.declined).toBe(true) + }) + + it('treats a prompt that throws as a skip', async () => { + const exec = gitExec() + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([]) }) // confirm() throws: no answers + await expect(initGit(ctx, ports)).resolves.toEqual({}) + expect(gitCalls(exec).some((c) => c.args.includes('commit'))).toBe(false) + expect(ports.telemetry.events.find((e) => e.event === 'create:git_init:finish:success')!.properties.declined).toBe(true) + }) + + it('records a declined skip without committing', async () => { + const exec = gitExec() + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([false]) }) + await initGit(ctx, ports) + expect(gitCalls(exec).some((c) => c.args.includes('commit'))).toBe(false) + const done = ports.telemetry.events.find((e) => e.event === 'create:git_init:finish:success')! + expect(done.properties.declined).toBe(true) + expect(done.properties.committed).toBe(false) + }) + + it('reports a GIT_INIT_FAILED error (and never throws) when the commit fails', async () => { + const exec = gitExec({ commitFails: true }) + const ports = createFakePorts({ exec, prompts: createScriptedPrompts([true]) }) + await expect(initGit(ctx, ports)).resolves.toEqual({}) + const err = ports.telemetry.events.find((e) => e.event === 'create:git_init:finish:error')! + expect(err).toBeTruthy() + expect(err.properties.code).toBe('GIT_INIT_FAILED') + }) +})