diff --git a/docs/content/architecture/frontend/visual-baselines.md b/docs/content/architecture/frontend/visual-baselines.md new file mode 100644 index 00000000..291e908b --- /dev/null +++ b/docs/content/architecture/frontend/visual-baselines.md @@ -0,0 +1,94 @@ +# Visual baselines + +Ten reference images, compared pixel for pixel, produced in one Linux container +and nowhere else. + +They exist to catch what an assertion cannot see: a token resolving to the wrong +colour, a font that failed to load, an icon that stopped drawing, a radius or a gap +that moved. They do not replace the behavioural specs beside them — those still own +navigation, focus, keyboard, requests and overflow, and a screenshot is a poor way +to ask any of those questions. + +## The canonical environment + +A screenshot compares two renderings, so everything the renderer reads is stated +rather than inherited. The `visual` project in `playwright.config.ts` pins the +locale, the timezone and `deviceScaleFactor`; the container pins the browser. + +``` +image mcr.microsoft.com/playwright:v1.62.1-noble +os Ubuntu 24.04 LTS +node v24.18.1 +pnpm 10.30.2 +playwright 1.62.1 +chromium 151.0.7922.34 +deviceScaleFactor 1 +locale en-US +timezone UTC +``` + +Baselines are **not** generated on macOS or Windows. Font rasterisation differs +between operating systems, and a baseline captured on one and compared on another +reports that difference as a product regression. + +## Running the comparison + +```bash +docker run --rm \ + -v "$PWD:/repo:ro" -w /repo \ + mcr.microsoft.com/playwright:v1.62.1-noble \ + bash -lc 'corepack enable && pnpm install --frozen-lockfile \ + && pnpm --filter @visionset/annotator build \ + && pnpm --filter @visionset/ui-core build \ + && cd frontend/app && CI=1 pnpm exec playwright test --project=visual' +``` + +The `visual` project is excluded from the default `chromium` project, so the +ordinary suite does not compare images and this command is the only thing that +does. + +## Updating a baseline + +Only when the visual change is intended. The same container, with +`--update-snapshots`: + +```bash +docker run --rm \ + -v "$PWD:/repo" -w /repo \ + mcr.microsoft.com/playwright:v1.62.1-noble \ + bash -lc 'corepack enable && pnpm install --frozen-lockfile \ + && pnpm --filter @visionset/annotator build \ + && pnpm --filter @visionset/ui-core build \ + && cd frontend/app && CI=1 pnpm exec playwright test --project=visual --update-snapshots' +``` + +Then **look at the images** and run the comparison again before committing. A +regenerated baseline that nobody opened is a regression that has been written down +as the new truth. `git diff --stat` naming a `.png` under +`e2e/visual.spec.ts-snapshots/` is the signal that a review is owed. + +Snapshots are never updated by an ordinary test run. There is no host-native path +for updating them. + +## When a comparison fails + +Playwright writes `expected`, `actual` and `diff` images plus a trace under +`frontend/app/test-results/`. Read the diff before reaching for a tolerance: +`maxDiffPixels` is `0` on purpose, and the first run was already stable in this +environment. A threshold added before the failure is understood is a threshold +that hides the next regression. + +The usual causes, in the order worth checking: a font that had not finished +loading, an animation captured mid-play, a timestamp that fell back inside the +relative-age window, and a genuine layout change. + +## Determinism, and what it rests on + +The fixtures in `e2e/_visual.ts` answer every request the reference surfaces make, +so nothing reaches a live workspace. Identifiers are literals. Timestamps sit in +2024, beyond `formatWhen`'s one-week relative window, so a "Created" cell prints a +fixed date instead of an age that changes while you read it. Images are one inlined +1×1 PNG. Fonts are awaited before capture. + +The suite is expected to pass three consecutive unchanged runs. If it does not, the +fixture is not deterministic yet and the baseline is not worth having. diff --git a/docs/src/sidebar.mjs b/docs/src/sidebar.mjs index 38a5ec4b..78f56c68 100644 --- a/docs/src/sidebar.mjs +++ b/docs/src/sidebar.mjs @@ -58,6 +58,7 @@ export const sidebar = [ { slug: "architecture/frontend/app" }, { slug: "architecture/frontend/ui-core" }, { slug: "architecture/frontend/annotator" }, + { slug: "architecture/frontend/visual-baselines" }, ], }, ], diff --git a/frontend/app/e2e/_visual.ts b/frontend/app/e2e/_visual.ts new file mode 100644 index 00000000..0d877f0a --- /dev/null +++ b/frontend/app/e2e/_visual.ts @@ -0,0 +1,457 @@ +/** + * A workspace that renders the same picture on every machine, on every day. + * + * The visual baselines and the hostile-content scenarios share this file because + * they need the same thing: the real application, drawing real components, over + * data that cannot drift. Everything a reference surface asks for is answered + * here, so nothing escapes to a developer's workspace and no snapshot depends on + * what happens to be in it. + * + * ## What makes it deterministic + * + * **Identifiers are literals.** No `crypto.randomUUID`, no counter, no generator: + * a fixture that invents its own ids is a fixture whose diff is unreadable. + * + * **Timestamps sit in 2024.** `formatWhen` renders a relative age under a week and + * an absolute date beyond it, so a date this old prints the same string forever — + * where "3m ago" would print a different one every run. It is the least invasive + * of the options: no clock is patched and the product's own formatting is what + * gets exercised. The locale that turns it into text is pinned by the `visual` + * project rather than inherited from the container. + * + * **Images are bytes, not URLs.** One 1x1 PNG answers every thumbnail and asset + * read, so nothing is fetched and nothing decodes differently on another machine. + * + * ## What it deliberately does not do + * + * No React component is mocked and no screen is replaced. The application under + * the snapshot is the application, or the snapshot is worth nothing. + */ + +import { expect, type Page, type Route } from "@playwright/test"; + +import { assetActions, batchActions, jobActions, type Wire } from "./_wire"; + +export const VISUAL = { + project: "aaaaaaaa-1111-4111-8111-aaaaaaaaaaaa", + batch: "bbbbbbbb-2222-4222-8222-bbbbbbbbbbbb", + dataset: "cccccccc-3333-4333-8333-cccccccccccc", + job: "dddddddd-4444-4444-8444-dddddddddddd", + connection: "eeeeeeee-5555-4555-8555-eeeeeeeeeeee", + asset: "ffffffff-6666-4666-8666-ffffffffffff", +} as const; + +/** + * Well beyond `formatWhen`'s one-week relative window, so every "Created" cell + * prints an absolute date instead of an age that changes while you read it. + */ +export const FIXED_ISO = "2024-03-14T09:00:00Z"; + +/** A 1x1 PNG. Small enough to inline, real enough for an `` to decode. */ +export const PIXEL = Buffer.from( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==", + "base64", +); + +/* ------------------------------------------------------------------ hostile content */ + +/** + * The shapes Phase 7 could not seed through the write path, as literals. + * + * Deliberately stressful and still legal wire data — a screen that breaks on one + * of these is breaking on something a real workspace can hold, not on a value the + * domain would have refused. + */ +export const LONG_PROJECT_NAME = + "Autonomous-Highway-Perception-Dataset-Q3-2026-Revalidation-Cohort-With-Adverse-Weather-And-Night-Conditions-Extended"; + +export const LONG_PROJECT_DESCRIPTION = + "Every frame in this cohort was re-validated against the revised occlusion policy " + + "after the Q2 audit found that partially visible pedestrians behind stationary " + + "vehicles were being labelled inconsistently between the day and night splits."; + +export const LONG_CLASS_NAME = + "pedestrian-crossing-with-partial-occlusion-under-adverse-weather-night-split"; + +export const LONG_MODEL_REFERENCE = "facebook/sam2.1-hiera-base-plus-vitdet-cascade-refiner"; + +export const LONG_MODEL_REVISION = "b73207f4a1c92e6d5b8f0a3c7e14d9b2f6a8c0e5"; + +export const LONG_CONNECTION_NAME = + "sam2-local-cuda-fp16-highway-perception-revalidation-cohort-node-07"; + +export const LONG_REFUSAL = + "This batch cannot be completed because 4 of its 6 frames are still unannotated, " + + "and completing a batch is what pins its schema version and cuts its jobs, which " + + "has no route back once it has happened."; + +export const LONG_CONTENT_HASH = + "fc4411996bddda3239f9a8fe9d3121394816f586c1162d000c28a158f658f99f"; + +/* ------------------------------------------------------------------ wire fixtures */ + +const NO_PROGRESS = { + unannotated: 0, + pre_labeled: 0, + annotated: 0, + skipped: 0, + review_pending: 0, + accepted: 0, + total: 0, +} satisfies Wire["ProgressCounts"]; + +export interface VisualOptions { + /** Swaps the ordinary class list for one carrying a very long name. */ + readonly longClassName?: boolean; + /** Swaps the ordinary project for one named and described at length. */ + readonly longProjectText?: boolean; + /** Swaps the ordinary connection for one with a long reference and name. */ + readonly longModelReference?: boolean; + /** How many annotations the job's frame carries. Default 3. */ + readonly annotationCount?: number; + /** Fails the project list with this refusal instead of answering it. */ + readonly refusal?: string; +} + +function project(options: VisualOptions): Wire["ProjectOut"] { + return { + id: VISUAL.project, + name: options.longProjectText === true ? LONG_PROJECT_NAME : "highway-perception", + description: options.longProjectText === true ? LONG_PROJECT_DESCRIPTION : "Motorway frames.", + thumbnail_asset_id: VISUAL.asset, + thumbnail_hash: LONG_CONTENT_HASH, + created_at: FIXED_ISO, + }; +} + +/** + * The list the reference image is taken of, four rows deep. + * + * One row would render the same components, and the picture would be seven parts + * empty space to one part table — which is a reference that fails for reasons + * nobody can read and protects a rhythm it never shows. Four rows put the row + * height, the divider, the column alignment and the thumbnail column on screen + * repeatedly, which is where a spacing regression actually becomes visible. + * + * Every field is a literal, including the dates: they differ from each other so + * the sort column has something to be right about, and all of them sit beyond the + * relative-age window so each prints a fixed string. + */ +function projectList(options: VisualOptions): Wire["ProjectPage"] { + const rest: Wire["ProjectOut"][] = [ + { + id: "aaaaaaaa-1111-4111-8111-aaaaaaaaaaab", + name: "urban-intersections", + description: "Junction footage, four cities.", + thumbnail_asset_id: VISUAL.asset, + thumbnail_hash: LONG_CONTENT_HASH, + created_at: "2024-02-29T09:00:00Z", + }, + { + id: "aaaaaaaa-1111-4111-8111-aaaaaaaaaaac", + name: "night-split-revalidation", + description: null, + thumbnail_asset_id: null, + thumbnail_hash: null, + created_at: "2024-02-11T09:00:00Z", + }, + { + id: "aaaaaaaa-1111-4111-8111-aaaaaaaaaaad", + name: "tunnel-lighting", + description: "Low-light frames held back from the main cohort.", + thumbnail_asset_id: VISUAL.asset, + thumbnail_hash: LONG_CONTENT_HASH, + created_at: "2024-01-05T09:00:00Z", + }, + ]; + return { items: [project(options), ...rest], total: 4 }; +} + +function schema(options: VisualOptions): Wire["SchemaVersionOut"] { + const classes: Wire["SchemaVersionOut"]["classes"] = + options.longClassName === true + ? [ + { name: LONG_CLASS_NAME, geometries: ["bbox"], color: "#38bdf8", attributes: [] }, + { name: "vehicle", geometries: ["bbox"], color: "#f56200", attributes: [] }, + ] + : [ + { name: "vehicle", geometries: ["bbox"], color: "#38bdf8", attributes: [] }, + { name: "lane", geometries: ["polygon"], color: "#f56200", attributes: [] }, + { name: "pedestrian", geometries: ["bbox"], color: "#22c55e", attributes: [] }, + ]; + return { project_id: VISUAL.project, version: 4, classes }; +} + +function batch(): Wire["BatchOut"] { + return { + id: VISUAL.batch, + project_id: VISUAL.project, + name: "drive-01", + state: "in_annotation", + allowed_actions: batchActions("in_annotation"), + promoted_asset_count: 0, + parent_batch_id: null, + pre_label_run: null, + schema_version: 4, + asset_count: 1, + progress: { ...NO_PROGRESS, unannotated: 1, total: 1 }, + }; +} + +function connection(options: VisualOptions): Wire["ConnectionOut"] { + const long = options.longModelReference === true; + return { + id: VISUAL.connection, + name: long ? LONG_CONNECTION_NAME : "sam2-local", + connection_type: "local", + model_id: long ? LONG_MODEL_REFERENCE : "facebook/sam2.1-hiera-base-plus", + model_revision: long ? LONG_MODEL_REVISION : "b73207", + device: "cuda", + precision: "fp16", + endpoint_url: null, + provider_id: "sam", + credential_env: null, + origin: "huggingface", + setup_state: "ready", + allowed_actions: ["download_weights", "update", "delete"], + capabilities: ["point_suggest"], + produces: ["bbox", "polygon"], + download: null, + integrity_check: null, + created_at: FIXED_ISO, + updated_at: FIXED_ISO, + }; +} + +/** + * One frame's annotations, laid out on a grid so the picture is reproducible. + * + * The ids are real UUIDs, derived from the index rather than generated: the wire + * declares `id` as `format: uuid` and the runtime checks the generated client runs + * enforce it, so a readable `annotation-0007` is rejected before the screen sees + * it — and a rejected page is indistinguishable from a slow one, because the + * workspace simply never leaves "Loading the job". + */ +function annotations(options: VisualOptions): Wire["AnnotationPage"] { + const count = options.annotationCount ?? 3; + const names = ["vehicle", "lane", "pedestrian"] as const; + const items: Wire["AnnotationOut"][] = Array.from({ length: count }, (_, index) => { + const tail = String(index).padStart(12, "0"); + return { + id: `a0000000-0000-4000-8000-${tail}`, + asset_id: VISUAL.asset, + job_id: VISUAL.job, + label_class: names[index % names.length] as string, + geometry: { + type: "bbox", + x: 24 + (index % 10) * 58, + y: 24 + Math.floor(index / 10) * 44, + width: 46, + height: 32, + }, + attributes: {}, + confidence: null, + model_ref: null, + provenance: "human", + schema_version: 4, + }; + }); + return { items, total: items.length }; +} + +/* ------------------------------------------------------------------ the server */ + +/** + * Answers every request the reference surfaces make. + * + * The fallback at the end is deliberate and loud in the only way a fixture can be: + * an unstubbed path gets an empty page rather than escaping to whatever is + * listening, so a screen that starts asking something new renders empty instead of + * rendering somebody's workspace. + */ +export async function serveVisualApi(page: Page, options: VisualOptions = {}): Promise { + await page.route("**/api/**", (route: Route) => { + const path = new URL(route.request().url()).pathname.replace(/^\/api/, ""); + const P = VISUAL.project; + + if (path === "/session") return route.fulfill({ json: { issued: false } }); + + if (path === "/home") { + return route.fulfill({ + json: { + totals: { projects: 1, assets: 1, annotations: 3, releases: 0 }, + resume: null, + } as unknown as Wire["HomeOut"], + }); + } + + if (path === "/projects") { + if (options.refusal !== undefined) { + return route.fulfill({ + status: 500, + json: { code: "CONSTRAINT_VIOLATED", message: options.refusal, detail: null }, + }); + } + return route.fulfill({ json: projectList(options) satisfies Wire["ProjectPage"] }); + } + if (path === `/projects/${P}`) return route.fulfill({ json: project(options) }); + if (path.endsWith("/schema")) return route.fulfill({ json: schema(options) }); + // The pinned version is asked for by more than one path shape - a batch + // resolves its own pin, a project asks for the active one - so this matches on + // the tail rather than on a route the caller happened to use when it was written. + if (path.includes("/schema/versions/")) return route.fulfill({ json: schema(options) }); + if (path === `/projects/${P}/batches`) { + return route.fulfill({ json: { items: [batch()], total: 1 } satisfies Wire["BatchPage"] }); + } + if (path === `/projects/${P}/dataset`) { + return route.fulfill({ + json: { + id: VISUAL.dataset, + project_id: P, + name: "highway-perception", + description: null, + } as unknown as Wire["DatasetOut"], + }); + } + if (path === `/datasets/${VISUAL.dataset}/stats`) { + return route.fulfill({ + json: { + dataset_id: VISUAL.dataset, + asset_count: 1, + annotated_asset_count: 1, + annotation_count: options.annotationCount ?? 3, + classes: [], + } as unknown as Wire["DatasetStatsOut"], + }); + } + if (path === `/projects/${P}/stats`) { + return route.fulfill({ + json: { + project_id: P, + asset_count: 1, + annotated_asset_count: 1, + annotation_count: options.annotationCount ?? 3, + class_count: schema(options).classes.length, + annotated_pct: 100, + classes: [], + last_ingest_at: FIXED_ISO, + } as unknown as Wire["ProjectStatsOut"], + }); + } + + if (path.startsWith("/inference/connections")) { + return route.fulfill({ + json: { items: [connection(options)], total: 1 } satisfies Wire["ConnectionPage"], + }); + } + if (path.startsWith("/inference/providers")) { + return route.fulfill({ json: { items: [], total: 0 } as unknown as Wire["ProviderPage"] }); + } + + if (path === `/jobs/${VISUAL.job}`) { + return route.fulfill({ + json: { + id: VISUAL.job, + batch_id: VISUAL.batch, + state: "in_progress", + asset_count: 1, + allowed_actions: jobActions("in_progress"), + assignee: null, + } satisfies Wire["JobOut"], + }); + } + if (path === `/jobs/${VISUAL.job}/progress`) { + return route.fulfill({ + json: { ...NO_PROGRESS, unannotated: 1, total: 1 } satisfies Wire["ProgressCounts"], + }); + } + if (path === `/batches/${VISUAL.batch}`) return route.fulfill({ json: batch() }); + if (path === `/batches/${VISUAL.batch}/assets`) { + return route.fulfill({ + json: { + items: [ + { + id: VISUAL.asset, + project_id: P, + modality: "image", + content_hash: LONG_CONTENT_HASH, + width: 640, + height: 480, + format: "png", + source_id: null, + frame_index: 0, + frame_timestamp: null, + thumbnail_hash: LONG_CONTENT_HASH, + ingested_at: FIXED_ISO, + job_id: VISUAL.job, + progress: "annotated", + allowed_actions: assetActions("annotated"), + annotation_count: options.annotationCount ?? 3, + min_confidence: null, + }, + ], + total: 1, + } satisfies Wire["BatchAssetPage"], + }); + } + + if (path.endsWith("/annotations")) return route.fulfill({ json: annotations(options) }); + if (path.endsWith("/content") || path.endsWith("/thumbnail")) { + return route.fulfill({ contentType: "image/png", body: PIXEL }); + } + + return route.fulfill({ json: { items: [], total: 0 } }); + }); +} + +/* ------------------------------------------------------------------ navigation */ + +/** Through the token gate and onto `path`, with the fixture already serving. */ +export async function openVisual( + page: Page, + path: string, + options: VisualOptions = {}, +): Promise { + await serveVisualApi(page, options); + await page.goto(path); + await page.getByTestId("token-input").fill("a-token"); + await page.getByTestId("token-submit").click(); +} + +/** The dark theme, applied the way the stylesheet expects to find it. */ +export async function useDark(page: Page): Promise { + await page.evaluate(() => document.documentElement.classList.add("dark")); +} + +/** + * Both bundled families, resolved before anything is captured. + * + * A screenshot taken while Inter or Geist is still loading is a screenshot of the + * fallback stack, and it differs from the next one by a whole typeface. + */ +export async function fontsReady(page: Page): Promise { + await page.evaluate(async () => { + await document.fonts.ready; + }); +} + +/** + * The page is settled enough to photograph: fonts resolved and no image still + * decoding. Both are awaited rather than slept through, per `e2e_discipline`. + */ +export async function readyForCapture(page: Page): Promise { + await fontsReady(page); + await page.evaluate(async () => { + const images = [...document.querySelectorAll("img")]; + await Promise.all(images.filter((i) => !i.complete).map((i) => i.decode().catch(() => {}))); + }); +} + +/** The page must not scroll sideways — Phase 7's floor, asserted here too. */ +export async function expectNoPageOverflow(page: Page, where: string): Promise { + const overflow = await page.evaluate(() => { + const de = document.documentElement; + return de.scrollWidth - de.clientWidth; + }); + expect(overflow, `${where} scrolls the document sideways`).toBeLessThanOrEqual(0); +} diff --git a/frontend/app/e2e/stress.spec.ts b/frontend/app/e2e/stress.spec.ts new file mode 100644 index 00000000..63f7d44e --- /dev/null +++ b/frontend/app/e2e/stress.spec.ts @@ -0,0 +1,147 @@ +/** + * Content the product has to survive, in shapes a real workspace can hold. + * + * Phase 7 measured the responsive floor against whatever data happened to exist, + * which is how a long project description reached production and stretched a table + * row to five hundred pixels. These are the cases that pass could not seed: the + * write path refuses some of them and the rest need a workspace nobody has. + * `_visual.ts` supplies them as typed wire instead. + * + * The assertions are about layout rather than appearance. A screenshot would show + * that something went wrong but not say what, and it cannot answer the question + * these scenarios exist to ask — whether the *page* scrolls sideways, and whether + * the controls are still reachable. The visual baseline is next door and covers + * what a picture is actually better at. + */ + +import { expect, test } from "@playwright/test"; + +import { + LONG_CLASS_NAME, + LONG_CONNECTION_NAME, + LONG_MODEL_REFERENCE, + LONG_PROJECT_NAME, + LONG_REFUSAL, + VISUAL, + expectNoPageOverflow, + openVisual, +} from "./_visual"; + +const FLOOR = [320, 375, 480] as const; + +test("a very long project name and description do not widen the page", async ({ page }) => { + await openVisual(page, "/projects", { longProjectText: true }); + await expect(page.getByTestId("projects-screen")).toBeVisible(); + await expect(page.getByText(LONG_PROJECT_NAME)).toBeVisible(); + + for (const width of FLOOR) { + await page.setViewportSize({ width, height: 800 }); + await expectNoPageOverflow(page, `Projects at ${width}px`); + } + + // The row stays a row. Unclamped, the description wrapped a fragment per line + // and took it past four hundred pixels, which is the defect this guards. + await page.setViewportSize({ width: 1280, height: 900 }); + const height = await page + .getByTestId("projects-screen") + .locator("tbody tr") + .first() + .evaluate((row) => row.getBoundingClientRect().height); + expect(height, "a long description must not stretch its row").toBeLessThan(160); +}); + +test("a very long class name does not widen the schema", async ({ page }) => { + await openVisual(page, `/projects/${VISUAL.project}/schema`, { longClassName: true }); + await expect(page.getByTestId("project-screen")).toBeVisible(); + + // The card, not the text. Matching on the name reaches two nodes - the rendered + // one and a zero-sized span beside it - and asking the first of those to be + // visible asserts something about a measurement node rather than about the + // screen. + const card = page.getByTestId("class-0"); + await expect(card).toBeVisible(); + await expect(card).toContainText(LONG_CLASS_NAME.slice(0, 24)); + + for (const width of FLOOR) { + await page.setViewportSize({ width, height: 800 }); + await expectNoPageOverflow(page, `Schema at ${width}px with a long class name`); + // The name may wrap or truncate, but the card that holds it may not grow past + // the column it sits in. + const escapes = await card.evaluate((el) => { + const parent = el.parentElement; + if (parent === null) return 0; + return Math.round(el.getBoundingClientRect().right - parent.getBoundingClientRect().right); + }); + expect(escapes, `the class card escapes its column at ${width}px`).toBeLessThanOrEqual(1); + } +}); + +test("a very long model reference does not widen the models grid", async ({ page }) => { + await openVisual(page, "/models", { longModelReference: true }); + await expect(page.getByTestId("models-screen")).toBeVisible(); + await expect(page.getByText(LONG_CONNECTION_NAME, { exact: false }).first()).toBeVisible(); + + for (const width of FLOOR) { + await page.setViewportSize({ width, height: 800 }); + await expectNoPageOverflow(page, `Models at ${width}px with a long model reference`); + } + + // The reference is technical text: it may wrap or truncate, but it may not push + // the card wider than the column it sits in. + await page.setViewportSize({ width: 1280, height: 900 }); + const escapes = await page.evaluate((reference) => { + const holder = [...document.querySelectorAll("*")].find( + (el) => el.childElementCount === 0 && (el.textContent ?? "").includes(reference), + ); + if (!holder) return null; + const card = holder.closest("[data-testid^='model-card'], article, li, section"); + if (!card) return null; + return holder.getBoundingClientRect().right - card.getBoundingClientRect().right; + }, LONG_MODEL_REFERENCE); + if (escapes !== null) { + expect(escapes, "the model reference must stay inside its card").toBeLessThanOrEqual(1); + } +}); + +test("a large annotation count keeps the panel usable", async ({ page }) => { + // Enough for three-digit numbering and a scrolling panel, and no more: a + // thousand rows would prove the same point and cost the suite a minute. + await openVisual(page, `/jobs/${VISUAL.job}`, { annotationCount: 120 }); + await expect(page.getByTestId("annotation-page")).toBeVisible(); + + const rows = page.locator("[data-testid^='object-row-']"); + await expect(rows.first()).toBeVisible(); + + // The count is rendered, and the row numbering reaches three digits without the + // counter or the row growing into its neighbours. + await expect(page.getByTestId("object-count")).toContainText("120"); + await expectNoPageOverflow(page, "the annotator with 120 annotations"); + + // The panel owns its own scrolling rather than handing it to the document. + const ownsScroll = await page.evaluate(() => { + const row = document.querySelector("[data-testid^='object-row-']"); + for (let el = row?.parentElement ?? null; el; el = el.parentElement) { + const oy = getComputedStyle(el).overflowY; + if ((oy === "auto" || oy === "scroll") && el.scrollHeight > el.clientHeight + 1) return true; + } + return false; + }); + expect(ownsScroll, "the annotations region should scroll within itself").toBe(true); +}); + +test("a long refusal wraps instead of widening the page", async ({ page }) => { + await openVisual(page, "/projects", { refusal: LONG_REFUSAL }); + // The read fails three times before the screen gives up, so the assertion waits + // for the refusal rather than for a moment in the retry. + await expect(page.getByText(LONG_REFUSAL, { exact: false })).toBeVisible({ timeout: 15_000 }); + + for (const width of FLOOR) { + await page.setViewportSize({ width, height: 800 }); + await expectNoPageOverflow(page, `a refusal at ${width}px`); + } + + // Whatever the refusal says, the way out of it stays on screen. + await page.setViewportSize({ width: 320, height: 800 }); + const retry = page.getByRole("button").first(); + await expect(retry).toBeVisible(); +}); diff --git a/frontend/app/e2e/visual.spec.ts b/frontend/app/e2e/visual.spec.ts new file mode 100644 index 00000000..f1626f41 --- /dev/null +++ b/frontend/app/e2e/visual.spec.ts @@ -0,0 +1,125 @@ +/** + * The visual baseline: a small set of reference surfaces, in both themes. + * + * ## Why these five and not the route list + * + * A snapshot is worth keeping when it would catch a change nothing else does — a + * token resolving to the wrong colour, a font that failed to load, an icon that + * stopped drawing, a radius or a gap that moved. Those are properties of the + * design system rather than of a route, so the set is chosen for the *variety of + * system* each surface puts on screen, and stops there: + * + * - **Projects** is the shell and a data surface at once — rail, page header, + * table density, a thumbnail, buttons and meta text. + * - **Schema** is the form vocabulary: inputs, labels, panels, the wrapped action + * row Phase 7 fixed. + * - **Models** is cards, badges and technical text, which is where a card grid's + * geometry regresses without any assertion noticing. + * - **An open menu** is the one surface whose palette is deliberately inverted, so + * it is the only place a `menuColor` regression shows. + * - **The annotator** carries the densest chrome in the product: top bar, tool + * palette, stage, panel, zoom and the highest icon count anywhere. + * + * Home, Batches, Dataset, Gallery and Ingest are covered by behavioural specs and + * add no design-system vocabulary these five do not already hold. A baseline that + * mostly photographs empty space is a baseline that fails for reasons nobody can + * read. + * + * ## Locators, mostly + * + * The screenshot is of the surface that owns the contract, not of the window, + * except where the composition between shell and content *is* the subject. A page + * screenshot of a long data surface amplifies every irrelevant difference, and + * `fullPage` would photograph a scroll position as well as a layout. + */ + +import { expect, test } from "@playwright/test"; + +import { VISUAL, openVisual, readyForCapture, useDark } from "./_visual"; + +const DESKTOP = { width: 1440, height: 900 }; +const NARROW = { width: 420, height: 900 }; + +test.describe("the shell and a data surface", () => { + for (const theme of ["light", "dark"] as const) { + test(`Projects, desktop, ${theme}`, async ({ page }) => { + await page.setViewportSize(DESKTOP); + await openVisual(page, "/projects"); + await expect(page.getByTestId("projects-screen")).toBeVisible(); + if (theme === "dark") await useDark(page); + await readyForCapture(page); + + // The window, because the rail beside the content is half of what this + // reference is for. + await expect(page).toHaveScreenshot(`projects-desktop-${theme}.png`); + }); + } +}); + +test.describe("a form surface", () => { + test("Schema, desktop, dark", async ({ page }) => { + await page.setViewportSize(DESKTOP); + await openVisual(page, `/projects/${VISUAL.project}/schema`); + await expect(page.getByTestId("project-screen")).toBeVisible(); + await useDark(page); + await readyForCapture(page); + await expect(page.getByTestId("project-screen")).toHaveScreenshot("schema-desktop-dark.png"); + }); + + test("Schema, narrow, light", async ({ page }) => { + // The width where the action row wraps rather than widening the page — the + // composition Phase 7 changed, and the one worth a picture. + await page.setViewportSize(NARROW); + await openVisual(page, `/projects/${VISUAL.project}/schema`); + await expect(page.getByTestId("project-screen")).toBeVisible(); + await readyForCapture(page); + await expect(page.getByTestId("project-screen")).toHaveScreenshot("schema-narrow-light.png"); + }); +}); + +test.describe("a card surface", () => { + for (const theme of ["light", "dark"] as const) { + test(`Models, desktop, ${theme}`, async ({ page }) => { + await page.setViewportSize(DESKTOP); + await openVisual(page, "/models"); + await expect(page.getByTestId("models-screen")).toBeVisible(); + if (theme === "dark") await useDark(page); + await readyForCapture(page); + await expect(page.getByTestId("models-screen")).toHaveScreenshot( + `models-desktop-${theme}.png`, + ); + }); + } +}); + +test.describe("an inverted floating surface", () => { + for (const theme of ["light", "dark"] as const) { + test(`an open menu, ${theme}`, async ({ page }) => { + await page.setViewportSize(DESKTOP); + await openVisual(page, `/projects/${VISUAL.project}/batches`); + await expect(page.getByTestId("batches-screen")).toBeVisible(); + if (theme === "dark") await useDark(page); + await readyForCapture(page); + + await page.getByTestId("batch-overflow-drive-01").click(); + const menu = page.getByRole("menu"); + await expect(menu).toBeVisible(); + // The menu carries the `dark` subtree in both themes; that inversion is the + // whole reason this surface earns a reference of its own. + await expect(menu).toHaveScreenshot(`menu-open-${theme}.png`); + }); + } +}); + +test.describe("the annotation workspace", () => { + for (const theme of ["light", "dark"] as const) { + test(`the annotator, desktop, ${theme}`, async ({ page }) => { + await page.setViewportSize(DESKTOP); + await openVisual(page, `/jobs/${VISUAL.job}`); + await expect(page.getByTestId("annotation-page")).toBeVisible(); + if (theme === "dark") await useDark(page); + await readyForCapture(page); + await expect(page).toHaveScreenshot(`annotator-desktop-${theme}.png`); + }); + } +}); diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/annotator-desktop-dark-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/annotator-desktop-dark-visual-linux.png new file mode 100644 index 00000000..bf1993e4 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/annotator-desktop-dark-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/annotator-desktop-light-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/annotator-desktop-light-visual-linux.png new file mode 100644 index 00000000..a3fc21ff Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/annotator-desktop-light-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/menu-open-dark-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/menu-open-dark-visual-linux.png new file mode 100644 index 00000000..93eeed36 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/menu-open-dark-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/menu-open-light-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/menu-open-light-visual-linux.png new file mode 100644 index 00000000..5720bfec Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/menu-open-light-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/models-desktop-dark-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/models-desktop-dark-visual-linux.png new file mode 100644 index 00000000..96fc29e2 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/models-desktop-dark-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/models-desktop-light-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/models-desktop-light-visual-linux.png new file mode 100644 index 00000000..780b8fa3 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/models-desktop-light-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/projects-desktop-dark-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/projects-desktop-dark-visual-linux.png new file mode 100644 index 00000000..c58e8660 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/projects-desktop-dark-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/projects-desktop-light-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/projects-desktop-light-visual-linux.png new file mode 100644 index 00000000..4e610f2d Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/projects-desktop-light-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/schema-desktop-dark-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/schema-desktop-dark-visual-linux.png new file mode 100644 index 00000000..8b00f4d0 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/schema-desktop-dark-visual-linux.png differ diff --git a/frontend/app/e2e/visual.spec.ts-snapshots/schema-narrow-light-visual-linux.png b/frontend/app/e2e/visual.spec.ts-snapshots/schema-narrow-light-visual-linux.png new file mode 100644 index 00000000..a442b967 Binary files /dev/null and b/frontend/app/e2e/visual.spec.ts-snapshots/schema-narrow-light-visual-linux.png differ diff --git a/frontend/app/playwright.config.ts b/frontend/app/playwright.config.ts index 3ade2faa..27d06ea2 100644 --- a/frontend/app/playwright.config.ts +++ b/frontend/app/playwright.config.ts @@ -104,7 +104,25 @@ export default defineConfig({ retries: process.env.CI ? 1 : 0, workers: resolveWorkers(), timeout: 20_000, - expect: { timeout: 5_000 }, + expect: { + timeout: 5_000, + /** + * Strict on purpose, and it stayed strict. + * + * `threshold` is per-pixel colour tolerance and `maxDiffPixels` is how many may + * differ at all; both are at their least forgiving because the first run was + * already stable in the environment the `visual` project pins. A tolerance + * added before a failure is understood is a tolerance that hides the next + * regression, so the order is: make the state deterministic, then compare + * exactly, and only widen with a reason written next to the number. + */ + toHaveScreenshot: { + animations: "disabled", + caret: "hide", + scale: "css", + maxDiffPixels: 0, + }, + }, reporter: process.env.CI ? [["github"], ["html", { open: "never" }]] : [["list"]], @@ -132,5 +150,44 @@ export default defineConfig({ // Chromium only, as v1 was. The subject of every scenario here is pointer and // keyboard semantics, and the React adapter implements those one way; tripling // the browser download would buy rendering coverage nothing in this suite asserts. - projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"], viewport: VIEWPORT } }], + projects: [ + { + name: "chromium", + use: { ...devices["Desktop Chrome"], viewport: VIEWPORT }, + // The visual project owns these; running them here too would compare a + // reference image against a second environment and call the difference a + // regression. + testIgnore: /visual\.spec\.ts$/, + }, + { + /** + * The reference images, and the environment that is allowed to produce them. + * + * A screenshot compares two renderings, so anything the renderer reads has to + * be stated rather than inherited. The locale decides what `toLocaleDateString` + * writes into a "Created" cell, the timezone decides which day it lands on, and + * `deviceScaleFactor` decides how many pixels each CSS pixel becomes — a + * baseline captured at 2 and compared at 1 differs everywhere at once. + * + * These run in one Linux Chromium and nowhere else: baselines from a second OS + * would disagree about font rasterisation, and the disagreement would be + * reported as a product regression. The Docker command that owns them is in + * `docs/content/architecture/frontend/visual-baselines.md`. + */ + name: "visual", + testMatch: /visual\.spec\.ts$/, + use: { + ...devices["Desktop Chrome"], + viewport: VIEWPORT, + deviceScaleFactor: 1, + locale: "en-US", + timezoneId: "UTC", + // Motion is settled by `expect.toHaveScreenshot`'s `animations: "disabled"` + // rather than by asking the context for reduced motion. The product honours + // `prefers-reduced-motion` by collapsing its animations, so asking for it + // here would photograph that mode instead of the ordinary one — a reference + // image of a setting most people do not have. + }, + }, + ], });