test(e2e): a deterministic visual baseline, and the hostile content Phase 7 could not seed - #776
Merged
Merged
Conversation
A screenshot is only worth keeping if the same code draws the same picture tomorrow, so this answers every request the reference surfaces make and none of it reaches a workspace. Identifiers are literals. Timestamps sit in 2024, past `formatWhen`'s one-week relative window, so a "Created" cell prints a fixed date rather than an age that changes while you read it - the product's own formatting is exercised and no clock is patched. Images are one inlined 1x1 PNG. The hostile values Phase 7 deferred are here as literals too: the write path refuses some of those shapes and the rest need a workspace nobody has, which is why that pass could only ever measure a long project name. No React component is mocked and no screen is replaced. The application under the snapshot has to be the application or the snapshot is worth nothing. Annotations are typed rather than cast. `AnnotationOut.id` is `format: uuid` and `job_id` is required, and the generated runtime checks enforce both - a readable `annotation-0007` is rejected before the screen sees it, and a rejected page is indistinguishable from a slow one because the workspace simply never leaves "Loading the job". The cast that hid that is gone.
…onment Five surfaces in both themes, chosen for the variety of design system each puts on screen rather than to cover routes: the shell over a data table, a form, a card grid, an open menu, and the annotation workspace. Between them they hold the tokens, both bundled fonts, the icon set, the radius scale, the card and dialog geometry and the deliberately inverted menu palette - which is the one surface where a `menuColor` regression can show at all. Home, Batches, Dataset, Gallery and Ingest add no vocabulary these five do not already carry. The `visual` project owns the environment because a screenshot compares two renderings and everything the renderer reads has to be stated: locale decides what a "Created" cell says, timezone decides which day it lands on, and `deviceScaleFactor` decides how many pixels a CSS pixel becomes. It is excluded from the default project, so the ordinary suite does not compare images. `maxDiffPixels` is 0 and stayed 0. The first run was already stable in this container, and a tolerance added before a failure is understood is a tolerance that hides the next regression. Motion is settled by `animations: "disabled"` at capture rather than by asking the context for reduced motion: the product honours that setting by collapsing its animations, so a baseline captured under it would describe a mode most people do not have. Every image was opened and read before it was committed. The first Projects reference was seven parts empty space to one part table, which protects a rhythm it never shows; the list is four rows deep now, and the picture carries row height, dividers, the thumbnail column in both its states, an absent description and a sort column with something to be right about.
Five scenarios over layout rather than appearance: a long project name and description, a long class name, a long model reference, a hundred and twenty annotations, and a long refusal. Each asserts what a picture is bad at answering - whether the *page* scrolls sideways, whether a card escapes its column, whether the panel still owns its own scrolling - and leaves the look of it to the baselines next door. Phase 7 measured the responsive floor against whatever data happened to exist, which is how a long description reached production and stretched a table row to five hundred pixels. These are the shapes it could not reach: the schema write path refuses one of them and the rest need a workspace nobody has. Both halves were proved by breaking them. Removing the description clamp fails "a long description must not stretch its row"; removing the schema row's wrap fails "Schema at 320px with a long class name scrolls the document sideways". The class-name scenario anchors on the card rather than on the text. Matching the name reaches two nodes - the rendered one and a zero-sized span beside it - and asking the first of those to be visible asserts something about a measurement node while the page underneath is perfectly fine.
Where the reference images come from, what pins the environment they are produced in, and the one command that updates them. Baselines are not generated on macOS or Windows: font rasterisation differs between operating systems, and a baseline captured on one and compared on another reports that difference as a product regression. The update path is deliberately a separate command rather than a flag an ordinary run can reach, and the note says the part that matters - look at the images, then compare again. A regenerated baseline nobody opened is a regression written down as the new truth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ten reference images and five hostile-content scenarios, both built on one typed
fixture layer that answers every request the surfaces make.
These were the two things Phase 7 deferred on purpose. It declined to add
screenshots because the product renders live thumbnails, relative timestamps and
generated ids, and a suite that fails on those is noise rather than protection; and
it could not seed a long class name or a long model reference because the write
path refuses those shapes. Both objections are answered by making the data typed
and fixed rather than by relaxing what a screenshot is allowed to contain.
New dependencies: 0. Playwright's own comparison does the work.
The canonical environment
A screenshot compares two renderings, so everything the renderer reads is stated
rather than inherited.
The
visualproject owns those and is excluded from the default project, so theordinary suite does not compare images. Baselines are generated in this container
and nowhere else: font rasterisation differs between operating systems, and a
baseline captured on one and compared on another reports that difference as a
product regression.
What earns a baseline
Five surfaces in both themes, chosen for the variety of design system each puts on
screen rather than to cover routes.
Home, Batches, Dataset, Gallery and Ingest add no vocabulary these five do not
already carry, and a baseline that mostly photographs empty space fails for reasons
nobody can read.
Most are locator screenshots. Projects and the annotator are page screenshots
because the composition between shell and content is itself the subject. Nothing
uses
fullPage, which would photograph a scroll position as well as a layout.maxDiffPixelsis 0 and stayed 0. The first run was already stable here, and atolerance added before a failure is understood is a tolerance that hides the next
regression.
Determinism
Three consecutive runs, no baseline update between them.
What it rests on: identifiers are literals; timestamps sit in 2024, past
formatWhen's one-week relative window, so a "Created" cell printsMar 14, 2024instead of an age that changes while you read it; images are one inlined 1×1 PNG;
fonts are awaited before capture; motion is settled by
animations: "disabled"atcapture time.
Not by asking the context for reduced motion, which was the first attempt and is
wrong twice over: the type does not exist on
usein this version, and the producthonours that setting by collapsing its animations — so a baseline captured under it
would describe a mode most people do not have.
Hostile content
These assert layout rather than appearance — whether the page scrolls sideways,
whether a card escapes its column, whether the annotations region still owns its
own scrolling. A picture is a poor way to ask any of those, and the baselines next
door cover what a picture is better at.
Both halves were proved by breaking them. Removing the description clamp fails with
a long description must not stretch its row; removing the schema row's wrap failswith
Schema at 320px with a long class name scrolls the document sideways.No new product defect was found. Every hostile case passes against current
main, which is the expected result after Phase 7 fixed the responsive floor and isworth stating rather than leaving as an absence.
Three things the work turned up
The annotator would not render, and the fixture was at fault. All eight requests
answered 200 with no console error, and the screen stayed on "Loading the job".
AnnotationOut.idisformat: uuidandjob_idis required; readable ids likeannotation-0007are rejected by the generated runtime checks before the screensees them — and a rejected page is indistinguishable from a slow one. An
as unknown ascast is what let the wrong shape through, so the annotations aretyped now.
The first Projects baseline was mostly empty. One row renders the same
components but protects a rhythm it never shows. The list is four rows deep, and the
image carries row height, dividers, the thumbnail column in both its states, an
absent description and a sort column with something to be right about.
A failing stress test that was not a defect. The long-class-name scenario failed
while the page overflow measured 0: the text matches two nodes, the rendered one and
a zero-sized span beside it, and the assertion had picked the measurement node. It
anchors on the card now.
Test plan
Every command ran inside Docker/Linux; no repository validation ran natively on the
host.
The update command is in
docs/content/architecture/frontend/visual-baselines.md, and there is nohost-native path to it.
Behavioural coverage
Nothing was deleted to make room for a screenshot. Navigation, accessible names,
focus, keyboard, requests, sorting, mutation and overflow assertions all remain
where they were; the baselines are additional.
Overlap
No file overlap with any open PR. One semantic note worth carrying: the frontend
dependency bump (#775) raises
shadcnfrom 4.18.0 to 4.19.0, and that package is aruntime CSS dependency here —
styles.cssimportsshadcn/tailwind.css. It doesnot touch Playwright, so the environment is unaffected, but if the bump moves base
styles this suite will go red on merge. That is the gate working. Read the diff
images before regenerating.
Scope
Remaining debt
Test infrastructure: the baseline covers five surfaces, and Batches, Dataset,
Gallery and Ingest could earn one later if they grow vocabulary these do not hold.
Nothing here is design-system debt.