diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..0456d9c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,67 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +An Astro Starlight static site of lab learning materials (Research 101, RL, Speculative Decoding, Cache Coherence). Content is MDX; the distinguishing feature is that nearly every concept ships with a **live React simulator** rather than a static figure. Deployed to GitHub Pages at `https://stable-lab.github.io/stable-learning` on every push to `main` (`.github/workflows/deploy.yml`). + +## Commands + +```bash +npm install # Node 22 (.nvmrc) +npm run dev # http://localhost:4321/stable-learning/ (base path applies in dev too) +npm run build # the only real check — no test suite, no lint script +npm run preview # serve dist/; needed to exercise search (Pagefind only builds in prod) +``` + +There is no test runner and no lint script. `@biomejs/biome` is a devDependency with **no `biome.json`**, so formatting is Biome defaults (tabs, double quotes) — matching the existing `.tsx` files. Run it explicitly if needed: `npx @biomejs/biome check src/`. + +`npm run build` is the acceptance gate. Per `MILESTONES.md`: run it after each increment and keep the site shippable. + +## Architecture + +**Sidebar is half-automatic.** Pages autogenerate within a directory, but each chapter directory must be registered by hand as an `items` entry in `astro.config.mjs`. Creating `src/content/docs//0N-chapter/` alone does nothing — add the `autogenerate: { directory: '/0N-chapter' }` line too. Within a chapter, ordering comes from `sidebar.order` in each file's frontmatter. + +**Base path is the recurring footgun.** `base: '/stable-learning'` means any absolute internal URL (`/rl/01-.../`, string-concatenated `import.meta.env.BASE_URL`) 404s on Pages while looking fine locally. Use **relative hrefs** in MDX (`href="rl/01-action-chain-rewards/"`) and import images through the asset pipeline (`import logo from '../../assets/x.jpeg'` → `logo.src`), as `src/components/starlight/Footer.astro` does. Both classes of bug have shipped before. + +**Math** is remark-math + rehype-katex, configured at the `markdown` level in `astro.config.mjs` — `$inline$` and `$$display$$` work in any MDX page with no per-file import. + +**Theme** (`src/styles/custom.css`) overrides Starlight's color tokens for a warm-paper editorial look: serif (Newsreader) prose, sans (Inter) for UI/tables/asides. Note Starlight's inverted semantics — `--sl-color-white` is the *text* color, `--sl-color-black` the *background*; they flip per theme. Dark is the default; light is `:root[data-theme='light']`. + +### Visualizations (`src/components/visualizations/`) + +Three rendering paths, deliberately split: + +- **Animated / frame-rate widgets** use the dependency-free stack in `lib/`: `useSimLoop` (rAF loop that batches ticks so high rates don't spam React renders, caps `dt` so a backgrounded tab doesn't fast-forward), `SimShell` (play/pause/step/reset chrome, speed selector, readout chips), `Sparkline` (SVG line chart, autoscaling, EMA + reference line). ~20 components. +- **Static parameter-explorers** use `LazyPlot.tsx`, a Plotly wrapper that lazy-loads `react-plotly.js` and injects site theme defaults (font, colorway, transparent backgrounds, hidden modebar) plus a `MutationObserver` on `data-theme` so charts re-render on theme toggle. Caller `layout`/`config` wins over the defaults. ~16 components. +- **Canvas**, for anything with hundreds of marks per frame — `lib/PixelCanvas.tsx` (images, via one `ImageData` blit) and inline `` refs in `MixingReversibility`/`GANDuel` (particle clouds). Introduced by the diffusion track: a 24×24 frame is 576 cells and re-rendering that many React ``s per frame across several panels spends the whole frame budget in reconciliation. **`PixelCanvas` renders grayscale in both themes on purpose** — theme-inverting an image makes one sprite read as two different images — so only its frame is theme-aware. It redraws on every commit (no dep array) because sims mutate their `Float32Array`s in place. + +**Domain libraries** under `lib/`, all pure and unit-testable outside React: `diffusionMath.ts` (schedule, forward sampling, exact posterior denoiser, DDIM/DDPM step), `sprites.ts` (procedural training images + poses + labels), `spectrum.ts` (radial power spectrum by separable DFT), `pca.ts` (power iteration), `tinynn.ts` (MLP with hand-written backprop and Adam, for the live GAN). + +Plotly cannot SSR, and the sim components read `performance.now()`/DOM on mount, so MDX embeds use `client:only="react"` almost everywhere (45 uses vs 4 `client:visible`). Prefer `client:only="react"` unless the component is pure and SSR-safe. Give widgets a fixed height to avoid layout shift. + +**Color semantics are shared across all widgets** — one color, one meaning, tokens in `custom.css`: `--viz-policy` blue (thing being learned), `--viz-value` amber (critic/baseline), `--viz-reward` green, `--viz-danger` red (variance/collapse), `--viz-ref` gray (reference/old policy), `--viz-kl` purple. Don't introduce ad-hoc colors for these roles. + +## Content conventions + +House style is documented and tracked in `MILESTONES.md` (read it before adding a chapter). The bar the repo holds itself to: + +- Prose is **tension-driven**: what breaks → why → the fix. Not a definition list. +- Every simulator gets a structured caption in a `
` with bolded **What it models** / **Knobs** / **Try this** paragraphs — see `research101/01-principles/why-do-research.mdx` or `rl/05-ppo/index.mdx`. +- Simulators should *run, learn, and be able to fail* — the failure regime (reward collapse, burnout, saturation) is usually what motivates the next chapter. +- Numeric and citation claims in prose are expected to be verified against the simulator's own behavior or a named source, and `MILESTONES.md` records the verification. + +**Verify claims before writing the caption, not after.** The `lib/` modules are plain TypeScript with no React imports, so they run directly under `node --experimental-strip-types` — write a throwaway script, measure the thing the caption is about, then write the caption around the result. On the diffusion track this repeatedly overturned the planned narrative: the step-count sweep converges at T=4 rather than climbing to 1000, a stronger GAN discriminator helps rather than causing collapse, and reverse KL does not mode-seek from an arbitrary start. Each of those would have shipped as a confident false statement. `MILESTONES.md` records what was measured and what it changed. + +**Never ship a control that does nothing.** A kernel-width knob and a dead Run button were both built and then removed on the diffusion track. If a knob's effect can't be measured, cut it and say why in a code comment so it isn't re-added. + +MDX imports reach up out of `src/content/docs///` with four levels: `import X from '../../../../components/visualizations/X';`. + +Frontmatter is Starlight's `docsSchema` (`src/content.config.ts`): `title`, `description`, `sidebar: { order }`. + +## Adding a track + +1. `src/content/docs//0N-
/*.mdx` +2. New sidebar group in `astro.config.mjs` with one `autogenerate` entry per section +3. `` block in `src/content/docs/index.mdx` — **relative href**, with a "Prerequisites" line, following the existing pattern diff --git a/MILESTONES.md b/MILESTONES.md index b401dc7..0bf5522 100644 --- a/MILESTONES.md +++ b/MILESTONES.md @@ -212,5 +212,169 @@ per principle, structured captions, verified references per page. --- +## M7 — New track: Image Generation & Diffusion + +Three chapters under `diffusion/`, placed before the specdec track because +`specdec/03-parallel-drafting` already leans on diffusion for DFlash and +assumed the reader knew what it was. House style throughout: tension-first +prose, a living simulator per idea, structured captions, verified references. + +Shared infrastructure — the track needed a rendering primitive the site did +not have: + +- [x] `lib/PixelCanvas.tsx` — canvas-backed image renderer. The site was 100% + SVG/Plotly; a 24×24 frame is 576 cells and re-rendering that many React + ``s per frame across four panels spends the frame budget in + reconciliation. Grayscale in both themes on purpose (theme-inverting an + image makes one sprite read as two different images). +- [x] `lib/sprites.ts` — eight procedural 24×24 glyphs plus `makeDataset()` + for posed variants. No image assets, no licensing, byte-identical for + every reader. +- [x] `lib/diffusionMath.ts` — cosine schedule as a *continuous* function of + t/T, forward sampling, the exact closed-form posterior denoiser, and a + unified DDIM/DDPM reverse step (η = 1 / η = 0). +- [x] `lib/spectrum.ts` — radially-averaged power spectrum by separable DFT. + +**Colour semantics** follow the existing tokens: policy/blue for the thing +being predicted, value/amber for x₀, reward/green for ε and for "signal +surviving", danger/red for noise floors and failure, kl/purple for score and +entropy. + +### Chapter 2 — Destroy It, Then Learn to Undo (`02-diffusion`) + +Built first, because it is the payoff chapter and the track ships standalone +with it. + +- [x] `index.mdx` + `MixingReversibility` — ink in water and dye in corn syrup + as one simulation with one knob. Both media get identical Taylor–Couette + advection (closed form, so cranking back undoes the shear exactly); the + only difference is Brownian jitter scaled √dt. Verified: at jitter 0 the + dye smears to 6.25 rad of angular spread and returns to displacement + **0.0000**; at jitter 0.25/0.5/1.0 the forward half is statistically + identical (6.19–6.23 rad) but comes home at 0.19/0.36/0.57. + The demo is used as a **contrast, not an analogy** — the syrup unmixes + because Stokes flow destroys nothing, whereas our forward process + genuinely does. It earns its place by separating "looks destroyed" from + "is destroyed", and the page says so explicitly. +- [x] `forward-process.mdx` + `ForwardNoise` — one t slider, three panels: + image dissolving, pixel histogram sliding off its bimodal ink/paper + spikes onto 𝒩(0,1), and the power spectrum showing the flat noise floor + rising through the falling signal curve. Verified cutoff frequency + 12 → 6 → 1 → 0 at t = 0/200/700/1000, so "detail dies before silhouette" + is measured rather than asserted. +- [x] `predict-the-noise.mdx` + `ThreeTargets` — the three parameterizations + shown to be one object, and separated by error amplification: + x₀-pred ×1 flat, ε-pred √(1-ᾱ)/√ᾱ, score-pred (1-ᾱ)/√ᾱ. Verified + ε-pred ×0.006/×0.042/×0.17 at t = 1/20/100 and ×6.4/×316 at t = 900/999 + — so ε-prediction is not uniformly better, it wins hugely where quality + is decided and loses where it is not, which is the argument for + v-prediction. +- [x] `one-step-vs-many.mdx` + `StepBudget` — real DDPM/DDIM sampling with the + exact denoiser over 128 posed shapes. The T=1 mode-averaging collapse is + the lesson, and it is the same failure as `mean-field-trap` (a marginal + substituted for a conditional) — cross-linked both ways. + +**Measured findings that changed this chapter**, recorded so they are not +re-litigated: + +- The planned claim "sharpness keeps improving from T=1 to T=1000" is **false + for an exact denoiser.** Distance-to-nearest-real-image runs 0.50 / 0.063 / + 0.0032 at T = 1 / 2 / 4 and then sits at 0.0032 through T=200. Growing the + training set 8 → 1024 changed nothing; injecting per-call denoiser error + (σ = 0.15/0.3/0.6) changed nothing either — it sets a floor ≈ σ at every T. + The page now says outright that four steps converges here, and that real + systems need more because their denoiser is a *learned approximation* — with + DDIM (1000→50) and consistency models (→1–4 steps) as the evidence. +- A kernel-width knob was built to demonstrate generalization and then + **removed**: it does nothing, measured flat at 0.0032 across widths 1–8 and + across an effective-noise floor up to 0.4. In 576 dimensions the posterior + over a finite training set is effectively deterministic. That failure became + the page's argument for why generalization needs a different function class + rather than a wider kernel. + +### Chapter 1 — What Generation Asks For (`01-the-problem`) + +- [x] `index.mdx` + `ManifoldSlice` — the same two endpoints joined by two + straight lines: one through all 576 pixels (double-exposure ghosts at + the midpoint), one through the four pose parameters the sprites are + actually drawn from (valid images throughout). The pose path is a real + latent space, not a stand-in, which is why the contrast is honest. + Plus uniformly-random pixel frames for scale. +- [x] `gan.mdx` + `GANDuel` — a genuine GAN: two 2→16→16 MLPs in + `lib/tinynn.ts` with hand-written backprop and Adam(β₁=0.5), on the + eight-Gaussians benchmark. Nothing scripted. +- [x] `why-gans-broke.mdx` + `ModeCoverage` — one Gaussian fitted to a + three-mode target under forward KL, reverse KL and Jensen–Shannon, by + direct numerical integration on a 480-point grid. Closes on the + generative trilemma. + +**Measured findings that rewrote this chapter:** + +- The planned GAN narrative — "crank the discriminator learning rate and watch + the generator stall/collapse" — is **false here**. A *stronger* discriminator + helps: 4 D-steps at lr 8e-3 gives 6–8 effective modes and 9–15% of mass off + the data, versus 3–8 modes and 11–79% off with 1 step at lr 2e-3. With the + non-saturating loss a *weak* D is the danger, because a critic that cannot + tell real from fake gives the generator nothing to follow. The folk story is + about the 2014 minimax loss; the page says so. +- What *is* reproducible is **seed variance**: over 8 seeds at one fixed + config, outcomes range from 3 modes / 79% off-data to 8 modes / 11% off, + with the loss curves looking much the same either way. That became the + page's thesis, and Reset became its most important control. +- `ModeCoverage` was going to show "reverse KL locks onto one mode." From the + original default start it does **not** — it goes broad and covers 3/3. The + real structure, measured over seven starting points: forward KL lands at + μ ≈ 0.1, σ = 2.65, covering 3/3 **every time**, while JS and reverse KL end + up on whichever mode they started beside (1/3 from −3.6, −3.0, 0.1, 3.0, + 3.6). So the *start position* became the widget's only knob, and the chapter + gained a much better thesis: maximum likelihood has one answer and finds it + from anywhere; the adversarial objective has many and takes the nearest. + This is the GAN seed variance reproduced with no networks in it. + +### Chapter 3 — How Stable Diffusion Works (`03-real-systems`) + +- [x] `index.mdx` + `LatentCompress` — a real encoder/decoder: PCA by power + iteration (`lib/pca.ts`) over the 256-image dataset, with the residual + panel showing exactly what each k discards. +- [x] `conditioning.mdx` — ε_θ(x_t, t) → ε_θ(x_t, t, c); why a *contrastive* + text encoder, cross-attention as the injection mechanism, and why + conditioning is a small change here and was a research programme for + GANs. Reuses `GuidanceDial` at w = 1. +- [x] `guidance.mdx` + `GuidanceDial` — real classifier-free guidance, + ε_uncond + w(ε_cond − ε_uncond), with both scores computed exactly. + +**Measured findings:** + +- Guidance steering reproduces exactly: prompt accuracy 25% at w = 0 (near the + 12.5% chance rate) → **100% for every w ≥ 0.5**. Within-class diversity then + decays: over 32 seeds on a fixed prompt, 13 of 16 possible images at w = 1, + 8 at w = 8 and w = 15, with the most frequent single image rising 16% → 25%. +- The high-guidance **saturation artifacts do not reproduce** — distance to + nearest real image holds at 0.003 from w = 1 through w = 15. With an exact + score, extrapolating it is harmless. The artifacts in real systems are the + *learned* score's approximation error, amplified by w. Written up as such. +- PCA is a **weak** encoder for this data and the page says so with numbers: + 68.6% of variance at k = 16, 92.8% at k = 64, still visibly soft. The cause + is that two of the four degrees of freedom are rotation and translation, + which are savagely non-linear in pixel space — which is precisely why real + latent diffusion uses a convolutional non-linear autoencoder. The widget is + presented as a lower bound establishing the *ordering* (semantics cheap, + exact rendering expensive), not the achievable ratio. + +### The through-line this track ended up with + +Three separate pathologies vanish once the denoiser is exact — the thousand +step count (converges at T=4), generalization-by-kernel-widening (flat at +0.0032 across every setting), and guidance saturation (0.003 at every w). +Everything that *survives* in the exact-denoiser toy is mathematical; every +pathology that disappears was approximation error. That distinction is stated +explicitly on `guidance.mdx` and is the most useful thing the track teaches. + +- Acceptance: `npm run build` clean; both themes checked; every simulator + driven to the failure regime its caption claims; numeric claims verified by + running the actual library, not by eye. + +--- + Working agreement: after each milestone increment, run `npm run build`; keep checkboxes here current; each iteration should leave the site shippable. diff --git a/astro.config.mjs b/astro.config.mjs index f16f57e..e20cc69 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -10,8 +10,11 @@ export default defineConfig({ site: 'https://stable-lab.github.io', base: '/stable-learning', redirects: { + // The target must carry the site base: Astro emits it verbatim into the + // meta-refresh, and an absolute path without /stable-learning resolves + // to the domain root and 404s on Pages. Same trap as commit c005587. '/specdec/03-parallel-drafting/mean-field-trap/': - '/specdec/03-parallel-drafting/dspark/', + '/stable-learning/specdec/03-parallel-drafting/dspark/', }, markdown: { remarkPlugins: [remarkMath], @@ -48,6 +51,14 @@ export default defineConfig({ { label: 'GRPO', autogenerate: { directory: 'rl/06-grpo' } }, ], }, + { + label: 'Image Generation & Diffusion', + items: [ + { label: 'What Generation Asks For', autogenerate: { directory: 'diffusion/01-the-problem' } }, + { label: 'Destroy It, Then Undo It', autogenerate: { directory: 'diffusion/02-diffusion' } }, + { label: 'How Stable Diffusion Works', autogenerate: { directory: 'diffusion/03-real-systems' } }, + ], + }, { label: 'Speculative Decoding', items: [ diff --git a/src/components/visualizations/ForwardNoise.tsx b/src/components/visualizations/ForwardNoise.tsx new file mode 100644 index 0000000..3e74c34 --- /dev/null +++ b/src/components/visualizations/ForwardNoise.tsx @@ -0,0 +1,465 @@ +import { useCallback, useMemo, useRef, useState } from "react"; +import PixelCanvas from "./lib/PixelCanvas"; +import SimShell from "./lib/SimShell"; +import { alphaBar, gaussian, makeRng, snr } from "./lib/diffusionMath"; +import { radialPowerSpectrum } from "./lib/spectrum"; +import { SPRITES, SPRITE_N, SPRITE_SIZE } from "./lib/sprites"; +import { useSimLoop } from "./lib/useSimLoop"; + +// The forward process, in three simultaneous views of the same slider. +// +// The noise realization ε is drawn ONCE and held fixed as t moves. That is +// the whole reason the left panel dissolves smoothly instead of reshuffling: +// the reader is watching one fixed noise vector being mixed in at increasing +// strength, which is what x_t = √ᾱ·x₀ + √(1-ᾱ)·ε actually says. Reset draws +// a fresh ε. +// +// The right panel is the load-bearing one. Signal power scales by ᾱ, the +// noise floor is n²(1-ᾱ) flat across all frequencies, and where they cross +// is the frequency above which detail is gone. That crossing sweeps +// leftward as t grows — fine detail dies first, silhouette dies last — and +// it is why denoising comes back coarse-to-fine. + +const T = 1000; +const SPEEDS = [ + { label: "1×", value: 90 }, + { label: "3×", value: 260 }, + { label: "8×", value: 700 }, +]; +const HIST_BINS = 46; +const HIST_LO = -3.2; +const HIST_HI = 3.2; + +export default function ForwardNoise() { + const [t, setT] = useState(0); + const [spriteIdx, setSpriteIdx] = useState(0); + const [seed, setSeed] = useState(1); + const [speed, setSpeed] = useState(SPEEDS[0].value); + const dir = useRef(1); + + const x0 = SPRITES[spriteIdx].data; + + // One fixed noise vector per (seed) — see note above. + const eps = useMemo(() => { + const rng = makeRng(seed * 7919 + 13); + const e = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) e[i] = gaussian(rng); + return e; + }, [seed]); + + const ab = alphaBar(t / T); + + const xt = useMemo(() => { + const sa = Math.sqrt(ab); + const sn = Math.sqrt(1 - ab); + const out = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) out[i] = sa * x0[i] + sn * eps[i]; + return out; + }, [ab, x0, eps]); + + const onTick = useCallback((ticks: number) => { + setT((prev) => { + let next = prev + dir.current * ticks; + if (next >= T) { + next = T; + dir.current = -1; + } else if (next <= 0) { + next = 0; + dir.current = 1; + } + return next; + }); + }, []); + const { playing, setPlaying, toggle } = useSimLoop(onTick, speed); + + const reset = useCallback(() => { + setPlaying(false); + dir.current = 1; + setT(0); + setSeed((s) => s + 1); + }, [setPlaying]); + + /* ---------------- histogram of pixel values ---------------- */ + + const hist = useMemo(() => { + const bins = new Float64Array(HIST_BINS); + const w = (HIST_HI - HIST_LO) / HIST_BINS; + for (let i = 0; i < xt.length; i++) { + const b = Math.floor((xt[i] - HIST_LO) / w); + if (b >= 0 && b < HIST_BINS) bins[b] += 1; + } + // Normalize to a density so the N(0,1) overlay is directly comparable. + let peak = 0; + for (let b = 0; b < HIST_BINS; b++) { + bins[b] /= xt.length * w; + if (bins[b] > peak) peak = bins[b]; + } + return { bins, w, peak: Math.max(peak, 0.42) }; + }, [xt]); + + /* ---------------- power spectra ---------------- */ + + const cleanSpec = useMemo(() => radialPowerSpectrum(x0, SPRITE_SIZE), [x0]); + const noisySpec = useMemo(() => radialPowerSpectrum(xt, SPRITE_SIZE), [xt]); + + // Flat expected noise power per DFT coefficient: n²·σ² with σ² = 1-ᾱ. + const noiseFloor = SPRITE_N * (1 - ab); + + // Highest frequency whose surviving signal still beats the noise floor. + const crossK = useMemo(() => { + let last = 0; + for (let i = 0; i < cleanSpec.k.length; i++) { + if (ab * cleanSpec.power[i] > noiseFloor) last = cleanSpec.k[i]; + } + return last; + }, [cleanSpec, ab, noiseFloor]); + + const detailPct = Math.round((crossK / (SPRITE_SIZE / 2)) * 100); + + return ( + onTick(40)} + speed={speed} + speeds={SPEEDS} + onSpeed={setSpeed} + readouts={[ + { label: "t", value: `${t} / ${T}` }, + { label: "ᾱ", value: ab.toFixed(3), color: "var(--viz-policy)" }, + { label: "SNR", value: `${(10 * Math.log10(snr(ab))).toFixed(1)} dB` }, + { + label: "detail surviving", + value: `${detailPct}%`, + color: detailPct > 0 ? "var(--viz-reward)" : "var(--viz-danger)", + }, + ]} + > +
+
+
the image
+
+ + +
+
+ {SPRITES.map((s, i) => ( + + ))} +
+
+ +
+
pixel histogram
+ +
+ +
+
power spectrum
+ +
+ + + signal left (ᾱ·clean) + + + + noise floor + + + + measured xₜ + + + + clean x₀ + +
+
+
+ + +
+ ); +} + +/* ------------------------------------------------------------------ */ + +function Histogram({ + hist, +}: { + hist: { bins: Float64Array; w: number; peak: number }; +}) { + const W = 300; + const H = 168; + const PAD = { l: 6, r: 6, t: 8, b: 18 }; + const plotW = W - PAD.l - PAD.r; + const plotH = H - PAD.t - PAD.b; + + const xOf = (v: number) => PAD.l + ((v - HIST_LO) / (HIST_HI - HIST_LO)) * plotW; + const yOf = (d: number) => PAD.t + plotH - (d / hist.peak) * plotH; + + const bars = []; + for (let b = 0; b < HIST_BINS; b++) { + const v0 = HIST_LO + b * hist.w; + const h = plotH - (yOf(hist.bins[b]) - PAD.t); + if (h <= 0) continue; + bars.push( + , + ); + } + + // The target: unit Gaussian, the distribution the forward process is + // walking toward regardless of which image it started from. + const curve: string[] = []; + for (let i = 0; i <= 80; i++) { + const v = HIST_LO + (i / 80) * (HIST_HI - HIST_LO); + const d = Math.exp(-(v * v) / 2) / Math.sqrt(2 * Math.PI); + curve.push(`${i === 0 ? "M" : "L"}${xOf(v).toFixed(1)},${yOf(d).toFixed(1)}`); + } + + return ( + + Pixel-value histogram versus the unit Gaussian + {bars} + + + {[-3, -2, -1, 0, 1, 2, 3].map((v) => ( + + {v} + + ))} + + 𝒩(0,1) + + + ); +} + +/* ------------------------------------------------------------------ */ + +function Spectrum({ + cleanK, + cleanP, + noisyP, + ab, + noiseFloor, + crossK, +}: { + cleanK: number[]; + cleanP: number[]; + noisyP: number[]; + ab: number; + noiseFloor: number; + crossK: number; +}) { + const W = 300; + const H = 168; + const PAD = { l: 30, r: 8, t: 8, b: 22 }; + const plotW = W - PAD.l - PAD.r; + const plotH = H - PAD.t - PAD.b; + + const FLOOR = 1e-2; + const lg = (p: number) => Math.log10(Math.max(p, FLOOR)); + + const yLo = -2; + const yHi = useMemo(() => { + let m = 1; + for (const p of cleanP) m = Math.max(m, lg(p)); + return Math.ceil(m) + 0.2; + }, [cleanP]); + + const kMax = cleanK[cleanK.length - 1] ?? 12; + const xOf = (k: number) => PAD.l + (Math.log10(k) / Math.log10(kMax)) * plotW; + const yOf = (p: number) => + PAD.t + plotH - ((lg(p) - yLo) / (yHi - yLo)) * plotH; + + const line = (vals: number[], scale: number) => + cleanK + .map( + (k, i) => + `${i === 0 ? "M" : "L"}${xOf(k).toFixed(1)},${yOf(vals[i] * scale).toFixed(1)}`, + ) + .join(" "); + + return ( + + Power spectrum: surviving signal against the flat noise floor + + {[yLo, 0, 2, 4].filter((v) => v >= yLo && v <= yHi).map((v) => ( + + + + 10{sup(v)} + + + ))} + + {/* Where surviving signal falls under the noise floor: everything to + the right of this line is detail that no longer exists. */} + {crossK > 0 && crossK < kMax && ( + <> + + + + )} + + {/* Clean spectrum, for reference: the 1/f-ish fall-off of real structure. */} + + {/* What survives at this noise level. */} + + {/* What is actually measured in x_t. */} + + {/* Flat noise floor. */} + + + + + coarse + + + fine → + + + ); +} + +function sup(v: number): string { + const map: Record = { + "-": "⁻", + "0": "⁰", + "1": "¹", + "2": "²", + "3": "³", + "4": "⁴", + "5": "⁵", + "6": "⁶", + }; + return String(v) + .split("") + .map((c) => map[c] ?? c) + .join(""); +} diff --git a/src/components/visualizations/GANDuel.tsx b/src/components/visualizations/GANDuel.tsx new file mode 100644 index 0000000..af1f8c4 --- /dev/null +++ b/src/components/visualizations/GANDuel.tsx @@ -0,0 +1,374 @@ +import { useCallback, useEffect, useRef, useState } from "react"; +import SimShell from "./lib/SimShell"; +import Sparkline from "./lib/Sparkline"; +import { gaussian, makeRng } from "./lib/diffusionMath"; +import { MLP, sigmoid } from "./lib/tinynn"; +import { useSimLoop } from "./lib/useSimLoop"; + +// A real GAN, trained in the browser: two 2→16→16→k MLPs, hand-written +// backprop, Adam(β₁=0.5), non-saturating loss by default. Nothing here is +// scripted or animated — every dot is the current generator's actual output. +// +// The measured behaviour, over 8 seeds at the default config, is the reason +// this widget exists and the reason its Reset button matters: +// +// 1 D step, lr 2e-3 → effective modes 3–8, mass off the ring 11%–79% +// 4 D steps, lr 8e-3 → effective modes 6–8, mass off the ring 9%–15% +// +// So: most runs work, some runs are terrible, the hyperparameters are +// identical, and the loss curves look about the same either way. That spread +// IS the pathology — not a scripted collapse. Note also that a *stronger* +// discriminator tightens the spread rather than causing collapse; with the +// non-saturating loss, a weak D is what starves the generator of gradient. + +const MODES = 8; +const RING = 1.0; +const SD = 0.06; +const BATCH = 64; +const N_SHOW = 400; +const SPEEDS = [ + { label: "1×", value: 30 }, + { label: "5×", value: 150 }, + { label: "15×", value: 450 }, +]; + +function modeCenter(m: number): [number, number] { + const a = (m / MODES) * 2 * Math.PI; + return [RING * Math.cos(a), RING * Math.sin(a)]; +} + +interface Nets { + G: MLP; + D: MLP; + rng: () => number; + iter: number; +} + +export default function GANDuel() { + const [dSteps, setDSteps] = useState(1); + const [saturating, setSaturating] = useState(false); + const [speed, setSpeed] = useState(SPEEDS[1].value); + const [seed, setSeed] = useState(1); + const [, forceRender] = useState(0); + + const netsRef = useRef(null); + const fakeRef = useRef([]); + const realRef = useRef([]); + const dLoss = useRef([]); + const gLoss = useRef([]); + const canvasRef = useRef(null); + + const realSample = (rng: () => number) => { + const m = Math.floor(rng() * MODES); + const [cx, cy] = modeCenter(m); + return Float64Array.from([cx + gaussian(rng) * SD, cy + gaussian(rng) * SD]); + }; + + const build = useCallback((s: number): Nets => { + const rng = makeRng(s * 104729 + 17); + const nets: Nets = { + G: new MLP([2, 16, 16, 2], ["tanh", "tanh", "none"], makeRng(s * 3 + 1)), + D: new MLP([2, 16, 16, 1], ["lrelu", "lrelu", "none"], makeRng(s * 7 + 2)), + rng, + iter: 0, + }; + // Populate both clouds before any training, so the opening frame shows + // the target distribution and the untrained generator's blob rather + // than an empty ring of circles. + const noise = () => { + const z = new Float64Array(2); + z[0] = gaussian(rng); + z[1] = gaussian(rng); + return z; + }; + realRef.current = Array.from({ length: N_SHOW }, () => realSample(rng)); + fakeRef.current = nets.G.forward(Array.from({ length: N_SHOW }, noise)).map( + (v) => v.slice(), + ); + return nets; + }, []); + + if (!netsRef.current) netsRef.current = build(1); + + const train = useCallback( + (iters: number) => { + const nets = netsRef.current; + if (!nets) return; + const { G, D, rng } = nets; + const noise = () => { + const z = new Float64Array(2); + z[0] = gaussian(rng); + z[1] = gaussian(rng); + return z; + }; + + for (let it = 0; it < iters; it++) { + let dl = 0; + for (let d = 0; d < dSteps; d++) { + const real = Array.from({ length: BATCH }, () => realSample(rng)); + const fake = G.forward(Array.from({ length: BATCH }, noise)).map((v) => + v.slice(), + ); + D.zeroGrad(); + const lr_ = D.forward(real); + D.backward(lr_.map((o) => Float64Array.from([sigmoid(o[0]) - 1]))); + const lf_ = D.forward(fake); + D.backward(lf_.map((o) => Float64Array.from([sigmoid(o[0])]))); + D.step(dSteps > 1 ? 8e-3 : 2e-3, 2 * BATCH); + if (d === dSteps - 1) { + for (let i = 0; i < BATCH; i++) { + dl -= Math.log(Math.max(sigmoid(lr_[i][0]), 1e-9)) / (2 * BATCH); + dl -= Math.log(Math.max(1 - sigmoid(lf_[i][0]), 1e-9)) / (2 * BATCH); + } + } + } + + const z = Array.from({ length: BATCH }, noise); + const fake = G.forward(z); + const lo = D.forward(fake.map((v) => v.slice())); + D.zeroGrad(); + // non-saturating −log D(G(z)) vs the original minimax log(1−D(G(z))) + const gd = D.backward( + lo.map((o) => + Float64Array.from([ + saturating ? -sigmoid(o[0]) : sigmoid(o[0]) - 1, + ]), + ), + ); + G.zeroGrad(); + G.backward(gd); + G.step(2e-3, BATCH); + + let gl = 0; + for (let i = 0; i < BATCH; i++) { + gl -= Math.log(Math.max(sigmoid(lo[i][0]), 1e-9)) / BATCH; + } + nets.iter++; + if (nets.iter % 10 === 0) { + dLoss.current.push(dl); + gLoss.current.push(gl); + if (dLoss.current.length > 400) { + dLoss.current.shift(); + gLoss.current.shift(); + } + } + } + + // Refresh the displayed clouds. + fakeRef.current = G.forward(Array.from({ length: N_SHOW }, noise)).map((v) => + v.slice(), + ); + realRef.current = Array.from({ length: N_SHOW }, () => realSample(rng)); + }, + [dSteps, saturating], + ); + + const onTick = useCallback( + (ticks: number) => { + train(Math.min(ticks, 40)); + forceRender((v) => v + 1); + }, + [train], + ); + const { playing, setPlaying, toggle } = useSimLoop(onTick, speed); + + const reset = useCallback( + (nextSeed?: number) => { + setPlaying(false); + const s = nextSeed ?? seed + 1; + setSeed(s); + netsRef.current = build(s); + fakeRef.current = []; + realRef.current = []; + dLoss.current = []; + gLoss.current = []; + forceRender((v) => v + 1); + }, + [seed, build, setPlaying], + ); + + /* ---------------- what the generator actually covers ---------------- */ + + const stats = (() => { + const pts = fakeRef.current; + if (pts.length === 0) return { eff: 0, off: 0, maxShare: 0 }; + const cnt = new Array(MODES).fill(0); + let off = 0; + for (const s of pts) { + let best = -1; + let bd = Number.POSITIVE_INFINITY; + for (let m = 0; m < MODES; m++) { + const [cx, cy] = modeCenter(m); + const d = Math.hypot(s[0] - cx, s[1] - cy); + if (d < bd) { + bd = d; + best = m; + } + } + if (bd < 0.22) cnt[best]++; + else off++; + } + const share = cnt.map((c) => c / pts.length); + return { + eff: share.filter((p) => p > 0.02).length, + off: off / pts.length, + maxShare: Math.max(...share), + }; + })(); + + /* ---------------- drawing ---------------- */ + + useEffect(() => { + const canvas = canvasRef.current; + if (!canvas) return; + const ctx = canvas.getContext("2d"); + if (!ctx) return; + const dpr = window.devicePixelRatio || 1; + const css = 250; + const px = Math.round(css * dpr); + if (canvas.width !== px) { + canvas.width = px; + canvas.height = px; + } + ctx.clearRect(0, 0, px, px); + const mid = px / 2; + const sc = mid / 1.45; + const X = (v: number) => mid + v * sc; + const Y = (v: number) => mid - v * sc; + + // Target modes. + ctx.strokeStyle = "rgba(140,130,110,0.5)"; + ctx.lineWidth = 1 * dpr; + for (let m = 0; m < MODES; m++) { + const [cx, cy] = modeCenter(m); + ctx.beginPath(); + ctx.arc(X(cx), Y(cy), 0.22 * sc, 0, 2 * Math.PI); + ctx.stroke(); + } + const dot = (p: Float64Array, color: string, r: number) => { + ctx.fillStyle = color; + ctx.fillRect(X(p[0]) - r, Y(p[1]) - r, r * 2, r * 2); + }; + for (const p of realRef.current) dot(p, "rgba(148,163,184,0.75)", 1.1 * dpr); + for (const p of fakeRef.current) dot(p, "#3b82f6", 1.3 * dpr); + }); + + const iter = netsRef.current?.iter ?? 0; + + return ( + reset()} + onStep={() => onTick(20)} + speed={speed} + speeds={SPEEDS} + onSpeed={setSpeed} + readouts={[ + { label: "seed", value: `${seed}` }, + { label: "iter", value: `${iter}` }, + { + label: "modes covered", + value: `${stats.eff} / ${MODES}`, + color: stats.eff >= 7 ? "var(--viz-reward)" : "var(--viz-danger)", + }, + { + label: "mass off the data", + value: `${(stats.off * 100).toFixed(0)}%`, + color: stats.off > 0.3 ? "var(--viz-danger)" : "var(--viz-reward)", + }, + ]} + > +
+
+ +
+ + + real data + + + + generated + +
+
+ +
+
discriminator loss
+ v.toFixed(2)} + /> +
+ generator loss +
+ v.toFixed(2)} + /> +

+ Neither curve tells you whether this run is good. Compare them + against the mode readout above as you reset. +

+
+
+ +
+
+ discriminator strength + + {[ + { label: "weak (1 step)", value: 1 }, + { label: "strong (4 steps)", value: 4 }, + ].map((o) => ( + + ))} + +
+
+ generator loss + + {[ + { label: "non-saturating", value: false }, + { label: "original minimax", value: true }, + ].map((o) => ( + + ))} + +
+
+
+ ); +} diff --git a/src/components/visualizations/GuidanceDial.tsx b/src/components/visualizations/GuidanceDial.tsx new file mode 100644 index 0000000..7ad1ecb --- /dev/null +++ b/src/components/visualizations/GuidanceDial.tsx @@ -0,0 +1,290 @@ +import { useCallback, useMemo, useRef, useState } from "react"; +import PixelCanvas from "./lib/PixelCanvas"; +import SimShell from "./lib/SimShell"; +import { + alphaBar, + epsFromX0, + expectedX0, + gaussian, + makeRng, + posteriorWeights, + reverseStep, + rmse, +} from "./lib/diffusionMath"; +import { makeLabeledDataset, meanImage, SHAPE_NAMES, SPRITE_N, SPRITE_SIZE } from "./lib/sprites"; +import { useSimLoop } from "./lib/useSimLoop"; + +// Real classifier-free guidance, with both scores computed exactly. +// +// ε_guided = ε_uncond + w·(ε_cond − ε_uncond) +// +// w = 0 ignores the prompt, w = 1 is ordinary conditional sampling, and w > 1 +// extrapolates *past* the conditional prediction, away from the +// unconditional one. +// +// Measured over 32 seeds on a fixed prompt: prompt accuracy goes 25% → 100% +// as soon as w reaches 0.5, and within-class diversity then decays — 13 of +// the 16 possible images at w = 1, 8 at w = 8 and 15, with the most frequent +// single image rising from 16% to 25% of draws. Distance-to-nearest-real +// stays 0.003 throughout: no saturation artifacts, because the score being +// extrapolated here is exact. In real systems it is a learned approximation, +// and extrapolating an approximation is where the burned-out look comes from. + +const N_DATA = 128; +const STEPS = 40; +const SPEEDS = [ + { label: "1×", value: 8 }, + { label: "4×", value: 32 }, + { label: "20×", value: 160 }, +]; + +export default function GuidanceDial() { + const [target, setTarget] = useState(0); + const [w, setW] = useState(1); + const [speed, setSpeed] = useState(SPEEDS[1].value); + const [seed, setSeed] = useState(1); + const [, forceRender] = useState(0); + + const { data, labels } = useMemo(() => makeLabeledDataset(N_DATA, 3), []); + const classSets = useMemo(() => { + const out: Float32Array[][] = Array.from({ length: SHAPE_NAMES.length }, () => []); + labels.forEach((l, i) => out[l].push(data[i])); + return out; + }, [data, labels]); + + const xRef = useRef(new Float32Array(SPRITE_N)); + const x0Ref = useRef(new Float32Array(SPRITE_N)); + const stepRef = useRef(0); + const rngRef = useRef<() => number>(makeRng(1)); + const gallery = useRef<{ img: Float32Array; label: number }[]>([]); + + const datasetMean = useMemo(() => meanImage(data), [data]); + + const begin = useCallback( + (s: number) => { + const rng = makeRng(s * 911 + 5); + rngRef.current = rng; + const x = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) x[i] = gaussian(rng); + xRef.current = x; + // Before the first step the model has seen only noise, so its best + // guess is the dataset average — show that rather than an empty buffer. + x0Ref.current = datasetMean; + stepRef.current = 0; + }, + [datasetMean], + ); + + if (stepRef.current === 0 && xRef.current.every((v) => v === 0)) begin(1); + + const nearestLabel = useCallback( + (x: Float32Array) => { + let bi = 0; + let bd = Number.POSITIVE_INFINITY; + data.forEach((d, i) => { + const r = rmse(x, d); + if (r < bd) { + bd = r; + bi = i; + } + }); + return { label: labels[bi], dist: bd }; + }, + [data, labels], + ); + + const advance = useCallback(() => { + if (stepRef.current >= STEPS) return; + const k = STEPS - stepRef.current; + const abT = alphaBar(k / STEPS); + const abPrev = alphaBar((k - 1) / STEPS); + const x = xRef.current; + const sub = classSets[target]; + + const wu = posteriorWeights(x, data, abT); + const x0u = expectedX0(data, wu); + const eu = epsFromX0(x, x0u, abT); + + const wc = posteriorWeights(x, sub, abT); + const x0c = expectedX0(sub, wc); + const ec = epsFromX0(x, x0c, abT); + + const eg = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) eg[i] = eu[i] + w * (ec[i] - eu[i]); + + const sa = Math.sqrt(abT); + const sn = Math.sqrt(1 - abT); + const x0g = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) x0g[i] = (x[i] - sn * eg[i]) / sa; + + xRef.current = reverseStep(x, x0g, eg, abT, abPrev, 1, rngRef.current); + x0Ref.current = x0g; + stepRef.current++; + + if (stepRef.current >= STEPS) { + const got = nearestLabel(xRef.current); + gallery.current = [ + { img: xRef.current.slice(), label: got.label }, + ...gallery.current, + ].slice(0, 8); + } + }, [classSets, data, target, w, nearestLabel]); + + const onTick = useCallback( + (ticks: number) => { + for (let i = 0; i < ticks; i++) { + if (stepRef.current >= STEPS) { + // Roll straight into a fresh sample so the gallery fills and the + // diversity question can actually be answered by watching. + setSeed((s) => s + 1); + begin(seed + 1 + i); + } + advance(); + } + forceRender((v) => v + 1); + }, + [advance, begin, seed], + ); + const { playing, setPlaying, toggle } = useSimLoop(onTick, speed); + + const reset = useCallback(() => { + setPlaying(false); + gallery.current = []; + const s = seed + 1; + setSeed(s); + begin(s); + forceRender((v) => v + 1); + }, [seed, begin, setPlaying]); + + const done = stepRef.current >= STEPS; + const got = done ? nearestLabel(xRef.current) : null; + const onPrompt = gallery.current.filter((g) => g.label === target).length; + const distinct = new Set(gallery.current.map((g) => g.img.join(","))).size; + + return ( + onTick(1)} + speed={speed} + speeds={SPEEDS} + onSpeed={setSpeed} + readouts={[ + { label: "prompt", value: SHAPE_NAMES[target], color: "var(--viz-policy)" }, + { label: "step", value: `${Math.min(stepRef.current, STEPS)} / ${STEPS}` }, + { label: "guidance w", value: w.toFixed(1), color: "var(--viz-kl)" }, + { + label: "on-prompt", + value: gallery.current.length + ? `${onPrompt}/${gallery.current.length}` + : "—", + color: + gallery.current.length && onPrompt === gallery.current.length + ? "var(--viz-reward)" + : "var(--viz-danger)", + }, + { + label: "distinct", + value: gallery.current.length ? `${distinct}/${gallery.current.length}` : "—", + color: "var(--viz-value)", + }, + ]} + > +
+
+
sampling
+
+ + +
+

+ {done && got + ? got.label === target + ? `Landed on "${SHAPE_NAMES[got.label]}" — the prompt.` + : `Landed on "${SHAPE_NAMES[got.label]}", not the prompt.` + : `Requested "${SHAPE_NAMES[target]}".`} +

+
+ +
+
recent samples
+
+ {gallery.current.map((g, i) => ( + + ))} +
+ {gallery.current.length === 0 && ( +

+ Press Run — finished samples collect here so you can judge + whether guidance is costing you variety. +

+ )} +
+
+ +
+ +
+ prompt + + {SHAPE_NAMES.map((n, i) => ( + + ))} + +
+
+
+ ); +} diff --git a/src/components/visualizations/LatentCompress.tsx b/src/components/visualizations/LatentCompress.tsx new file mode 100644 index 0000000..cce8638 --- /dev/null +++ b/src/components/visualizations/LatentCompress.tsx @@ -0,0 +1,195 @@ +import { useMemo, useState } from "react"; +import PixelCanvas from "./lib/PixelCanvas"; +import SimShell from "./lib/SimShell"; +import { rmse } from "./lib/diffusionMath"; +import { decode, encode, fitPCA } from "./lib/pca"; +import { makeDataset, SPRITE_N, SPRITE_SIZE } from "./lib/sprites"; + +// How much of a 576-pixel image is actually load-bearing. +// +// The encoder/decoder here is PCA, fitted on the same 256-image dataset the +// samplers use. A linear autoencoder trained to convergence learns exactly +// this subspace, so it is a real bottleneck rather than a mock-up — it just +// understates a real VAE, which is non-linear and perceptually trained. +// +// The point stands either way: the reconstruction is visually indistinguishable +// long before k reaches 576, and every dimension you drop is a dimension the +// diffusion model no longer has to denoise, at every one of its steps. + +const DATA = makeDataset(256, 3); +const MAX_K = 64; +const K_CHOICES = [1, 2, 4, 8, 16, 32, 64]; + +export default function LatentCompress() { + const [k, setK] = useState(8); + const [idx, setIdx] = useState(5); + + const pca = useMemo(() => fitPCA(DATA, MAX_K), []); + + const source = DATA[idx]; + const recon = useMemo( + () => decode(pca, encode(pca, source, k)), + [pca, source, k], + ); + const residual = useMemo(() => { + const r = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) r[i] = source[i] - recon[i]; + return r; + }, [source, recon]); + + const explained = useMemo(() => { + let acc = 0; + for (let i = 0; i < k; i++) acc += pca.eigenvalues[i] ?? 0; + return acc / pca.totalVariance; + }, [pca, k]); + + const err = rmse(source, recon); + const compression = SPRITE_N / k; + + return ( + {}} + onReset={() => { + setK(8); + setIdx(5); + }} + readouts={[ + { label: "latent dims", value: `${k} / ${SPRITE_N}`, color: "var(--viz-policy)" }, + { + label: "variance kept", + value: `${(explained * 100).toFixed(1)}%`, + color: "var(--viz-reward)", + }, + { label: "reconstruction error", value: err.toFixed(3) }, + { + label: "work per denoise step", + value: `÷${compression.toFixed(0)}`, + color: "var(--viz-value)", + }, + ]} + > +
+
+
encode → decode
+
+ + + +
+
+ {[0, 1, 2, 3, 4, 5, 6, 7].map((i) => ( + + ))} +
+
+ +
+
variance captured by the first k components
+ +
+
+ +
+ + latent dimensions k = {k} + + + {K_CHOICES.map((n) => ( + + ))} + +
+
+ ); +} + +function Scree({ eigen, total, k }: { eigen: number[]; total: number; k: number }) { + const W = 300; + const H = 160; + const PAD = { l: 30, r: 8, t: 10, b: 20 }; + const pw = W - PAD.l - PAD.r; + const ph = H - PAD.t - PAD.b; + const n = eigen.length; + + const cum: number[] = []; + let acc = 0; + for (let i = 0; i < n; i++) { + acc += eigen[i]; + cum.push(acc / total); + } + + const X = (i: number) => PAD.l + (i / (n - 1)) * pw; + const Y = (v: number) => PAD.t + ph - v * ph; + + return ( + + Cumulative variance explained + {[0.25, 0.5, 0.75, 0.9, 1].map((g) => ( + + + + {Math.round(g * 100)}% + + + ))} + `${i === 0 ? "M" : "L"}${X(i).toFixed(1)},${Y(v).toFixed(1)}`).join(" ")} + fill="none" + stroke="var(--viz-reward)" + strokeWidth={2} + /> + + + + 1 component + + + {n} + + + ); +} diff --git a/src/components/visualizations/ManifoldSlice.tsx b/src/components/visualizations/ManifoldSlice.tsx new file mode 100644 index 0000000..2c182ec --- /dev/null +++ b/src/components/visualizations/ManifoldSlice.tsx @@ -0,0 +1,222 @@ +import { useMemo, useState } from "react"; +import PixelCanvas from "./lib/PixelCanvas"; +import SimShell from "./lib/SimShell"; +import { gaussian, makeRng, rmse } from "./lib/diffusionMath"; +import { makeDataset, poseSprite, SHAPE_NAMES, SPRITE_N, SPRITE_SIZE } from "./lib/sprites"; + +// Two routes between the same pair of images. +// +// The top strip walks a straight line through the 576 pixel values. The +// bottom strip walks a straight line through the four pose numbers the images +// were actually drawn from — a genuine latent space, not a stand-in, because +// these sprites really are generated from those parameters. +// +// Both paths start and end at identical images. Only one stays on the set of +// things that are images. + +const STEPS = 7; +const REF = makeDataset(256, 3); + +const POSE_A = { dx: -0.16, dy: 0.1, scale: 0.85, rot: -0.3 }; +const POSE_B = { dx: 0.18, dy: -0.12, scale: 1.12, rot: 0.35 }; + +function midtone(x: Float32Array): number { + let n = 0; + for (const v of x) if (Math.abs(v) < 0.5) n++; + return n / x.length; +} + +function nearestReal(x: Float32Array): number { + let best = Number.POSITIVE_INFINITY; + for (const d of REF) best = Math.min(best, rmse(x, d)); + return best; +} + +export default function ManifoldSlice() { + const [shapeA, setShapeA] = useState(0); + const [shapeB, setShapeB] = useState(3); + const [lambda, setLambda] = useState(0.5); + + const endA = useMemo(() => poseSprite(shapeA, POSE_A), [shapeA]); + const endB = useMemo(() => poseSprite(shapeB, POSE_B), [shapeB]); + + // Straight line through pixel space. + const pixelPath = useMemo(() => { + const out: Float32Array[] = []; + for (let i = 0; i < STEPS; i++) { + const l = i / (STEPS - 1); + const f = new Float32Array(SPRITE_N); + for (let k = 0; k < SPRITE_N; k++) f[k] = (1 - l) * endA[k] + l * endB[k]; + out.push(f); + } + return out; + }, [endA, endB]); + + // Straight line through the pose parameters, re-rendered at every step. + const posePath = useMemo(() => { + const out: Float32Array[] = []; + for (let i = 0; i < STEPS; i++) { + const l = i / (STEPS - 1); + // Below the halfway point the shape identity is A, above it is B — + // shape is categorical, so it is the one coordinate that cannot be + // blended without leaving the set of real images. + const shape = l < 0.5 ? shapeA : shapeB; + out.push( + poseSprite(shape, { + dx: (1 - l) * POSE_A.dx + l * POSE_B.dx, + dy: (1 - l) * POSE_A.dy + l * POSE_B.dy, + scale: (1 - l) * POSE_A.scale + l * POSE_B.scale, + rot: (1 - l) * POSE_A.rot + l * POSE_B.rot, + }), + ); + } + return out; + }, [shapeA, shapeB]); + + const li = Math.round(lambda * (STEPS - 1)); + const pixMid = pixelPath[li]; + const poseMid = posePath[li]; + + // What a point drawn uniformly from pixel space looks like. + const randomImages = useMemo(() => { + const rng = makeRng(7); + return [0, 1, 2].map(() => { + const f = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) f[i] = Math.tanh(gaussian(rng)); + return f; + }); + }, []); + + return ( + {}} + onReset={() => { + setShapeA(0); + setShapeB(3); + setLambda(0.5); + }} + readouts={[ + { label: "λ", value: lambda.toFixed(2) }, + { + label: "pixel path — dist. to nearest real", + value: nearestReal(pixMid).toFixed(3), + color: "var(--viz-danger)", + }, + { + label: "pose path — dist. to nearest real", + value: nearestReal(poseMid).toFixed(3), + color: "var(--viz-reward)", + }, + ]} + > +
straight line through the 576 pixels
+
+ {pixelPath.map((f, i) => ( + 0.25 ? "ghost" : ""}`} + /> + ))} +
+ +
+ straight line through the four pose parameters +
+
+ {posePath.map((f, i) => ( + + ))} +
+ + + +
+
+ start shape + + {SHAPE_NAMES.map((n, i) => ( + + ))} + +
+
+ end shape + + {SHAPE_NAMES.map((n, i) => ( + + ))} + +
+
+ +
+ for scale: three points drawn uniformly at random from pixel space +
+
+ {randomImages.map((f, i) => ( + + ))} + + Sample pixel space at random for the rest of your life and you will + never once hit an image. + +
+
+ ); +} diff --git a/src/components/visualizations/MixingReversibility.tsx b/src/components/visualizations/MixingReversibility.tsx new file mode 100644 index 0000000..28992c4 --- /dev/null +++ b/src/components/visualizations/MixingReversibility.tsx @@ -0,0 +1,319 @@ +import { useCallback, useEffect, useRef, useState } from "react"; +import SimShell from "./lib/SimShell"; +import { gaussian, makeRng } from "./lib/diffusionMath"; +import { useSimLoop } from "./lib/useSimLoop"; + +// Ink in water and dye in corn syrup, as one simulation with one knob. +// +// Both fluids get the SAME advection: Taylor–Couette flow between two +// cylinders, whose angular velocity profile is a closed form +// +// ω(r) = Ω · (r_in² / (r_out² - r_in²)) · (r_out²/r² - 1) +// +// so the shear is exact rather than eyeballed, and cranking backwards undoes +// it exactly. The only difference between the two media is the molecular +// jitter added each tick, scaled √dt like real Brownian motion. +// +// That single knob is the whole argument of the page. At jitter = 0 the +// dye smears into spirals, looks thoroughly mixed, and then reassembles +// perfectly when you crank back — nothing was destroyed, only rearranged. +// Turn the jitter up and the advection still reverses but the jitter does +// not, so the blobs never come back. "Looks mixed" and "is mixed" are +// different claims, and only one of them is reversible. + +const R_IN = 0.26; +const R_OUT = 0.97; +const N_PER_BLOB = 340; +const BLOBS = [ + { color: "#3b82f6", angle: -Math.PI / 2 }, + { color: "#f59e0b", angle: -Math.PI / 2 + (2 * Math.PI) / 3 }, + { color: "#22c55e", angle: -Math.PI / 2 + (4 * Math.PI) / 3 }, +]; +const TURN_STEP = 0.014; +const MAX_TURNS = 4; +const SPEEDS = [ + { label: "1×", value: 60 }, + { label: "3×", value: 180 }, + { label: "8×", value: 480 }, +]; + +interface Particles { + x: Float32Array; + y: Float32Array; + x0: Float32Array; + y0: Float32Array; + blob: Uint8Array; +} + +function seedParticles(): Particles { + const n = BLOBS.length * N_PER_BLOB; + const p: Particles = { + x: new Float32Array(n), + y: new Float32Array(n), + x0: new Float32Array(n), + y0: new Float32Array(n), + blob: new Uint8Array(n), + }; + const rng = makeRng(20260728); + let i = 0; + BLOBS.forEach((b, bi) => { + // A compact radial blob partway out from the inner cylinder. + const cr = (R_IN + R_OUT) / 2; + for (let k = 0; k < N_PER_BLOB; k++) { + const rr = cr + gaussian(rng) * 0.075; + const aa = b.angle + gaussian(rng) * 0.11; + const r = Math.min(R_OUT - 0.01, Math.max(R_IN + 0.01, rr)); + p.x[i] = r * Math.cos(aa); + p.y[i] = r * Math.sin(aa); + p.x0[i] = p.x[i]; + p.y0[i] = p.y[i]; + p.blob[i] = bi; + i++; + } + }); + return p; +} + +/** Exact Couette angular velocity, normalized to 1 at the inner wall. */ +function omega(r: number): number { + const rr = Math.min(R_OUT, Math.max(R_IN, r)); + const k = (R_IN * R_IN) / (R_OUT * R_OUT - R_IN * R_IN); + return k * ((R_OUT * R_OUT) / (rr * rr) - 1); +} + +export default function MixingReversibility() { + const [jitter, setJitter] = useState(0); + const [speed, setSpeed] = useState(SPEEDS[0].value); + const [turns, setTurns] = useState(0); + const [phase, setPhase] = useState<"forward" | "back">("forward"); + const [seed, setSeed] = useState(1); + + const partsRef = useRef(seedParticles()); + const rngRef = useRef<() => number>(makeRng(99)); + const canvasRef = useRef(null); + const [, forceRender] = useState(0); + + const crank = useCallback( + (dTurns: number) => { + const p = partsRef.current; + const rng = rngRef.current; + // Brownian displacement grows as √time, so the jitter per tick scales + // with the square root of the step — not linearly with it. + const sigma = jitter * Math.sqrt(Math.abs(dTurns)) * 0.055; + for (let i = 0; i < p.x.length; i++) { + let x = p.x[i]; + let y = p.y[i]; + const r = Math.hypot(x, y); + const a = omega(r) * dTurns * 2 * Math.PI; + const c = Math.cos(a); + const s = Math.sin(a); + const nx = x * c - y * s; + const ny = x * s + y * c; + x = nx; + y = ny; + if (sigma > 0) { + x += sigma * gaussian(rng); + y += sigma * gaussian(rng); + // Keep the dye inside the vessel. + const rr = Math.hypot(x, y); + if (rr > R_OUT || rr < R_IN) { + const clamped = Math.min(R_OUT - 0.004, Math.max(R_IN + 0.004, rr)); + x = (x / rr) * clamped; + y = (y / rr) * clamped; + } + } + p.x[i] = x; + p.y[i] = y; + } + }, + [jitter], + ); + + const onTick = useCallback( + (ticks: number) => { + for (let i = 0; i < ticks; i++) { + if (phase === "forward") { + if (turns + TURN_STEP >= MAX_TURNS) { + crank(MAX_TURNS - turns); + setTurns(MAX_TURNS); + setPhase("back"); + break; + } + crank(TURN_STEP); + setTurns((t) => t + TURN_STEP); + } else { + if (turns - TURN_STEP <= 0) { + crank(-turns); + setTurns(0); + setPhase("forward"); + break; + } + crank(-TURN_STEP); + setTurns((t) => t - TURN_STEP); + } + } + forceRender((v) => v + 1); + }, + [crank, phase, turns], + ); + const { playing, setPlaying, toggle } = useSimLoop(onTick, speed); + + const reset = useCallback(() => { + setPlaying(false); + partsRef.current = seedParticles(); + rngRef.current = makeRng(seed * 7717 + 3); + setSeed((s) => s + 1); + setTurns(0); + setPhase("forward"); + forceRender((v) => v + 1); + }, [seed, setPlaying]); + + /* --------------------------- drawing --------------------------- */ + + useEffect(() => { + const canvas = canvasRef.current; + if (!canvas) return; + const ctx = canvas.getContext("2d"); + if (!ctx) return; + const dpr = window.devicePixelRatio || 1; + const css = 250; + const px = Math.round(css * dpr); + if (canvas.width !== px) { + canvas.width = px; + canvas.height = px; + } + ctx.clearRect(0, 0, px, px); + const mid = px / 2; + const scale = mid * 0.97; + const toPx = (v: number) => mid + v * scale; + + // Vessel walls. + ctx.strokeStyle = "rgba(140,130,110,0.55)"; + ctx.lineWidth = 1 * dpr; + ctx.beginPath(); + ctx.arc(mid, mid, R_OUT * scale, 0, 2 * Math.PI); + ctx.stroke(); + ctx.beginPath(); + ctx.arc(mid, mid, R_IN * scale, 0, 2 * Math.PI); + ctx.stroke(); + + const p = partsRef.current; + const size = Math.max(1.4 * dpr, 1); + for (let i = 0; i < p.x.length; i++) { + ctx.fillStyle = BLOBS[p.blob[i]].color; + ctx.fillRect(toPx(p.x[i]) - size / 2, toPx(p.y[i]) - size / 2, size, size); + } + }); + + /** Mean distance from each particle's starting point, in vessel radii. */ + const displacement = (() => { + const p = partsRef.current; + let sum = 0; + for (let i = 0; i < p.x.length; i++) { + sum += Math.hypot(p.x[i] - p.x0[i], p.y[i] - p.y0[i]); + } + return sum / p.x.length; + })(); + + const backHome = turns < 0.02; + const medium = + jitter === 0 ? "corn syrup" : jitter < 0.45 ? "glycerine" : "water"; + + return ( + onTick(12)} + speed={speed} + speeds={SPEEDS} + onSpeed={setSpeed} + readouts={[ + { label: "medium", value: medium }, + { label: "turns", value: turns.toFixed(2) }, + { + label: "phase", + value: phase === "forward" ? "cranking in →" : "← cranking back", + }, + { + label: "displacement", + value: displacement.toFixed(3), + color: + backHome && displacement > 0.05 + ? "var(--viz-danger)" + : "var(--viz-reward)", + }, + ]} + > +
+
+ +
+
+
what to watch
+

+ {jitter === 0 ? ( + <> + Every particle is following a reversible path. + The dye looks thoroughly mixed at 4 turns, but no information + has been destroyed — crank back and it returns to three clean + blobs, displacement ≈ 0. + + ) : ( + <> + The shear is identical, but each particle now also takes a + random walk that does not reverse. Crank back + and the spirals unwind while the jitter stays. The blobs never + return, and the displacement readout never comes home. + + )} +

+ {backHome && turns === 0 && displacement > 0.001 && ( +

0.05 + ? "var(--viz-danger)" + : "var(--viz-reward)", + }} + > + {displacement > 0.05 + ? `Back at zero turns, and the dye is ${displacement.toFixed(2)} radii from where it started. That gap is genuinely destroyed information.` + : "Back at zero turns, and the dye is home. Nothing was destroyed."} +

+ )} +
+
+ + +
+ ); +} diff --git a/src/components/visualizations/ModeCoverage.tsx b/src/components/visualizations/ModeCoverage.tsx new file mode 100644 index 0000000..ea9375b --- /dev/null +++ b/src/components/visualizations/ModeCoverage.tsx @@ -0,0 +1,253 @@ +import { useCallback, useRef, useState } from "react"; +import SimShell from "./lib/SimShell"; +import { useSimLoop } from "./lib/useSimLoop"; + +// Why "drop a mode" is cheap under one objective and catastrophic under +// another. +// +// The target p is a fixed three-component mixture. The model q is a single +// Gaussian — deliberately too simple to fit p, which is the whole point: when +// a model cannot represent everything, the *divergence* decides what it gives +// up. Three copies of q are fitted simultaneously from the same start, under +// forward KL, reverse KL, and Jensen–Shannon. +// +// Everything is computed by direct numerical integration on a fixed grid, so +// these are the actual objectives rather than sampled estimates, and the +// gradients are finite differences on two parameters. + +const GRID_LO = -6; +const GRID_HI = 6; +const NG = 480; +const DX = (GRID_HI - GRID_LO) / NG; + +const COMPONENTS = [ + { mu: -3.0, sd: 0.5, w: 0.35 }, + { mu: 0.1, sd: 0.45, w: 0.3 }, + { mu: 3.2, sd: 0.6, w: 0.35 }, +]; + +const XS = Float64Array.from({ length: NG }, (_, i) => GRID_LO + (i + 0.5) * DX); +const P = (() => { + const p = new Float64Array(NG); + for (let i = 0; i < NG; i++) { + let v = 0; + for (const c of COMPONENTS) { + v += + (c.w / (c.sd * Math.sqrt(2 * Math.PI))) * + Math.exp(-((XS[i] - c.mu) ** 2) / (2 * c.sd * c.sd)); + } + p[i] = v; + } + return p; +})(); + +function qDensity(mu: number, logSd: number): Float64Array { + const sd = Math.exp(logSd); + const q = new Float64Array(NG); + for (let i = 0; i < NG; i++) { + q[i] = + (1 / (sd * Math.sqrt(2 * Math.PI))) * + Math.exp(-((XS[i] - mu) ** 2) / (2 * sd * sd)); + } + return q; +} + +const EPS = 1e-12; + +/** ∫ p log(p/q) — pays an unbounded price wherever p has mass and q does not. */ +function forwardKL(mu: number, logSd: number): number { + const q = qDensity(mu, logSd); + let s = 0; + for (let i = 0; i < NG; i++) { + if (P[i] > EPS) s += P[i] * Math.log(P[i] / Math.max(q[i], EPS)) * DX; + } + return s; +} + +/** ∫ q log(q/p) — pays nothing for regions of p that q simply avoids. */ +function reverseKL(mu: number, logSd: number): number { + const q = qDensity(mu, logSd); + let s = 0; + for (let i = 0; i < NG; i++) { + if (q[i] > EPS) s += q[i] * Math.log(q[i] / Math.max(P[i], EPS)) * DX; + } + return s; +} + +/** What the original GAN objective reduces to at an optimal discriminator. */ +function jensenShannon(mu: number, logSd: number): number { + const q = qDensity(mu, logSd); + let s = 0; + for (let i = 0; i < NG; i++) { + const m = 0.5 * (P[i] + q[i]); + if (P[i] > EPS) s += 0.5 * P[i] * Math.log(P[i] / Math.max(m, EPS)) * DX; + if (q[i] > EPS) s += 0.5 * q[i] * Math.log(q[i] / Math.max(m, EPS)) * DX; + } + return s; +} + +type Objective = (mu: number, logSd: number) => number; + +const FITTERS: { key: string; label: string; color: string; fn: Objective }[] = [ + { key: "fwd", label: "forward KL(p‖q) — likelihood", color: "var(--viz-reward)", fn: forwardKL }, + { key: "js", label: "Jensen–Shannon — the GAN", color: "var(--viz-value)", fn: jensenShannon }, + { key: "rev", label: "reverse KL(q‖p)", color: "var(--viz-danger)", fn: reverseKL }, +]; + +interface Fit { + mu: number; + logSd: number; +} + +const START_SD = 0.55; + +export default function ModeCoverage() { + // Where all three fits begin. This is the control that matters: measured + // over seven starting points, forward KL lands at mu ≈ 0.1, sd = 2.65 + // covering all three modes EVERY time, while JS and reverse KL end up + // wherever they began — on the left mode from -3.6, the middle from 0.1, + // the right from 3.0. Same model, same data, same optimizer; the + // initialization picks the answer. + const [start, setStart] = useState(3); + const fits = useRef(FITTERS.map(() => ({ mu: 3, logSd: Math.log(START_SD) }))); + const [, forceRender] = useState(0); + const [iters, setIters] = useState(0); + + const onTick = useCallback((ticks: number) => { + const h = 1e-4; + const lr = 0.02; + for (let n = 0; n < ticks; n++) { + FITTERS.forEach((f, i) => { + const cur = fits.current[i]; + // Finite-difference gradient on two parameters — exact enough at + // this scale and far simpler than differentiating three objectives. + const base = f.fn(cur.mu, cur.logSd); + const gMu = (f.fn(cur.mu + h, cur.logSd) - base) / h; + const gSd = (f.fn(cur.mu, cur.logSd + h) - base) / h; + cur.mu -= lr * Math.max(-4, Math.min(4, gMu)); + cur.logSd -= lr * Math.max(-4, Math.min(4, gSd)); + cur.logSd = Math.max(Math.log(0.12), Math.min(Math.log(4), cur.logSd)); + }); + } + setIters((v) => v + ticks); + forceRender((v) => v + 1); + }, []); + const { playing, setPlaying, toggle } = useSimLoop(onTick, 40); + + const restart = useCallback((mu: number) => { + fits.current = FITTERS.map(() => ({ mu, logSd: Math.log(START_SD) })); + setIters(0); + forceRender((v) => v + 1); + }, []); + + const reset = useCallback(() => { + setPlaying(false); + restart(start); + }, [restart, start, setPlaying]); + + /** How many of the three components sit under a given fit's bulk. */ + const covered = (f: Fit) => { + const sd = Math.exp(f.logSd); + return COMPONENTS.filter((c) => Math.abs(c.mu - f.mu) < 2 * sd).length; + }; + + return ( + onTick(10)} + readouts={[ + { label: "steps", value: `${iters}` }, + ...FITTERS.map((f, i) => ({ + label: f.key === "fwd" ? "fwd KL covers" : f.key === "js" ? "JS covers" : "rev KL covers", + value: `${covered(fits.current[i])}/3`, + color: f.color, + })), + ]} + > + + + +
+ + + target p (three modes) + + {FITTERS.map((f) => ( + + + {f.label} + + ))} +
+
+ ); +} + +function Densities({ fits, start }: { fits: Fit[]; start: number }) { + const W = 560; + const H = 210; + const PAD = { l: 8, r: 8, t: 10, b: 20 }; + const pw = W - PAD.l - PAD.r; + const ph = H - PAD.t - PAD.b; + + let peak = 0; + for (const v of P) peak = Math.max(peak, v); + const curves = fits.map((f) => qDensity(f.mu, f.logSd)); + for (const c of curves) for (const v of c) peak = Math.max(peak, v); + + const X = (x: number) => PAD.l + ((x - GRID_LO) / (GRID_HI - GRID_LO)) * pw; + const Y = (v: number) => PAD.t + ph - (v / peak) * ph; + const path = (d: Float64Array) => + Array.from(d, (v, i) => `${i === 0 ? "M" : "L"}${X(XS[i]).toFixed(1)},${Y(v).toFixed(1)}`).join(" "); + + return ( + + One Gaussian fitted to a three-mode target under three divergences + + {curves.map((c, i) => ( + + ))} + + + {COMPONENTS.map((c) => ( + + mode + + ))} + + ); +} diff --git a/src/components/visualizations/StepBudget.tsx b/src/components/visualizations/StepBudget.tsx new file mode 100644 index 0000000..e62b167 --- /dev/null +++ b/src/components/visualizations/StepBudget.tsx @@ -0,0 +1,381 @@ +import { useCallback, useMemo, useRef, useState } from "react"; +import PixelCanvas from "./lib/PixelCanvas"; +import SimShell from "./lib/SimShell"; +import Sparkline from "./lib/Sparkline"; +import { + alphaBar, + entropyBits, + epsFromX0, + expectedX0, + gaussian, + makeRng, + posteriorWeights, + reverseStep, +} from "./lib/diffusionMath"; +import { makeDataset, meanImage, SPRITE_N, SPRITE_SIZE } from "./lib/sprites"; +import { useSimLoop } from "./lib/useSimLoop"; + +// Reverse sampling with the exact denoiser, run at a step budget the reader +// picks. +// +// The one-step case is the point of the widget: with a single step the model +// must jump from pure noise straight to a clean image, its posterior over +// which training image it is looking at is uniform, and the mean of that +// posterior is the average of the whole dataset — a blur. Measured at 42% +// midtone pixels against 5% for a real sprite. +// +// A kernel-width knob was built here and then removed: widening the posterior +// does NOT buy generalization, measured flat at 0.0032 distance-to-nearest +// training image across widths 1-8 and across an effective-noise floor up to +// 0.4. In 576 dimensions the posterior over a finite training set is +// effectively deterministic, so smoothing only delays saturation instead of +// preventing it. Generalization needs a different function class, not a +// wider kernel — which is the argument the page makes. +// +// What this widget deliberately does NOT claim is that quality keeps climbing +// to T = 1000. It does not, and measurement says so: with an *exact* +// denoiser this converges by about T = 4 regardless of dataset size. Real +// systems need far more because their denoiser is learned and approximate. +// The page says that in prose rather than faking a knob for it. + +const DATASET_SIZE = 128; +const STEP_CHOICES = [1, 2, 4, 8, 16, 50, 200]; +const SPEEDS = [ + { label: "1×", value: 6 }, + { label: "4×", value: 24 }, + { label: "20×", value: 120 }, +]; + +interface Run { + /** Current iterate. */ + x: Float32Array; + /** The denoiser's current guess at the clean image. */ + x0hat: Float32Array; + /** Sorted posterior mass, largest first — how sure the model is. */ + top: number[]; + entropy: number; + step: number; + ab: number; + done: boolean; +} + +export default function StepBudget() { + const [steps, setSteps] = useState(1); + const [eta, setEta] = useState(1); + const [speed, setSpeed] = useState(SPEEDS[0].value); + const [seed, setSeed] = useState(1); + const [, forceRender] = useState(0); + + const dataset = useMemo(() => makeDataset(DATASET_SIZE, 3), []); + const datasetMean = useMemo(() => meanImage(dataset), [dataset]); + + const rngRef = useRef<() => number>(makeRng(1)); + const entropyTrace = useRef([]); + const gallery = useRef([]); + + const start = useCallback( + (s: number): Run => { + const rng = makeRng(s * 6151 + 7); + rngRef.current = rng; + entropyTrace.current = []; + const x = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) x[i] = gaussian(rng); + // Before any step has run, the model has seen nothing but noise: the + // posterior is exactly uniform and its mean is the dataset average. + // Seed the display with that rather than an empty buffer, so the + // opening state shows the claim the caption makes about it. + return { + x, + x0hat: datasetMean, + top: new Array(12).fill(1 / DATASET_SIZE), + entropy: Math.log2(DATASET_SIZE), + step: 0, + ab: alphaBar(1), + done: false, + }; + }, + [datasetMean], + ); + + const runRef = useRef(start(1)); + + const advance = useCallback(() => { + const run = runRef.current; + if (run.done) return; + const k = steps - run.step; + const abT = alphaBar(k / steps); + const abPrev = alphaBar((k - 1) / steps); + + const w = posteriorWeights(run.x, dataset, abT); + const x0hat = expectedX0(dataset, w); + const eps = epsFromX0(run.x, x0hat, abT); + const next = reverseStep(run.x, x0hat, eps, abT, abPrev, eta, rngRef.current); + + const sorted = Array.from(w).sort((a, b) => b - a).slice(0, 12); + const h = entropyBits(w); + entropyTrace.current.push(h); + + const done = run.step + 1 >= steps; + runRef.current = { + x: next, + x0hat, + top: sorted, + entropy: h, + step: run.step + 1, + ab: abPrev, + done, + }; + if (done) { + gallery.current = [next.slice(), ...gallery.current].slice(0, 6); + } + }, [dataset, steps, eta]); + + const onTick = useCallback( + (ticks: number) => { + for (let i = 0; i < ticks; i++) { + if (runRef.current.done) break; + advance(); + } + forceRender((v) => v + 1); + }, + [advance], + ); + const { playing, setPlaying, toggle } = useSimLoop(onTick, speed); + + const restart = useCallback( + (nextSeed: number) => { + runRef.current = start(nextSeed); + forceRender((v) => v + 1); + }, + [start], + ); + + const reset = useCallback(() => { + setPlaying(false); + gallery.current = []; + const s = seed + 1; + setSeed(s); + restart(s); + }, [seed, restart, setPlaying]); + + /** Fraction of pixels stranded between ink and paper — a blend's signature. */ + const midtone = useMemo(() => { + const src = runRef.current.done ? runRef.current.x : runRef.current.x0hat; + let n = 0; + for (const v of src) if (Math.abs(v) < 0.5) n++; + return n / src.length; + // runRef is a ref: recompute whenever the render was forced. + }, [runRef.current.step, runRef.current.done]); + + const run = runRef.current; + const started = run.step > 0; + + return ( + { + if (run.done) { + const s = seed + 1; + setSeed(s); + restart(s); + } + toggle(); + }} + onReset={reset} + onStep={() => onTick(1)} + speed={speed} + speeds={SPEEDS} + onSpeed={setSpeed} + readouts={[ + { label: "step", value: `${run.step} / ${steps}` }, + { label: "ᾱ", value: run.ab.toFixed(3), color: "var(--viz-policy)" }, + { + label: "posterior", + value: `${run.entropy.toFixed(2)} bits`, + color: "var(--viz-kl)", + }, + { + label: "midtone", + value: `${(midtone * 100).toFixed(0)}%`, + color: midtone > 0.2 ? "var(--viz-danger)" : "var(--viz-reward)", + }, + ]} + > +
+
+
the sample
+
+ + +
+
+ + {started + ? "left: where the chain is. right: what it thinks the answer is." + : "before the first step, every training image is equally likely — so the best guess is their average."} + +
+
+ +
+
how sure is it? (posterior mass)
+ +
+ posterior entropy over the run +
+ `${v.toFixed(1)}b`} + /> +
+
+ +
+
+ + step budget T = {steps} + + + {STEP_CHOICES.map((n) => ( + + ))} + +
+ +
+ sampler + + {[ + { label: "DDPM (η=1)", value: 1 }, + { label: "DDIM (η=0)", value: 0 }, + ].map((o) => ( + + ))} + +
+ +
+ + {gallery.current.length > 0 && ( + <> +
+ finished samples (newest first) — is it covering the dataset? +
+
+ {gallery.current.map((g, i) => ( + + ))} +
+ + )} +
+ ); +} + +/* ------------------------------------------------------------------ */ + +function WeightBars({ top, total }: { top: number[]; total: number }) { + const W = 300; + const H = 66; + const n = 12; + const bw = W / n; + const uniform = 1 / total; + // Uniform mass is 1/128 — on a 0..1 axis that is half a pixel tall and + // reads as "no bars at all", which is the opposite of what the flat + // posterior is meant to show. Scale to the leading weight instead, with a + // floor of 8x uniform so the flat state stays legible. + const scale = Math.max(top[0] ?? uniform, uniform * 8); + return ( + + Twelve largest posterior weights + {Array.from({ length: n }, (_, i) => { + const w = top[i] ?? 0; + const h = Math.min(1, w / scale) * (H - 12); + return ( + + ); + })} + {/* Where the bars would sit if every training image were equally likely. */} + + + + top 12 of {total} training images + + + ); +} diff --git a/src/components/visualizations/ThreeTargets.tsx b/src/components/visualizations/ThreeTargets.tsx new file mode 100644 index 0000000..e91a3ce --- /dev/null +++ b/src/components/visualizations/ThreeTargets.tsx @@ -0,0 +1,318 @@ +import { useCallback, useMemo, useRef, useState } from "react"; +import PixelCanvas from "./lib/PixelCanvas"; +import SimShell from "./lib/SimShell"; +import { alphaBar, gaussian, makeRng } from "./lib/diffusionMath"; +import { useSimLoop } from "./lib/useSimLoop"; +import { SPRITES, SPRITE_N, SPRITE_SIZE } from "./lib/sprites"; + +// Three ways to parameterize the same prediction, and why the choice matters. +// +// x̂₀, ε̂ and the score are algebraically interchangeable — given any one of +// them plus x_t you can compute the other two. So the choice cannot change +// what the model *knows*. What it changes is how a prediction error of fixed +// size propagates into the thing we actually care about, the clean image: +// +// predict x₀ : error in x̂₀ = δ (flat, 1.0 everywhere) +// predict ε : error in x̂₀ = δ·√(1-ᾱ)/√ᾱ (→ 0 as t → 0) +// predict s : error in x̂₀ = δ·(1-ᾱ)/√ᾱ (→ 0 faster, but the +// target itself blows up) +// +// The ε row is the free skip connection everyone talks about, made numeric: +// at low noise a mistake in the predicted noise is multiplied by something +// near zero before it reaches the image. And low noise is exactly where the +// final quality of a sample is decided. + +const T = 1000; + +function rms(a: Float32Array): number { + let s = 0; + for (const v of a) s += v * v; + return Math.sqrt(s / a.length); +} + +export default function ThreeTargets() { + const [t, setT] = useState(620); + const [spriteIdx, setSpriteIdx] = useState(3); + const dir = useRef(-1); + + // Run sweeps the noise level rather than sitting dead: the whole point of + // the chart is how the three curves separate as t moves, so the default + // animation is a tour of that separation. + const onTick = useCallback((ticks: number) => { + setT((prev) => { + let next = prev + dir.current * ticks; + if (next >= T - 1) { + next = T - 1; + dir.current = -1; + } else if (next <= 1) { + next = 1; + dir.current = 1; + } + return next; + }); + }, []); + const { playing, setPlaying, toggle } = useSimLoop(onTick, 110); + + const x0 = SPRITES[spriteIdx].data; + const ab = alphaBar(t / T); + + const eps = useMemo(() => { + const rng = makeRng(4242); + const e = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) e[i] = gaussian(rng); + return e; + }, []); + + const { xt, score } = useMemo(() => { + const sa = Math.sqrt(ab); + const sn = Math.sqrt(1 - ab); + const xt = new Float32Array(SPRITE_N); + const score = new Float32Array(SPRITE_N); + for (let i = 0; i < SPRITE_N; i++) { + xt[i] = sa * x0[i] + sn * eps[i]; + // s = ∇ log q(x_t) = -ε / √(1-ᾱ) + score[i] = -eps[i] / sn; + } + return { xt, score }; + }, [ab, x0, eps]); + + // Error amplification: a unit prediction error in each target, expressed as + // the error it causes in the reconstructed clean image. + const amp = { + x0: 1, + eps: Math.sqrt(1 - ab) / Math.sqrt(ab), + score: (1 - ab) / Math.sqrt(ab), + }; + + const scoreRms = rms(score); + + return ( + onTick(25)} + onReset={() => { + setPlaying(false); + dir.current = -1; + setT(620); + setSpriteIdx(3); + }} + readouts={[ + { label: "t", value: `${t}` }, + { label: "ᾱ", value: ab.toFixed(3), color: "var(--viz-policy)" }, + { label: "RMS ε", value: "1.00", color: "var(--viz-reward)" }, + { + label: "RMS score", + value: scoreRms > 99 ? scoreRms.toExponential(1) : scoreRms.toFixed(2), + color: scoreRms > 8 ? "var(--viz-danger)" : "var(--viz-kl)", + }, + ]} + > +
+
+
what the network could be asked for
+
+ + + + +
+
+ {SPRITES.map((s, i) => ( + + ))} +
+
+ +
+
+ how a unit prediction error lands on the image +
+ +
+ + + predict x₀ + + + + predict ε + + + + predict score + +
+
+ + x₀-pred + + ×{amp.x0.toFixed(2)} + + + + ε-pred + + ×{amp.eps < 0.01 ? amp.eps.toExponential(1) : amp.eps.toFixed(2)} + + + + score-pred + + ×{amp.score < 0.01 ? amp.score.toExponential(1) : amp.score.toFixed(2)} + + +
+
+
+ + +
+ ); +} + +/* ------------------------------------------------------------------ */ + +function AmpChart({ t }: { t: number }) { + const W = 300; + const H = 150; + const PAD = { l: 34, r: 8, t: 8, b: 20 }; + const pw = W - PAD.l - PAD.r; + const ph = H - PAD.t - PAD.b; + + const yLo = -3; + const yHi = 1.5; + const xOf = (tt: number) => PAD.l + (tt / T) * pw; + const yOf = (v: number) => + PAD.t + ph - ((Math.log10(Math.max(v, 10 ** yLo)) - yLo) / (yHi - yLo)) * ph; + + const path = (f: (ab: number) => number) => { + const pts: string[] = []; + for (let i = 0; i <= 120; i++) { + const tt = 1 + (i / 120) * (T - 2); + const ab = alphaBar(tt / T); + pts.push(`${i === 0 ? "M" : "L"}${xOf(tt).toFixed(1)},${yOf(f(ab)).toFixed(1)}`); + } + return pts.join(" "); + }; + + return ( + + Error amplification by parameterization + {[-3, -2, -1, 0, 1].map((e) => ( + + + + {e === 0 ? "1×" : `10${e < 0 ? "⁻" : ""}${"¹²³".charAt(Math.abs(e) - 1)}`} + + + ))} + + 1)} fill="none" stroke="var(--viz-value)" strokeWidth={1.8} /> + Math.sqrt(1 - ab) / Math.sqrt(ab))} + fill="none" + stroke="var(--viz-reward)" + strokeWidth={1.8} + /> + (1 - ab) / Math.sqrt(ab))} + fill="none" + stroke="var(--viz-kl)" + strokeWidth={1.8} + /> + + + + + + t = 0 (clean) + + + t = T (noise) + + + ); +} diff --git a/src/components/visualizations/lib/PixelCanvas.tsx b/src/components/visualizations/lib/PixelCanvas.tsx new file mode 100644 index 0000000..c52828f --- /dev/null +++ b/src/components/visualizations/lib/PixelCanvas.tsx @@ -0,0 +1,122 @@ +import { useEffect, useRef } from "react"; + +/** + * Canvas-backed renderer for the small images on the diffusion track. + * + * Everything else on this site draws with SVG, and for tens of marks that is + * the right call. It is the wrong call here: a 24×24 frame is 576 cells, and + * re-rendering that many React s every animation frame — across four + * panels — spends the whole frame budget in reconciliation. One canvas and + * one ImageData blit costs effectively nothing. + * + * Images render as grayscale in both themes rather than following the site's + * ink/paper colors. Theme-inverting an image would mean the same sprite + * appears as white-on-black in dark mode and black-on-white in light, which + * reads as two different images. The frame around it is theme-aware instead. + */ + +export interface PixelCanvasProps { + /** Row-major pixel values, `cols * rows` of them. */ + data: Float32Array; + cols: number; + rows: number; + /** Rendered size in CSS pixels (square unless `height` is given). */ + size?: number; + height?: number; + /** Value range mapped onto the ramp. DDPM convention by default. */ + domain?: [number, number]; + /** Override the ramp. Receives a 0–1 value, returns 8-bit RGB. */ + toRGB?: (v: number) => [number, number, number]; + /** Caption rendered under the frame. */ + label?: string; + /** Highlight color for the frame, e.g. a `--viz-*` token. */ + accent?: string; + title?: string; +} + +function grayscale(v: number): [number, number, number] { + const g = Math.round(v * 255); + return [g, g, g]; +} + +export default function PixelCanvas({ + data, + cols, + rows, + size = 132, + height, + domain = [-1, 1], + toRGB = grayscale, + label, + accent, + title, +}: PixelCanvasProps) { + const canvasRef = useRef(null); + const bufferRef = useRef(null); + + // No dependency array: the sims reuse and mutate their Float32Arrays in + // place to avoid per-frame allocation, so array identity is not a signal + // that the pixels changed. Redraw on every commit instead — 576 pixels is + // far cheaper than the bookkeeping needed to track it properly. + useEffect(() => { + const canvas = canvasRef.current; + if (!canvas) return; + const ctx = canvas.getContext("2d"); + if (!ctx) return; + + if (!bufferRef.current) { + bufferRef.current = document.createElement("canvas"); + } + const buffer = bufferRef.current; + if (buffer.width !== cols || buffer.height !== rows) { + buffer.width = cols; + buffer.height = rows; + } + const bufferCtx = buffer.getContext("2d"); + if (!bufferCtx) return; + + const image = bufferCtx.createImageData(cols, rows); + const [lo, hi] = domain; + const span = hi - lo || 1; + const n = Math.min(data.length, cols * rows); + for (let i = 0; i < n; i++) { + const v = Math.min(1, Math.max(0, (data[i] - lo) / span)); + const [r, g, b] = toRGB(v); + const o = i * 4; + image.data[o] = r; + image.data[o + 1] = g; + image.data[o + 2] = b; + image.data[o + 3] = 255; + } + bufferCtx.putImageData(image, 0, 0); + + const cssH = height ?? size; + const dpr = window.devicePixelRatio || 1; + const pxW = Math.round(size * dpr); + const pxH = Math.round(cssH * dpr); + if (canvas.width !== pxW || canvas.height !== pxH) { + canvas.width = pxW; + canvas.height = pxH; + } + ctx.imageSmoothingEnabled = false; + ctx.clearRect(0, 0, pxW, pxH); + ctx.drawImage(buffer, 0, 0, pxW, pxH); + }); + + return ( +
+ + {label &&
{label}
} +
+ ); +} diff --git a/src/components/visualizations/lib/diffusionMath.ts b/src/components/visualizations/lib/diffusionMath.ts new file mode 100644 index 0000000..aa7423f --- /dev/null +++ b/src/components/visualizations/lib/diffusionMath.ts @@ -0,0 +1,257 @@ +/** + * The diffusion math shared by every widget on the track. One source of + * truth, so a schedule change cannot make two figures disagree. + * + * Everything here is exact, not illustrative. `expectedX0` is the *optimal* + * denoiser for the sprite training set, not an approximation of one: with a + * finite training set the posterior over clean images is a softmax, and its + * mean is a closed form. That is what lets the sampler run real DDPM + * ancestral sampling in a browser with no trained network anywhere. + * + * Its one limitation is a teaching asset rather than a defect — an exact + * empirical posterior can only ever reproduce training images. That is + * memorization, and it is precisely why real systems need a network that + * generalizes. The `temperature` knob on `posteriorWeights` widens the + * kernel to show generalization being bought. + */ + +/* ------------------------------------------------------------------ * + * Randomness — seeded, so Reset reproduces a run exactly. + * ------------------------------------------------------------------ */ + +/** mulberry32: small, fast, good enough for visuals, and seedable. */ +export function makeRng(seed: number): () => number { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +/** Standard normal via Box–Muller. Discards the second variate; fine here. */ +export function gaussian(rng: () => number): number { + // u must be strictly positive for the log. + const u = Math.max(rng(), 1e-12); + const v = rng(); + return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v); +} + +/* ------------------------------------------------------------------ * + * Noise schedule + * ------------------------------------------------------------------ */ + +const COSINE_S = 0.008; +const COSINE_F0 = Math.cos((COSINE_S / (1 + COSINE_S)) * (Math.PI / 2)) ** 2; + +/** + * Cosine schedule (Nichol & Dhariwal 2021) as a *continuous* function of + * u = t/T in [0, 1]. Continuous matters: `StepBudget` slides T from 1 to + * 1000, and a continuous ᾱ means every step count reads off the same curve + * instead of needing its own recomputed product. + * + * ᾱ(0) = 1 (clean) and ᾱ(1) = 0 (pure noise), clamped away from both ends so + * the divisions downstream stay finite. + */ +export function alphaBar(u: number): number { + const c = Math.min(1, Math.max(0, u)); + const f = Math.cos(((c + COSINE_S) / (1 + COSINE_S)) * (Math.PI / 2)) ** 2; + return Math.min(1 - 1e-5, Math.max(1e-5, f / COSINE_F0)); +} + +/** Signal-to-noise ratio at a given ᾱ — the honest x-axis for "how far gone". */ +export function snr(ab: number): number { + return ab / (1 - ab); +} + +/* ------------------------------------------------------------------ * + * Forward process + * ------------------------------------------------------------------ */ + +/** + * One closed-form jump to any noise level: x_t = √ᾱ·x₀ + √(1-ᾱ)·ε. + * No stepwise simulation — that closure under addition is the whole reason + * the forward process is Gaussian. + * + * Writes the noise it drew into `eps` when given, since the training target + * is that noise. + */ +export function forwardSample( + x0: Float32Array, + ab: number, + rng: () => number, + out?: Float32Array, + eps?: Float32Array, +): Float32Array { + const dst = out ?? new Float32Array(x0.length); + const sa = Math.sqrt(ab); + const sn = Math.sqrt(1 - ab); + for (let i = 0; i < x0.length; i++) { + const e = gaussian(rng); + if (eps) eps[i] = e; + dst[i] = sa * x0[i] + sn * e; + } + return dst; +} + +/* ------------------------------------------------------------------ * + * The exact denoiser + * ------------------------------------------------------------------ */ + +/** + * Posterior over which training image produced x_t. + * + * w_i ∝ exp( -‖x_t - √ᾱ·x⁽ⁱ⁾‖² / (2(1-ᾱ)·temperature) ) + * + * `temperature` > 1 widens the kernel: the model stops being certain which + * training image it is looking at and starts blending them, which is a crude + * but honest stand-in for a network that generalizes instead of memorizing. + * + * Computed in log space with the max subtracted — at low noise the exponents + * reach the thousands and a naive exp() would be all zeros or all infinities. + */ +export function posteriorWeights( + xt: Float32Array, + dataset: Float32Array[], + ab: number, + temperature = 1, + out?: Float32Array, +): Float32Array { + const w = out ?? new Float32Array(dataset.length); + const sa = Math.sqrt(ab); + const denom = 2 * (1 - ab) * Math.max(temperature, 1e-6); + + let max = Number.NEGATIVE_INFINITY; + for (let k = 0; k < dataset.length; k++) { + const ref = dataset[k]; + let sq = 0; + for (let i = 0; i < xt.length; i++) { + const d = xt[i] - sa * ref[i]; + sq += d * d; + } + w[k] = -sq / denom; + if (w[k] > max) max = w[k]; + } + + let sum = 0; + for (let k = 0; k < w.length; k++) { + w[k] = Math.exp(w[k] - max); + sum += w[k]; + } + for (let k = 0; k < w.length; k++) w[k] /= sum; + return w; +} + +/** E[x₀ | x_t] — the posterior-weighted average of the training images. */ +export function expectedX0( + dataset: Float32Array[], + weights: Float32Array, + out?: Float32Array, +): Float32Array { + const n = dataset[0].length; + const dst = out ?? new Float32Array(n); + dst.fill(0); + for (let k = 0; k < dataset.length; k++) { + const w = weights[k]; + if (w < 1e-9) continue; + const ref = dataset[k]; + for (let i = 0; i < n; i++) dst[i] += w * ref[i]; + } + return dst; +} + +/** + * The same object, three ways. ε-prediction, x₀-prediction and the score are + * one quantity reparameterized (Tweedie), which is the payoff of + * `predict-the-noise`: + * + * ε̂ = (x_t - √ᾱ·x̂₀) / √(1-ᾱ) + * score = -ε̂ / √(1-ᾱ) + */ +export function epsFromX0( + xt: Float32Array, + x0hat: Float32Array, + ab: number, + out?: Float32Array, +): Float32Array { + const dst = out ?? new Float32Array(xt.length); + const sa = Math.sqrt(ab); + const sn = Math.sqrt(1 - ab); + for (let i = 0; i < xt.length; i++) dst[i] = (xt[i] - sa * x0hat[i]) / sn; + return dst; +} + +export function scoreFromEps( + eps: Float32Array, + ab: number, + out?: Float32Array, +): Float32Array { + const dst = out ?? new Float32Array(eps.length); + const sn = Math.sqrt(1 - ab); + for (let i = 0; i < eps.length; i++) dst[i] = -eps[i] / sn; + return dst; +} + +/* ------------------------------------------------------------------ * + * Reverse process + * ------------------------------------------------------------------ */ + +/** + * One reverse step, in the DDIM parameterization that covers both samplers + * (Song et al. 2020, eq. 12): + * + * x_{t-1} = √ᾱ_prev·x̂₀ + √(1 - ᾱ_prev - σ²)·ε̂ + σ·z + * σ = η·√((1-ᾱ_prev)/(1-ᾱ_t))·√(1 - ᾱ_t/ᾱ_prev) + * + * η = 1 is ancestral DDPM sampling; η = 0 is deterministic DDIM. Exposing η + * as a knob costs nothing and makes "where did the randomness go?" a thing + * the reader can test rather than take on faith. + */ +export function reverseStep( + xt: Float32Array, + x0hat: Float32Array, + epsHat: Float32Array, + abT: number, + abPrev: number, + eta: number, + rng: () => number, + out?: Float32Array, +): Float32Array { + const dst = out ?? new Float32Array(xt.length); + const ratio = Math.min(1, abT / abPrev); + const sigma = + eta * Math.sqrt((1 - abPrev) / (1 - abT)) * Math.sqrt(Math.max(0, 1 - ratio)); + const dirCoef = Math.sqrt(Math.max(0, 1 - abPrev - sigma * sigma)); + const sa = Math.sqrt(abPrev); + for (let i = 0; i < xt.length; i++) { + const z = sigma > 0 ? gaussian(rng) : 0; + dst[i] = sa * x0hat[i] + dirCoef * epsHat[i] + sigma * z; + } + return dst; +} + +/* ------------------------------------------------------------------ * + * Small helpers + * ------------------------------------------------------------------ */ + +/** Root-mean-square error, for "how far is this from the true image". */ +export function rmse(a: Float32Array, b: Float32Array): number { + let sq = 0; + for (let i = 0; i < a.length; i++) { + const d = a[i] - b[i]; + sq += d * d; + } + return Math.sqrt(sq / a.length); +} + +/** Shannon entropy of the posterior, in bits — "how sure is it, really". */ +export function entropyBits(weights: Float32Array): number { + let h = 0; + for (let k = 0; k < weights.length; k++) { + const p = weights[k]; + if (p > 1e-12) h -= p * Math.log2(p); + } + return h; +} diff --git a/src/components/visualizations/lib/pca.ts b/src/components/visualizations/lib/pca.ts new file mode 100644 index 0000000..4165120 --- /dev/null +++ b/src/components/visualizations/lib/pca.ts @@ -0,0 +1,125 @@ +/** + * Top-k principal components by power iteration with deflation. + * + * A linear autoencoder trained to convergence learns exactly the PCA + * subspace, so this gives the latent-diffusion page a real encoder/decoder + * pair — computed rather than gestured at — without shipping trained weights. + * It understates what a real VAE achieves (that one is non-linear and + * perceptually trained), and the page says so; what it gets right is the part + * that matters: how few dimensions the interesting variation actually needs. + */ + +export interface PCA { + mean: Float32Array; + /** `k` unit-norm components, each of length `dim`. */ + components: Float32Array[]; + /** Variance captured by each component. */ + eigenvalues: number[]; + totalVariance: number; +} + +export function fitPCA(data: Float32Array[], k: number, iters = 60): PCA { + const n = data.length; + const dim = data[0].length; + + const mean = new Float32Array(dim); + for (const d of data) for (let i = 0; i < dim; i++) mean[i] += d[i]; + for (let i = 0; i < dim; i++) mean[i] /= n; + + // Centred copy — deflation mutates it, so never touch the caller's arrays. + const X: Float32Array[] = data.map((d) => { + const r = new Float32Array(dim); + for (let i = 0; i < dim; i++) r[i] = d[i] - mean[i]; + return r; + }); + + let totalVariance = 0; + for (const r of X) for (let i = 0; i < dim; i++) totalVariance += r[i] * r[i]; + totalVariance /= n; + + const components: Float32Array[] = []; + const eigenvalues: number[] = []; + + // Deterministic start vector: reproducible across reloads. + let seed = 12345; + const rand = () => { + seed = (seed * 1664525 + 1013904223) >>> 0; + return seed / 4294967296 - 0.5; + }; + + for (let c = 0; c < Math.min(k, dim); c++) { + let v = new Float32Array(dim); + for (let i = 0; i < dim; i++) v[i] = rand(); + normalize(v); + + for (let it = 0; it < iters; it++) { + // w = Xᵀ X v / n, without ever forming the dim×dim covariance. + const w = new Float32Array(dim); + for (const r of X) { + let dot = 0; + for (let i = 0; i < dim; i++) dot += r[i] * v[i]; + if (dot === 0) continue; + for (let i = 0; i < dim; i++) w[i] += dot * r[i]; + } + for (let i = 0; i < dim; i++) w[i] /= n; + const norm = normalize(w); + if (norm === 0) break; + v = w; + } + + // Rayleigh quotient gives the variance along v. + let lambda = 0; + for (const r of X) { + let dot = 0; + for (let i = 0; i < dim; i++) dot += r[i] * v[i]; + lambda += dot * dot; + } + lambda /= n; + + components.push(v); + eigenvalues.push(lambda); + + // Deflate so the next iteration finds the next component. + for (const r of X) { + let dot = 0; + for (let i = 0; i < dim; i++) dot += r[i] * v[i]; + for (let i = 0; i < dim; i++) r[i] -= dot * v[i]; + } + } + + return { mean, components, eigenvalues, totalVariance }; +} + +function normalize(v: Float32Array): number { + let s = 0; + for (const x of v) s += x * x; + const n = Math.sqrt(s); + if (n > 1e-12) for (let i = 0; i < v.length; i++) v[i] /= n; + return n; +} + +/** Project onto the first `k` components — the encoder. */ +export function encode(pca: PCA, x: Float32Array, k: number): Float64Array { + const kk = Math.min(k, pca.components.length); + const z = new Float64Array(kk); + for (let c = 0; c < kk; c++) { + let dot = 0; + const comp = pca.components[c]; + for (let i = 0; i < x.length; i++) dot += (x[i] - pca.mean[i]) * comp[i]; + z[c] = dot; + } + return z; +} + +/** Rebuild from the code — the decoder. */ +export function decode(pca: PCA, z: Float64Array, out?: Float32Array): Float32Array { + const dim = pca.mean.length; + const dst = out ?? new Float32Array(dim); + dst.set(pca.mean); + for (let c = 0; c < z.length; c++) { + const comp = pca.components[c]; + const zc = z[c]; + for (let i = 0; i < dim; i++) dst[i] += zc * comp[i]; + } + return dst; +} diff --git a/src/components/visualizations/lib/spectrum.ts b/src/components/visualizations/lib/spectrum.ts new file mode 100644 index 0000000..b446d09 --- /dev/null +++ b/src/components/visualizations/lib/spectrum.ts @@ -0,0 +1,121 @@ +/** + * Radially-averaged power spectrum of a small square image. + * + * This exists to make one specific claim checkable on screen: natural images + * have roughly 1/f power — most of their energy sits at low spatial + * frequencies — while Gaussian noise is flat across all frequencies. Adding + * noise therefore does not degrade an image uniformly. It drowns the high + * frequencies first and the coarse structure last, which is why denoising + * runs coarse-to-fine and why the reverse process *looks* like an image + * coming into focus. + * + * Images here are 24×24, so a naive separable DFT is a few tens of thousands + * of multiplies — far cheaper than the frame it is drawn into, and worth + * more than an FFT's complexity. + */ + +interface Twiddle { + cos: Float64Array; + sin: Float64Array; +} + +const twiddleCache = new Map(); + +function twiddles(n: number): Twiddle { + const hit = twiddleCache.get(n); + if (hit) return hit; + const cos = new Float64Array(n * n); + const sin = new Float64Array(n * n); + for (let k = 0; k < n; k++) { + for (let x = 0; x < n; x++) { + const angle = (-2 * Math.PI * k * x) / n; + cos[k * n + x] = Math.cos(angle); + sin[k * n + x] = Math.sin(angle); + } + } + const t = { cos, sin }; + twiddleCache.set(n, t); + return t; +} + +/** 2D DFT by two passes of 1D DFTs (rows, then columns). */ +function dft2(data: Float32Array, n: number): { re: Float64Array; im: Float64Array } { + const { cos, sin } = twiddles(n); + const rowRe = new Float64Array(n * n); + const rowIm = new Float64Array(n * n); + + for (let r = 0; r < n; r++) { + for (let kx = 0; kx < n; kx++) { + let re = 0; + let im = 0; + for (let c = 0; c < n; c++) { + const v = data[r * n + c]; + re += v * cos[kx * n + c]; + im += v * sin[kx * n + c]; + } + rowRe[r * n + kx] = re; + rowIm[r * n + kx] = im; + } + } + + const re = new Float64Array(n * n); + const im = new Float64Array(n * n); + for (let kx = 0; kx < n; kx++) { + for (let ky = 0; ky < n; ky++) { + let sre = 0; + let sim = 0; + for (let r = 0; r < n; r++) { + const ar = rowRe[r * n + kx]; + const ai = rowIm[r * n + kx]; + const c = cos[ky * n + r]; + const s = sin[ky * n + r]; + // (ar + i·ai)(c + i·s) + sre += ar * c - ai * s; + sim += ar * s + ai * c; + } + re[ky * n + kx] = sre; + im[ky * n + kx] = sim; + } + } + return { re, im }; +} + +export interface RadialSpectrum { + /** Spatial frequency in cycles per image, starting at 1 (DC excluded). */ + k: number[]; + /** Mean power in each frequency ring. */ + power: number[]; +} + +/** + * Mean power per frequency ring. DC (k = 0) is dropped: it is just the mean + * brightness, it dwarfs everything else, and it says nothing about detail. + */ +export function radialPowerSpectrum(data: Float32Array, n: number): RadialSpectrum { + const { re, im } = dft2(data, n); + const maxK = Math.floor(n / 2); + const sums = new Float64Array(maxK + 1); + const counts = new Float64Array(maxK + 1); + + for (let ky = 0; ky < n; ky++) { + // Fold the upper half of each axis onto negative frequencies. + const fy = ky <= n / 2 ? ky : ky - n; + for (let kx = 0; kx < n; kx++) { + const fx = kx <= n / 2 ? kx : kx - n; + const ring = Math.round(Math.hypot(fx, fy)); + if (ring < 1 || ring > maxK) continue; + const p = re[ky * n + kx] ** 2 + im[ky * n + kx] ** 2; + sums[ring] += p; + counts[ring] += 1; + } + } + + const k: number[] = []; + const power: number[] = []; + for (let ring = 1; ring <= maxK; ring++) { + if (counts[ring] === 0) continue; + k.push(ring); + power.push(sums[ring] / counts[ring]); + } + return { k, power }; +} diff --git a/src/components/visualizations/lib/sprites.ts b/src/components/visualizations/lib/sprites.ts new file mode 100644 index 0000000..bf3ea2b --- /dev/null +++ b/src/components/visualizations/lib/sprites.ts @@ -0,0 +1,198 @@ +/** + * The training set for every diffusion widget on the track: eight 24×24 + * grayscale glyphs, generated procedurally so there are no image assets to + * host and every reader sees byte-identical data. + * + * Stored in the DDPM convention — values in [-1, 1], background -1, ink +1 — + * so the forward process can add unit-variance noise without rescaling. + * + * The shapes are chosen to overlap heavily in the middle of the frame. That + * is deliberate: their pixelwise mean is a centred blur, which is exactly + * what a one-step denoiser returns, and that blur is the point of + * `one-step-vs-many`. + */ + +export const SPRITE_SIZE = 24; +export const SPRITE_N = SPRITE_SIZE * SPRITE_SIZE; + +/** Coverage test in normalized coords, both axes spanning [-1, 1]. */ +type Inside = (x: number, y: number) => boolean; + +/** Subpixel samples per axis — cheap antialiasing so edges survive noising. */ +const SUPERSAMPLE = 4; + +const SHAPES: { name: string; inside: Inside }[] = [ + { + name: "disc", + inside: (x, y) => Math.hypot(x, y) < 0.62, + }, + { + name: "ring", + inside: (x, y) => { + const r = Math.hypot(x, y); + return r > 0.38 && r < 0.68; + }, + }, + { + name: "cross", + inside: (x, y) => + Math.max(Math.abs(x), Math.abs(y)) < 0.72 && + (Math.abs(x) < 0.19 || Math.abs(y) < 0.19), + }, + { + name: "triangle", + inside: (x, y) => y > -0.6 && y < 0.68 - 1.75 * Math.abs(x), + }, + { + name: "frame", + inside: (x, y) => { + const m = Math.max(Math.abs(x), Math.abs(y)); + return m > 0.42 && m < 0.68; + }, + }, + { + name: "slash", + inside: (x, y) => + Math.abs(x + y) < 0.26 && Math.abs(x) < 0.78 && Math.abs(y) < 0.78, + }, + { + name: "chevron", + inside: (x, y) => { + const arm = Math.abs(Math.abs(x) - (y + 0.42)) < 0.24; + return arm && Math.abs(x) < 0.72 && y > -0.62 && y < 0.62; + }, + }, + { + name: "bars", + inside: (x, y) => { + if (Math.abs(x) > 0.7) return false; + const band = Math.abs((((y + 1) * 3) % 1) - 0.5); + return band < 0.22 && Math.abs(y) < 0.78; + }, + }, +]; + +/** Rigid placement of a shape inside the frame. */ +export interface Pose { + dx?: number; + dy?: number; + scale?: number; + rot?: number; +} + +function rasterize(inside: Inside, pose: Pose = {}): Float32Array { + const { dx = 0, dy = 0, scale = 1, rot = 0 } = pose; + const cos = Math.cos(-rot); + const sin = Math.sin(-rot); + const out = new Float32Array(SPRITE_N); + const step = 2 / SPRITE_SIZE; + const sub = step / SUPERSAMPLE; + for (let row = 0; row < SPRITE_SIZE; row++) { + for (let col = 0; col < SPRITE_SIZE; col++) { + let hits = 0; + for (let sy = 0; sy < SUPERSAMPLE; sy++) { + for (let sx = 0; sx < SUPERSAMPLE; sx++) { + // Pixel centre in [-1,1], then walk the subpixel grid. + const px = -1 + col * step + sub * (sx + 0.5); + const py = 1 - row * step - sub * (sy + 0.5); + // Undo the pose, then test against the canonical shape. + const tx = (px - dx) / scale; + const ty = (py - dy) / scale; + if (inside(tx * cos - ty * sin, tx * sin + ty * cos)) hits++; + } + } + const cover = hits / (SUPERSAMPLE * SUPERSAMPLE); + out[row * SPRITE_SIZE + col] = cover * 2 - 1; + } + } + return out; +} + +export interface Sprite { + name: string; + data: Float32Array; +} + +/** The eight-glyph training set. Built once at module load. */ +export const SPRITES: Sprite[] = SHAPES.map(({ name, inside }) => ({ + name, + data: rasterize(inside), +})); + +export const SHAPE_NAMES: string[] = SHAPES.map((s) => s.name); + +/** + * Render one shape at an arbitrary pose. + * + * This is the honest "latent space" for the interpolation demo: the four pose + * numbers are the actual generative parameters these images were drawn from, + * so walking a straight line through them produces valid images at every + * point by construction. Walking a straight line through the 576 *pixels* + * between the same two endpoints does not, and the gap between those two + * paths is what a manifold is. + */ +export function poseSprite(shapeIdx: number, pose: Pose): Float32Array { + return rasterize(SHAPES[shapeIdx % SHAPES.length].inside, pose); +} + +/** + * A *rich* training set: every shape at many positions, sizes and angles. + * + * This is not decoration. With only the eight canonical glyphs, the exact + * posterior collapses onto the right one within about four reverse steps — + * measured, not assumed — so a step-count demo built on them would show + * nothing between 4 and 1000 and any caption claiming otherwise would be + * false. Eight discrete points is simply not a hard distribution. + * + * Posing each shape continuously makes neighbouring training images genuinely + * close together, so a coarse step lands *between* them and produces a + * visible blend rather than a clean sample. That blend is the phenomenon the + * whole step-count argument is about, and it only exists if the distribution + * is dense enough to fall between. + */ +export function makeDataset(count: number, seed = 1): Float32Array[] { + // Local LCG — keeps this module free of imports and reproducible. + let s = (seed * 2654435761) >>> 0; + const rand = () => { + s = (s * 1664525 + 1013904223) >>> 0; + return s / 4294967296; + }; + const out: Float32Array[] = []; + for (let i = 0; i < count; i++) { + const shape = SHAPES[i % SHAPES.length]; + out.push( + rasterize(shape.inside, { + dx: (rand() - 0.5) * 0.44, + dy: (rand() - 0.5) * 0.44, + scale: 0.78 + rand() * 0.42, + rot: (rand() - 0.5) * 0.9, + }), + ); + } + return out; +} + +/** The same dataset, carrying the shape each image was drawn from — the + * "prompt" for the class-conditional and guidance widgets. */ +export function makeLabeledDataset( + count: number, + seed = 1, +): { data: Float32Array[]; labels: Uint8Array } { + const data = makeDataset(count, seed); + const labels = new Uint8Array(count); + for (let i = 0; i < count; i++) labels[i] = i % SHAPES.length; + return { data, labels }; +} + +/** Pixelwise mean of a set — what a denoiser returns when everything is + * equally plausible, i.e. the output of one giant reverse step. */ +export function meanImage(set: Float32Array[]): Float32Array { + const mean = new Float32Array(SPRITE_N); + for (const d of set) { + for (let i = 0; i < SPRITE_N; i++) mean[i] += d[i]; + } + for (let i = 0; i < SPRITE_N; i++) mean[i] /= set.length; + return mean; +} + +export const SPRITE_MEAN: Float32Array = meanImage(SPRITES.map((s) => s.data)); diff --git a/src/components/visualizations/lib/tinynn.ts b/src/components/visualizations/lib/tinynn.ts new file mode 100644 index 0000000..0cd5b6d --- /dev/null +++ b/src/components/visualizations/lib/tinynn.ts @@ -0,0 +1,172 @@ +/** + * A very small MLP with hand-written backprop and Adam, sized for the 2D toy + * problems on the diffusion track. + * + * This exists so the GAN page can train an *actual* adversarial pair in the + * browser rather than animate a scripted impression of one. The networks are + * 2→16→16→k, so a batch of 64 is a few tens of thousands of multiplies — + * cheap enough to run several optimizer steps per animation frame. + * + * Adam defaults to β₁ = 0.5, the DCGAN setting, because the whole point of + * the page is to reproduce GAN training dynamics faithfully, including the + * ones that go wrong. + */ + +export type Activation = "tanh" | "lrelu" | "none"; + +function actFwd(kind: Activation, v: number): number { + if (kind === "tanh") return Math.tanh(v); + if (kind === "lrelu") return v > 0 ? v : 0.2 * v; + return v; +} + +/** Derivative expressed in terms of the *output* of the activation. */ +function actBwd(kind: Activation, y: number): number { + if (kind === "tanh") return 1 - y * y; + if (kind === "lrelu") return y > 0 ? 1 : 0.2; + return 1; +} + +interface Layer { + nIn: number; + nOut: number; + w: Float64Array; + b: Float64Array; + gw: Float64Array; + gb: Float64Array; + mw: Float64Array; + vw: Float64Array; + mb: Float64Array; + vb: Float64Array; + act: Activation; + /** Per-sample caches from the last forward pass. */ + inputs: Float64Array[]; + outputs: Float64Array[]; +} + +export class MLP { + layers: Layer[] = []; + private t = 0; + + constructor(sizes: number[], acts: Activation[], rng: () => number) { + for (let i = 0; i < sizes.length - 1; i++) { + const nIn = sizes[i]; + const nOut = sizes[i + 1]; + const w = new Float64Array(nIn * nOut); + // He-ish init; the exact constant does not matter at this size. + const scale = Math.sqrt(2 / nIn); + for (let k = 0; k < w.length; k++) { + // Box–Muller from the supplied rng keeps runs reproducible. + const u = Math.max(rng(), 1e-12); + const v = rng(); + w[k] = Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v) * scale; + } + this.layers.push({ + nIn, + nOut, + w, + b: new Float64Array(nOut), + gw: new Float64Array(nIn * nOut), + gb: new Float64Array(nOut), + mw: new Float64Array(nIn * nOut), + vw: new Float64Array(nIn * nOut), + mb: new Float64Array(nOut), + vb: new Float64Array(nOut), + act: acts[i], + inputs: [], + outputs: [], + }); + } + } + + /** Forward a whole batch, caching activations for the backward pass. */ + forward(batch: Float64Array[]): Float64Array[] { + let cur = batch; + for (const L of this.layers) { + L.inputs = cur; + const out: Float64Array[] = []; + for (const x of cur) { + const y = new Float64Array(L.nOut); + for (let o = 0; o < L.nOut; o++) { + let s = L.b[o]; + for (let i = 0; i < L.nIn; i++) s += x[i] * L.w[i * L.nOut + o]; + y[o] = actFwd(L.act, s); + } + out.push(y); + } + L.outputs = out; + cur = out; + } + return cur; + } + + /** + * Backprop a batch of output gradients. Accumulates parameter gradients and + * returns the gradient with respect to the batch input — which is what the + * generator needs from the discriminator. + */ + backward(gradOut: Float64Array[]): Float64Array[] { + let g = gradOut; + for (let li = this.layers.length - 1; li >= 0; li--) { + const L = this.layers[li]; + const gIn: Float64Array[] = []; + for (let s = 0; s < g.length; s++) { + const gy = g[s]; + const y = L.outputs[s]; + const x = L.inputs[s]; + // Fold the activation derivative in first. + const gz = new Float64Array(L.nOut); + for (let o = 0; o < L.nOut; o++) gz[o] = gy[o] * actBwd(L.act, y[o]); + for (let o = 0; o < L.nOut; o++) L.gb[o] += gz[o]; + const gx = new Float64Array(L.nIn); + for (let i = 0; i < L.nIn; i++) { + const xi = x[i]; + let acc = 0; + for (let o = 0; o < L.nOut; o++) { + L.gw[i * L.nOut + o] += xi * gz[o]; + acc += L.w[i * L.nOut + o] * gz[o]; + } + gx[i] = acc; + } + gIn.push(gx); + } + g = gIn; + } + return g; + } + + zeroGrad(): void { + for (const L of this.layers) { + L.gw.fill(0); + L.gb.fill(0); + } + } + + /** Adam. `scale` divides the accumulated gradients (i.e. the batch size). */ + step(lr: number, scale = 1, b1 = 0.5, b2 = 0.999): void { + this.t++; + const c1 = 1 - b1 ** this.t; + const c2 = 1 - b2 ** this.t; + for (const L of this.layers) { + const upd = ( + p: Float64Array, + g: Float64Array, + m: Float64Array, + v: Float64Array, + ) => { + for (let k = 0; k < p.length; k++) { + const gk = g[k] / scale; + m[k] = b1 * m[k] + (1 - b1) * gk; + v[k] = b2 * v[k] + (1 - b2) * gk * gk; + p[k] -= (lr * (m[k] / c1)) / (Math.sqrt(v[k] / c2) + 1e-8); + } + }; + upd(L.w, L.gw, L.mw, L.vw); + upd(L.b, L.gb, L.mb, L.vb); + } + } +} + +export function sigmoid(v: number): number { + return v >= 0 ? 1 / (1 + Math.exp(-v)) : Math.exp(v) / (1 + Math.exp(v)); +} diff --git a/src/content/docs/diffusion/01-the-problem/gan.mdx b/src/content/docs/diffusion/01-the-problem/gan.mdx new file mode 100644 index 0000000..942ae03 --- /dev/null +++ b/src/content/docs/diffusion/01-the-problem/gan.mdx @@ -0,0 +1,75 @@ +--- +title: Before Diffusion, the GAN +description: A counterfeiter, a detective, and the most exciting idea in generative modelling for six years — running live, including the part where identical settings give different answers +sidebar: + order: 2 +--- + +import GANDuel from '../../../../components/visualizations/GANDuel'; + +We need a network that maps simple noise onto the image manifold. The obstacle is training it: we cannot write down a loss, because we have no formula for "how much does this look like a real image." + +In 2014 Ian Goodfellow proposed skipping the formula entirely. **If you cannot write the loss, learn it.** Train a second network whose only job is to tell real images from generated ones, and use *its* opinion as the first network's loss. The generator is a counterfeiter, the discriminator is a detective, and they improve by trying to beat each other. + +It is one of the genuinely beautiful ideas in machine learning, and for about six years it was the only thing that worked. + +## The Game + +Two networks, opposite objectives: + +$$ +\min_G \max_D \; \mathbb{E}_{x \sim p_\text{data}}[\log D(x)] + \mathbb{E}_{z \sim \mathcal{N}}[\log(1 - D(G(z)))] +$$ + +$D$ maximises: get real images right, get fakes right. $G$ minimises the second term: make $D$ wrong about fakes. At the theoretical optimum $D$ can do no better than a coin flip, and $G$ has matched the data distribution exactly. + +The appeal was not only elegance. A GAN generates in **one forward pass** — noise in, image out — which was and remains far faster than anything on the diffusion side. And because nothing forces the generator to explain the data, only to fool a critic, GAN samples were *sharp* at a time when everything else was blurry. + +Here is one, actually training. Two 2→16→16 networks, real backprop, real Adam: + + + +
+**What it models.** A genuine GAN on the standard eight-Gaussians benchmark — nothing scripted, every blue dot is the current generator's real output. Grey is real data, the eight circles are the target modes. "Modes covered" counts how many hold more than 2% of the generator's mass, so a mode with a token handful of samples does not count. "Mass off the data" is the fraction of samples landing outside every mode. + +**Knobs.** Discriminator strength switches between one update per generator step and four stronger ones. The loss switch chooses the original 2014 minimax objective or the non-saturating variant everyone actually uses. Reset draws a new seed and starts over. + +**Try this.** Press Run at the defaults and watch it work — the blue cloud usually finds all eight modes within a couple of thousand iterations, which is the point: **GANs are not broken, they mostly work.** Then press Reset and run it again. And again. Over eight seeds at these exact settings, the measured outcome ranges from **3 modes covered with 79% of mass off the data** to **8 modes with 11% off**, with nothing changed but the random seed. Now watch the two loss curves while you do it. They look about the same on the good runs and the bad ones. +
+ +## What the Loss Curves Are Not Telling You + +That last observation is the one to carry forward, and it is worth being precise about why it happens rather than filing it under "GANs are finicky." + +In ordinary supervised learning the loss is a **progress measure**. It goes down, the model is better; it plateaus, you are done. You can watch a number and know how the run is going. + +A GAN loss is not a progress measure, and cannot be. It is the score in a game between two players who are both still learning. If the discriminator loss rises, that might mean the generator improved — or that the discriminator got worse. If it falls, the generator may be failing, or the discriminator may have found a new tell that the generator will shortly patch. **The number moves for two unrelated reasons and gives you no way to separate them.** A GAN at equilibrium and a GAN whose two networks are both incompetent produce similar-looking curves. + +This is not a subtlety anyone missed; it is the central motivation of the Wasserstein GAN paper, whose selling point was a loss that actually correlates with sample quality. It also explains why GAN papers of that era are full of grids of hand-picked samples: looking at the pictures was, genuinely, the evaluation method. + +:::note[The discriminator is not the enemy] +Folk wisdom says a GAN fails when the discriminator gets *too good* — it wins, gradients vanish, the generator starves. That story is about the **original 2014 minimax loss**, where the generator's gradient really does vanish once $D$ is confident. + +It is not what this widget measures. Switching to four stronger discriminator steps *tightens* the spread rather than causing collapse: 6–8 modes covered and 9–15% of mass off the data, against 3–8 modes and 11–79% with the weak discriminator. With the non-saturating loss — which is what Goodfellow proposed as the fix, and what everyone uses — a weak discriminator is the greater danger, because a critic that cannot tell real from fake gives the generator no signal worth following. +::: + +## What It Cost + +Everything above is the *good* version of the story. What sank GANs at scale was not any single failure but the accumulation: + +**They are unstable in a way you cannot monitor.** Not "sometimes need tuning" — you cannot tell from the training curves whether the run you are watching is one of the good ones. + +**They drop parts of the data.** Mode collapse, in the small: the generator finds a region the discriminator is currently bad at and piles its mass there. In the large: a face model that never generates certain kinds of faces, discovered by a user rather than a metric. + +**Adding capacity does not reliably help.** Bigger models made classifiers better on a predictable curve. Bigger GANs got harder to balance. That mattered enormously in the years when scaling turned out to be the field's main lever. + +None of that is fatal on its own; a great deal of engineering went into managing all three. What ended the era was something else arriving that did not need managing — where the loss is a plain regression, the target is known exactly, and the same run twice gives you the same model. + +The [next page](../why-gans-broke/) takes the deepest of these — dropping parts of the data — and shows it is not a training bug at all. It is what the objective *asks for*. + +## References + +- Goodfellow et al., *Generative Adversarial Networks.* NeurIPS 2014. [arXiv:1406.2661](https://arxiv.org/abs/1406.2661) — the original game, and the non-saturating loss fix in §3. +- Arjovsky, Chintala & Bottou, *Wasserstein GAN.* ICML 2017. [arXiv:1701.07875](https://arxiv.org/abs/1701.07875) — the argument that the GAN loss does not track sample quality, and a loss that does. +- Metz et al., *Unrolled Generative Adversarial Networks.* ICLR 2017. [arXiv:1611.02163](https://arxiv.org/abs/1611.02163) — the eight-Gaussians benchmark used here, and mode collapse as cycling. +- Dhariwal & Nichol, *Diffusion Models Beat GANs on Image Synthesis.* NeurIPS 2021. [arXiv:2105.05233](https://arxiv.org/abs/2105.05233) — the paper whose title marks the end of the period. diff --git a/src/content/docs/diffusion/01-the-problem/index.mdx b/src/content/docs/diffusion/01-the-problem/index.mdx new file mode 100644 index 0000000..d47f21d --- /dev/null +++ b/src/content/docs/diffusion/01-the-problem/index.mdx @@ -0,0 +1,56 @@ +--- +title: What Generation Actually Asks For +description: Why an image is a point in a 576-dimensional space, why almost none of that space is images, and why that single fact shapes every generative model +sidebar: + order: 1 +--- + +import ManifoldSlice from '../../../../components/visualizations/ManifoldSlice'; + +Ask someone what it would take to generate an image and the honest first answer is: pick some numbers. A 24×24 grayscale picture is 576 numbers. A 512×512 colour photograph is 786,432. Generation is choosing values for all of them at once. + +Put that way it sounds like the problem should already be solved, because picking numbers is easy. The difficulty is entirely in *which* numbers, and the shape of that difficulty is the thing worth internalising before any method is discussed. It is the reason generative modelling is hard, and it is the reason all the methods that work look strange. + +## Almost Nothing Is an Image + +Take the 576-number version and start choosing at random. You will get static — every time, without exception, for as long as you care to run it. Not "usually static", not "static with occasional faint structure". The set of real images is so vanishingly thin inside that space that random search is not a slow method for finding one, it is not a method at all. + +The images that mean something occupy a **manifold**: a low-dimensional, curved, connected surface inside the ambient space. Our sprites have 576 pixels but only about four real degrees of freedom — which shape, where, how big, at what angle. Everything else is determined. Real photographs are the same story with bigger numbers: millions of pixels, but far fewer independent knobs, because pixels in a photograph are ferociously correlated. Neighbouring pixels are usually similar; lighting is consistent; objects are opaque and have edges. + +So "generate an image" means: **produce a point on a surface you cannot write down, that occupies effectively none of the space it lives in, and that you only know through examples.** Every generative model in this track is an attempt to get onto that surface. + + + +
+**What it models.** Two routes between the same two images. The top strip walks a straight line through all 576 pixel values — the obvious thing, and the thing that fails. The bottom strip walks a straight line through the four pose parameters the images were actually generated from, re-rendering at every step. That second path is a genuine latent space rather than a stand-in: these sprites really are drawn from those four numbers. The readouts measure, for the current position, the distance to the nearest of 256 reference images. Read the *gap*, not the absolute number: the reference set is finite, so even a perfectly valid pose it happens not to contain scores around 0.2, while a pixel-space ghost scores around 0.53. + +**Knobs.** λ moves along both paths at once; the shape buttons change the endpoints. + +**Try this.** Set λ to the middle and compare the two rows. The pixel path shows a **double exposure** — both shapes faintly present, neither of them real, a picture no camera and no renderer would ever produce. The pose path shows a perfectly ordinary shape at every step. Both paths connect the same two endpoints, and both are straight lines; they just live in different spaces. Then look at the three random-pixel frames at the bottom, which is what the overwhelming majority of that 576-dimensional space actually looks like. +
+ +## The Straight Line Is the Whole Problem + +The two rows above are the same idea told twice. Average two images pixelwise and you get a ghost, because **the manifold is curved and the straight line between two points on it leaves it immediately**. Average their *poses* and you stay on it, because in that parameterization the manifold is flat by construction. + +This is why "just interpolate the training data" is not a generative model, and it is why the interesting question in this field has always been *what coordinates to work in*. A model that has learned good coordinates makes generation easy — sample coordinates, decode, done. A model working in raw pixels has to somehow avoid the overwhelming majority of its own output space. + +Every method in this track is a different answer to that: + +- **GANs** learn a decoder directly: a network mapping simple noise onto the manifold, trained by a second network whose job is to notice when it lands off. That is the [next page](../gan/). +- **VAEs** learn an encoder and a decoder together, and pay for it with blurry samples — a cost the [mode-coverage page](../why-gans-broke/) will explain in the same terms as everything else. +- **Diffusion** does something stranger. It refuses to learn coordinates at all, works in raw pixel space the whole way, and instead learns a *direction field* pointing back toward the manifold from every point outside it. That sounds much worse and turned out to be much better. + +## What Makes This Different From Classification + +One more framing, because it explains why generation stayed hard long after recognition got easy. + +A classifier maps a huge input to a small output — 786,432 numbers in, one label out. It is allowed to throw away almost everything, and mostly what it learns is *what to ignore*. Generation runs the other way: a handful of numbers in, 786,432 out, every one of which has to be right, and jointly right. Get the pixels individually plausible but jointly inconsistent and you have not made a slightly flawed image, you have made an obvious fake. + +That "jointly" is where the difficulty concentrates, and it is worth naming now because it is the same difficulty that will reappear as **mode collapse** in GANs, as **blur** in one-step diffusion, and — in a different track entirely — as the [mean-field trap](../../../specdec/03-parallel-drafting/dspark/#the-trap-a-product-of-marginals) in parallel token drafting. Three fields, three names, one problem: getting each part right individually is not the same as getting them right together. + +## References + +- Bengio, Courville & Vincent, *Representation Learning: A Review and New Perspectives.* IEEE TPAMI 2013. [arXiv:1206.5538](https://arxiv.org/abs/1206.5538) — the manifold hypothesis and why good coordinates are the whole game. +- Goodfellow, Bengio & Courville, *Deep Learning*, ch. 20 — generative models and the dimensionality argument sketched above. +- Fefferman, Mitter & Narayanan, *Testing the Manifold Hypothesis.* J. Amer. Math. Soc. 2016. [arXiv:1310.0425](https://arxiv.org/abs/1310.0425) — what it would take to actually verify the claim, rather than assume it. diff --git a/src/content/docs/diffusion/01-the-problem/why-gans-broke.mdx b/src/content/docs/diffusion/01-the-problem/why-gans-broke.mdx new file mode 100644 index 0000000..1432e4d --- /dev/null +++ b/src/content/docs/diffusion/01-the-problem/why-gans-broke.mdx @@ -0,0 +1,81 @@ +--- +title: Why the Game Was the Problem +description: Mode collapse is not a training bug — it is what the objective asks for. The measured difference between an objective that finds one answer and one that finds whichever answer it started nearest. +sidebar: + order: 3 +--- + +import ModeCoverage from '../../../../components/visualizations/ModeCoverage'; + +The previous page ended on an observation: the same GAN, the same hyperparameters, a different seed, and the outcome swings from covering all eight modes to covering three. The natural reading is that GAN training is delicate and needs better engineering. + +That reading is too kind to the engineering and too harsh on the engineers. **The instability is in the objective, not the optimiser**, and you can see it without any neural network at all. + +## An Experiment With No Networks In It + +Take a target distribution with three modes. Take a model too simple to fit it — a single Gaussian. It *must* fail; the only question is how. And "how" is decided entirely by the divergence you minimise. + +Three candidates: + +$$ +\underbrace{\mathrm{KL}(p \parallel q) = \int p \log \frac{p}{q}}_{\text{maximum likelihood}} +\qquad +\underbrace{\mathrm{KL}(q \parallel p) = \int q \log \frac{q}{p}}_{\text{reverse}} +\qquad +\underbrace{\mathrm{JS}(p, q)}_{\text{what the GAN minimises}} +$$ + +The asymmetry is everything, and it is visible directly in the integrals. Forward KL integrates **against $p$**: wherever the data has mass and the model does not, $\log(p/q)$ blows up. Missing a mode is unboundedly expensive. Reverse KL integrates **against $q$**: wherever the model has no mass, it contributes nothing at all, whatever the data is doing there. Ignoring a mode is free. + +And the GAN's objective, at an optimal discriminator, is the Jensen–Shannon divergence — which sits between the two, and behaves, as we are about to measure, like the mode-seeking one. + + + +
+**What it models.** One Gaussian fitted to a fixed three-mode target, three times over, under the three divergences — all from the same starting point, all by gradient descent. The objectives are computed by direct numerical integration on a 480-point grid, so these are the actual divergences rather than sampled estimates. Each readout counts how many of the three modes fall within two standard deviations of that fit. + +**Knobs.** The slider sets where all three fits begin. That is the entire experiment. + +**Try this.** Leave the start at 3.0 — near the right-hand mode — and press Run. Forward KL (green) spreads to cover everything; JS (amber) and reverse KL (red) settle onto the right-hand mode and stop, having achieved 1 of 3. Now drag the start to −3.0 and run it again. Forward KL goes to **exactly the same place**. The other two collapse onto the *left* mode instead. Move the start to 0.1 and they take the middle one. Measured across seven starting points, forward KL lands at μ ≈ 0.1, σ = 2.65, covering 3 of 3, **every single time**; JS and reverse KL cover 1 of 3 from −3.6, −3.0, 0.1, 3.0 and 3.6, and each time it is whichever mode they began beside. +
+ +## The Result, Stated Plainly + +**Maximum likelihood has one answer and finds it from anywhere.** The adversarial objective has many answers and takes whichever one it happened to start nearest. + +That is the seed variance from the previous page, reproduced with no networks, no Adam, no minibatch noise, and no discriminator. Two independent experiments, one cause. The GAN's 3-to-8-mode swing is not an optimisation artefact that better tuning would remove — it is a two-parameter optimisation problem faithfully finding a local optimum of an objective that has several, and a mode-dropping solution is a perfectly good local optimum that the objective declines to punish. + +This is what "mode collapse" actually means. Not a failure to converge — a **successful** convergence to something the loss is content with. + +## Three More Differences, Now That the First One Is Clear + +**The target moves.** A GAN generator does gradient descent on a loss that is itself being trained. The landscape it is descending is redrawn every step by an adversary. There is no fixed objective anywhere in the system, which is why there is no number you can watch. + +Diffusion's loss is a regression onto $\varepsilon$ — a target you generated yourself, that you therefore know exactly, and that will still be the same target tomorrow. If it goes down, the model is better. That is the entire difference, and it is worth more than any architectural trick. + +**The signal is far denser.** A discriminator returns one scalar per sample: *fake, 0.7 confidence*. That is the entire gradient signal for a whole image. Diffusion supervises **every pixel, at every noise level** — 576 numbers per training example on our sprites, millions on a real image, and a fresh $t$ drawn each time so one image yields effectively unlimited distinct training targets. Diffusion extracts vastly more supervision from the same dataset. + +**Nothing has to be balanced.** GAN training is a two-player game that only works while the players stay matched; every practical GAN paper is partly about maintaining that balance. Diffusion has one network and one loss. There is no equilibrium to lose. + +## The Trilemma, and Why Diffusion Chose Differently + +The tidiest summary of the pre-2020 landscape is the **generative learning trilemma**: high sample quality, mode coverage, and fast sampling — pick two. + +| | quality | coverage | speed | +|---|---|---|---| +| **GAN** | ✅ sharp | ❌ drops modes | ✅ one pass | +| **VAE / flows** | ❌ blurry | ✅ likelihood-trained | ✅ one pass | +| **Diffusion** | ✅ sharp | ✅ likelihood-ish | ❌ many passes | + +Read the VAE row alongside the widget and it stops being a separate fact. VAEs are trained on a likelihood bound — forward KL, mode-covering — so they cover the data and pay for it by spreading mass across regions between modes, which renders as blur. It is the green curve in the figure above: broad, honest, and not committed to anything. The GAN is the red one: sharp, committed, and quietly missing a third of the distribution. + +Diffusion's trade is the third row, and its bet was the right one. Sampling speed is an *engineering* problem — you can attack it with better solvers and distillation, and the field did, taking image models from 1000 steps to [one to four](../../02-diffusion/one-step-vs-many/). Mode coverage and training stability are *objective* problems. You cannot engineer your way out of a loss that is content to drop a third of your data. + +Which is the whole argument in one line: **diffusion gave up the thing that could be fixed later, and kept the things that could not.** + +## References + +- Goodfellow et al., *Generative Adversarial Networks.* NeurIPS 2014. [arXiv:1406.2661](https://arxiv.org/abs/1406.2661) — §4.1 derives the JS divergence at an optimal discriminator. +- Arjovsky & Bottou, *Towards Principled Methods for Training GANs.* ICLR 2017. [arXiv:1701.04862](https://arxiv.org/abs/1701.04862) — why JS is a poor thing to descend when the supports barely overlap. +- Minka, *Divergence Measures and Message Passing.* MSR-TR-2005-173 — the canonical treatment of mode-seeking versus mode-covering, and the figure this widget reproduces. +- Xiao, Kreis & Vahdat, *Tackling the Generative Learning Trilemma with Denoising Diffusion GANs.* ICLR 2022. [arXiv:2112.07804](https://arxiv.org/abs/2112.07804) — the trilemma framing and the table above. diff --git a/src/content/docs/diffusion/02-diffusion/forward-process.mdx b/src/content/docs/diffusion/02-diffusion/forward-process.mdx new file mode 100644 index 0000000..6c6bab8 --- /dev/null +++ b/src/content/docs/diffusion/02-diffusion/forward-process.mdx @@ -0,0 +1,75 @@ +--- +title: The Forward Process +description: Why the destruction step is Gaussian, why that buys a closed-form jump to any noise level, and why fine detail always dies before coarse structure +sidebar: + order: 2 +--- + +import ForwardNoise from '../../../../components/visualizations/ForwardNoise'; + +We are stuck with a distribution we cannot touch. $p(\text{image})$ is not something anyone can write down, normalize, or sample. But there is exactly one distribution in this business that we *can* sample, trivially, a billion times a second: the standard Gaussian. So the plan writes itself, and it is the only plan in generative modeling that does not require knowing $p$ at all — **build a path from the image distribution to the Gaussian, then learn to walk it backwards.** Everything else on this page is about making that path cheap enough to be useful. + +## Why Gaussian, Specifically + +The forward path is a chain of tiny corruptions. At each step we shrink the image slightly toward zero and mix in a little fresh noise: + +$$ +q(x_t \mid x_{t-1}) = \mathcal{N}\big(x_t;\ \sqrt{1-\beta_t}\,x_{t-1},\ \beta_t \mathbf{I}\big) +$$ + +You could destroy an image many other ways — blur it, mask it, quantize it. Gaussian noise is chosen for a property none of those have: **Gaussians are closed under addition.** Add two Gaussians and you get a Gaussian, with variances that simply sum. Chain a thousand of these steps and the composition collapses into a single one: + +$$ +x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon, +\qquad \varepsilon \sim \mathcal{N}(0, \mathbf{I}), +\qquad \bar\alpha_t = \prod_{i \le t}(1-\beta_i) +$$ + +Read that as an accounting identity and the whole design falls out of it. $\bar\alpha_t$ is how much of the original image is left; $1 - \bar\alpha_t$ is how much noise has taken its place. They sum to one, so the total variance never drifts. + +The payoff is not elegance, it is **cost**. To train on the noise level $t = 700$ you do not simulate seven hundred steps. You draw one $\varepsilon$, evaluate one line of arithmetic, and land exactly where a seven-hundred-step simulation would have put you — with the correct distribution, not an approximation of it. Training samples $t$ uniformly at random and jumps straight there every time. A forward process built from blurring or masking has no such closed form, and every training example would cost a full simulation. This one property is why the method is affordable. + +:::note[The schedule is a design choice, not a law] +$\beta_t$ is set by hand — how fast to destroy. Early work used a linear ramp; the cosine schedule used here (Nichol & Dhariwal) spends far more steps in the middle range where the image is half-visible, because that is where nearly all the learnable signal lives. Both ends are cheap: near $t=0$ there is nothing to do, and near $t=T$ there is nothing left to do it to. +::: + +## The Endpoint Is the Whole Point + +Push $\bar\alpha_t$ to zero and the identity above reads $x_T = \varepsilon$. Not "approximately noise" — the image term is gone, and what remains is a draw from $\mathcal{N}(0,\mathbf{I})$ that carries no information about where it started. Every image in the dataset, every image that could ever exist, funnels into the same featureless Gaussian. + +That sounds like pure loss, and forward it is. Backward it is the entire trick. The reverse process has to start *somewhere*, and "somewhere" has to be a distribution we can draw from without already knowing the answer. The forward process is engineered to end at the one distribution that qualifies. We destroy the data into a Gaussian because a Gaussian is the only thing we know how to conjure from nothing. + +Watch all three consequences at once — the image dissolving, the pixel histogram sliding onto the unit Gaussian, and the frequency content going out: + + + +
+**What it models.** One image being noised by the closed-form jump above, with $\varepsilon$ drawn once and held fixed so the slider mixes in *the same* noise at increasing strength rather than reshuffling it. Left: $x_0$ beside $x_t$. Middle: the histogram of pixel values against $\mathcal{N}(0,1)$ in red — at $t=0$ the pixels are bimodal, piled at ink and paper, which is emphatically not a Gaussian; by $t=T$ they have slid onto the red curve exactly. Right: power per spatial frequency, coarse structure on the left and fine detail on the right. The dashed gray line is the clean image's spectrum, green is what survives ($\bar\alpha$ times it), the flat red line is the noise floor $n^2(1-\bar\alpha)$, and blue is what is actually measured in $x_t$. Where green drops under red, detail is gone; the shaded region is everything already lost. + +**Knobs.** The $t$ slider is the noise level. Run sweeps it up and back down so you can watch the crossing move. The thumbnails swap which training image is being destroyed; Reset draws a fresh $\varepsilon$. + +**Try this.** Drag $t$ slowly from 0 and watch the red noise floor rise as a flat line while the green signal curve sinks as a whole — they cross on the *right* first, and the crossing sweeps leftward. That is the claim worth taking away: noise is flat across frequencies, real images are not, so detail is never lost uniformly. Fine texture dies at $t \approx 200$ while the silhouette is still standing at $t \approx 700$. Now watch the middle panel over the same range: the two spikes of the ink/paper histogram melt into a single bell. By $t = 1000$ the readout says 0% detail surviving and the histogram sits exactly on $\mathcal{N}(0,1)$ — which is precisely the state the reverse process will have to start from. +
+ +## Detail Dies First, and That Is Not an Accident + +The right-hand panel is doing more work than it looks. It explains something you have seen with your own eyes every time a diffusion model generates: the image does not fade in uniformly, it **resolves** — a blurry blob becomes a shape becomes a face becomes eyelashes. + +The reason is a mismatch between two spectra. Gaussian noise is *white*: equal power at every spatial frequency, a flat line. Natural images are the opposite — their power falls off roughly as $1/f$, with most energy in large, smooth regions and comparatively little in fine texture. Layer a flat noise floor under a falling signal curve and they must cross, and the crossing is always on the fine-detail side. + +So the noise level is not really a measure of "how corrupted" the image is. It is a **cutoff frequency**. Sander Dieleman puts the mechanism well: large features cover many pixels, so many pixels vote for them and the noise averages out; a fine detail spans a handful of pixels and gets outvoted immediately. Turning $t$ up slides that cutoff toward coarser and coarser structure until even the silhouette is outvoted. + +Run the reverse of that and you get, for free, exactly the behavior the models exhibit: the earliest reverse steps operate where only the coarsest frequencies are above water, so they can only commit to layout and gross shape. Detail is not being withheld for dramatic effect. It is genuinely not decidable yet, and the model recovers it in the same order the forward process took it, last-destroyed-first-restored. + +## What This Sets Up + +We now have a path with two known ends: real images at $t=0$, pure Gaussian noise at $t=T$, and a closed form for every point between. Training data is free — pick an image, pick a $t$, jump. + +What we do not have is any way back. The forward step is a known Gaussian, but the reverse step $q(x_{t-1} \mid x_t)$ requires knowing $p(\text{image})$ — the very thing we could not write down when we started. The next two pages are about the two ideas that break that circle: [what the network should predict](../predict-the-noise/), and [why it must be asked in many small steps rather than one large one](../one-step-vs-many/). + +## References + +- Sohl-Dickstein et al., *Deep Unsupervised Learning using Nonequilibrium Thermodynamics.* ICML 2015. [arXiv:1503.03585](https://arxiv.org/abs/1503.03585) — the forward/reverse framing, taken directly from nonequilibrium thermodynamics. +- Ho, Jain & Abbeel, *Denoising Diffusion Probabilistic Models.* NeurIPS 2020. [arXiv:2006.11239](https://arxiv.org/abs/2006.11239) — the closed-form jump and the $\bar\alpha$ notation used here. +- Nichol & Dhariwal, *Improved Denoising Diffusion Probabilistic Models.* ICML 2021. [arXiv:2102.09672](https://arxiv.org/abs/2102.09672) — the cosine schedule the widget uses. +- Dieleman, *Diffusion models are autoencoders* (2022). [sander.ai](https://sander.ai/2022/01/31/diffusion.html) — noise level as feature scale; the argument that larger features survive because more pixels vote for them. diff --git a/src/content/docs/diffusion/02-diffusion/index.mdx b/src/content/docs/diffusion/02-diffusion/index.mdx new file mode 100644 index 0000000..c9b7e7c --- /dev/null +++ b/src/content/docs/diffusion/02-diffusion/index.mdx @@ -0,0 +1,71 @@ +--- +title: Destroy It, Then Learn to Undo +description: Ink in water, the one physics demo that seems to prove diffusion is reversible, and what actually makes the reverse process possible +sidebar: + order: 1 +--- + +import MixingReversibility from '../../../../components/visualizations/MixingReversibility'; + +Drop ink into a glass of water and you already know the whole story. A dark thread uncoils, softens, spreads, and after a minute the glass is a uniform pale blue. Everyone has watched this. Nobody has ever watched the reverse. + +That asymmetry is not a quirk of ink. It is the second law of thermodynamics doing its most familiar trick: a system with structure drifts toward one without, because there are overwhelmingly more ways to be spread out than to be a thread. Show someone a video of ink gathering itself back into a drop and they will identify it as reversed footage instantly and without effort, because the arrow of time is the most legible thing in the frame. + +So the founding move of this entire field looks, at first, like a joke. In 2015 Sohl-Dickstein and colleagues wrote down the ink-in-water process formally — a diffusion, in the literal physics sense, from their paper's title onward — and then proposed to **run it backwards on purpose**. Destroy an image gradually into structureless noise, learn the reverse of each small step, and then start from noise you generated yourself and walk back up the gradient of entropy into an image that never existed. + +The obvious objection is the one you should have. The forward direction is easy because it is what physics does anyway. Where does the reverse come from? + +## The Demo That Seems to Answer This, and Doesn't + +There is a famous physics demonstration that looks like it settles the question. Coloured dye is injected into corn syrup held between two concentric cylinders. Turn the crank and the dye smears into long spirals until the vessel looks uniformly, hopelessly mixed. Turn the crank back the same number of times and the dye **reassembles** into the original drops, sharp as they started. + +It is a genuinely startling thing to watch, and it is the single most misused analogy in explanations of diffusion models. Here is the same experiment, with the one knob that matters: + + + +
+**What it models.** Dye in an annular vessel under Taylor–Couette flow. Both media get exactly the same advection — the angular velocity profile $\omega(r) = \Omega \frac{r_{\text{in}}^2}{r_{\text{out}}^2 - r_{\text{in}}^2}\big(\frac{r_{\text{out}}^2}{r^2} - 1\big)$, a closed form, so cranking backwards undoes the shear exactly. The only difference between syrup and water is the molecular jitter added each tick, scaled as $\sqrt{\Delta t}$ the way real Brownian motion is. Run cranks four turns in and then four turns back out; displacement reports how far the dye sits from where it began. + +**Knobs.** One: molecular jitter. At 0 you have corn syrup — Reynolds number near zero, pure laminar flow, no diffusion at all. Turn it up and you have water. + +**Try this.** Leave jitter at 0 and press Run. Watch the blobs stretch into spirals that look completely mixed at four turns — then watch every strand walk backwards and reassemble into three clean drops, displacement returning to ≈ 0. Now set jitter to 0.5 and Run again. The forward half looks *identical*; the spirals form the same way. But on the way back the shear unwinds and the jitter does not, and the dye never comes home. That is the entire distinction the page is about, and it lives in one slider. +
+ +## What the Demo Actually Proves + +The syrup unmixes because **nothing was ever mixed.** At a Reynolds number near zero the flow is laminar and deterministic: every parcel of dye follows a smooth path, no two paths cross, and the state of the system at four turns contains every bit of information the state at zero turns did. It is scrambled, not destroyed. Reversing the crank applies the inverse map, and inverse maps are exact. + +Real diffusion is not that. Molecules take independent random walks, and a random walk has no inverse — running the clock backwards does not un-draw the dice. Information genuinely leaves the system, which is precisely why the ink never returns and why the second law is a law. + +So the honest reading of the demo is not "diffusion is reversible." It is the narrower and more useful claim: **"looks destroyed" and "is destroyed" are different claims, and your eye cannot tell them apart.** The four-turn syrup looks exactly as mixed as the four-turn water. One is recoverable and one is not, and no amount of staring at the vessel will tell you which. + +And our forward process is the *water*, not the syrup. Adding Gaussian noise really does destroy information; $x_T$ genuinely carries nothing about $x_0$. If reversibility were the requirement, the method would be dead on arrival. + +:::caution[Where this analogy is usually oversold] +You will find the corn-syrup demo cited as though it explains why diffusion models work — as if the reverse process were an unmixing. It is not. Our noise is irreversible in exactly the way the demo's syrup is not. The demo is worth its place because it separates *looking* mixed from *being* mixed, and because it shows you cannot judge which by inspection. Anything more than that is borrowed credibility. +::: + +## So Where Does the Reverse Actually Come From? + +Three ideas, none of which require undoing anything. + +**We do not have to invert a sample. We have to sample from a distribution.** This is the substitution that makes the whole field possible. Nobody is asking for *the* image that produced this particular noise — that question has no answer, and the second law says so. The task is to produce *some* image with the right statistics. Failure to recover the original is not an error; on almost every run it is the desired outcome. + +**The space of plausible answers is vanishingly small.** A 24×24 grayscale image has 576 free numbers, and almost every setting of them is static. Real images occupy a sliver of that space so thin as to be effectively invisible. So when noise destroys information, it destroys information *about which point on a very thin manifold you were on* — and the manifold itself is a colossal prior that was never in the noise to begin with. It is in the training data, and that is what the network is for. + +**Small steps make each reverse question answerable.** This is the technical core, and it gets [its own page](../one-step-vs-many/). For a diffusion with small enough steps, the reverse conditional has the same Gaussian form as the forward one — so the network never has to invert anything, only to predict a mean. Ask it to undo the whole process in one leap and it fails completely, and the way it fails is instructive enough to be worth watching. + +None of this is unmixing. It is closer to a different trick entirely: destroy the data along a path that ends somewhere you can start from, then learn the *statistics* of each small step of that path, and walk back up it generating a new sample rather than recovering an old one. + +## What's in This Chapter + +- **[The forward process](../forward-process/)** — why the destruction step is Gaussian, why that buys a closed-form jump to any noise level, and why fine detail always dies before coarse structure. +- **[Why predict the noise](../predict-the-noise/)** — what the network is actually asked for, and why the seemingly perverse choice of predicting the noise instead of the image is the one that works. +- **[Why one big step fails](../one-step-vs-many/)** — the mode-averaging collapse, watched live, and an honest reading of what the step count is really buying. + +## References + +- Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli, *Deep Unsupervised Learning using Nonequilibrium Thermodynamics.* ICML 2015. [arXiv:1503.03585](https://arxiv.org/abs/1503.03585) — the paper that took the physics literally. +- *Reversible Fluid Mixing*, Harvard Natural Sciences Lecture Demonstrations. [sciencedemonstrations.fas.harvard.edu](https://sciencedemonstrations.fas.harvard.edu/presentations/reversible-fluid-mixing) — the corn-syrup demonstration and the Stokes-flow explanation for it. +- Taylor, G. I., *Stability of a Viscous Liquid Contained between Two Rotating Cylinders.* Phil. Trans. R. Soc. A, 1923 — the flow the widget integrates. +- Smarter Every Day 217, *Laminar Flow / Unmixing Color Machine.* [youtube.com](https://www.youtube.com/watch?v=57IMufyoCnQ) — the demonstration performed with real syrup, if you want to see it done in a physical vessel. diff --git a/src/content/docs/diffusion/02-diffusion/one-step-vs-many.mdx b/src/content/docs/diffusion/02-diffusion/one-step-vs-many.mdx new file mode 100644 index 0000000..7b03d1f --- /dev/null +++ b/src/content/docs/diffusion/02-diffusion/one-step-vs-many.mdx @@ -0,0 +1,76 @@ +--- +title: Why One Big Step Fails +description: A single jump from noise to image returns the average of every answer at once — and why the step counts real systems use are about approximation, not arithmetic +sidebar: + order: 4 +--- + +import StepBudget from '../../../../components/visualizations/StepBudget'; + +We have a forward process that ends in a Gaussian and a network that predicts the noise. The obvious question is why we cannot simply run it once. Draw $x_T \sim \mathcal{N}(0,\mathbf{I})$, ask the network for the noise, subtract it, and collect the image. One forward pass — as fast as a GAN, with none of the adversarial machinery. + +Try it and you get a gray smudge. Not a bad image: **the average of all of them**, every time, no matter what noise you start from. The reason is the most important idea on this page, and it is not a limitation of neural networks. + +## The Model Is Not Allowed to Pick + +Recall what the network is trained to output. It is fit with a squared-error loss, and squared error has exactly one minimizer: the **conditional mean**. Whatever the network is asked, it answers with the average of every value consistent with its input. That was harmless everywhere it appeared before now, because it was what we wanted. + +At $t = T$ it stops being harmless. The input is pure noise and carries no information about the destination, so *every* image in the training distribution is equally consistent with it. The conditional mean of "every image at once" is the dataset average, and the dataset average of a pile of distinct shapes is a centered blur. The network is not failing. It is answering correctly, and the correct answer is useless. + +$$ +\mathbb{E}[x_0 \mid x_T] = \sum_i p(x^{(i)} \mid x_T)\, x^{(i)} \;\longrightarrow\; \frac{1}{N}\sum_i x^{(i)} \quad\text{as } \bar\alpha_T \to 0 +$$ + +This is the same failure the [mean-field trap](../../../specdec/03-parallel-drafting/dspark/#the-trap-a-product-of-marginals) describes for parallel token drafting, and it is worth seeing that it is literally the same failure. There, a drafter samples each position from its marginal and lands between two coherent futures, producing `np.tensor` — text no model would ever write. Here, a one-step sampler averages over every plausible image and lands between them, producing a shape no dataset ever contained. In both cases the individual quantity is computed *correctly*; the error is using an average where a commitment was required. + + + +
+**What it models.** Real reverse sampling, run at a step budget you choose, against a training set of 128 shapes at random positions, sizes and angles. The denoiser is not a neural network and not an approximation — with a finite training set the posterior over "which image is this?" is a softmax and its mean is a closed form, so this is the *optimal* denoiser, computed exactly. Left panel: $x_t$, the chain's current state, beside $\hat{x}_0$, its best guess at the clean image. Right: the largest posterior weights, with the red dashed line marking where they would sit if all 128 were equally likely, and the entropy of that posterior across the run — 7 bits is total ignorance, 0 bits is certainty. + +**Knobs.** T is the step budget. The sampler switch is η: DDPM injects fresh noise at every step, DDIM is fully deterministic — with an exact denoiser you can watch both land in the same place. + +**Try this.** Start at T = 1 and look before you step: the posterior bars sit flat on the red line, entropy reads the full 7 bits, and $\hat{x}_0$ is a symmetric smudge — the average of all 128 shapes. Press Step and that smudge *is* the finished sample. Now switch to T = 2 and step twice: the first step commits to a region of the dataset, the second resolves it, and a real shape drops out. Watch the entropy trace fall from 7 bits toward 0 as it goes — that fall is the generation. Then walk T up through 4, 16 and 200 and watch the improvement stop: by T = 4 the output is already a clean shape, and everything past that changes nothing. The next section is about why. +
+ +## The Fix, and Why It Works + +The escape is to never ask the impossible question. Instead of jumping from $t = T$ to $t = 0$, take a step so small that the posterior barely has room to be ambiguous. + +The formal statement behind this is due to Feller: for a diffusion with sufficiently small steps, **the reverse conditional has the same functional form as the forward one.** If $q(x_t \mid x_{t-1})$ is Gaussian and $\beta_t$ is small, then $q(x_{t-1} \mid x_t)$ is Gaussian too. That is the entire licence for the method. A Gaussian is fixed by its mean, the mean is what squared-error training gives you for free, and so a network that can only ever produce conditional means is nevertheless exactly the right tool — provided you only ever ask it about small steps. + +Take one big step and that guarantee evaporates. The true $q(x_0 \mid x_T)$ is as multimodal as the data itself, and a Gaussian fitted to it can only straddle the modes. Take a thousand small ones and every individual question is one the model can answer honestly. + +## An Honest Reading of the Step Count + +Here is where this widget will mislead you if you let it, so let us be direct about what it shows. + +Sweep T upward and the improvement stops almost immediately. Measured over sixteen seeds on exactly the configuration above, mean distance from the finished sample to the nearest real training image goes **0.50 at T=1, 0.063 at T=2, 0.0032 at T=4 — and then sits at 0.0032 through T=200.** By the blur measure it is the same story: 42% of pixels stranded in the midtones at T=1 against 4.9% for a real training image, and 6.6% by T=4. Enlarging the training set from 8 images to 1024 does not move those numbers. Four steps is converged. + +That is not what production systems do. Stable Diffusion shipped at 50 steps; the original DDPM paper used 1000. So either they were wasting 99.6% of their compute, or this toy is missing something. It is missing something, and naming it precisely is more useful than hiding it: + +**The step count buys accuracy in the denoiser, not correctness in the mathematics.** Our denoiser is exact — it has the training set in hand and computes the true posterior in closed form. Given an exact denoiser, the reverse process is a well-conditioned integration problem and a handful of steps solves it. A real denoiser is a network approximating that posterior over a distribution incomparably richer than 128 shapes, and it is wrong by a little every time it is called. Small steps keep each individual query inside the region where the network is accurate, and keep any single error from steering the whole trajectory. + +The best evidence that this reading is right is what happened next. If a thousand steps were mathematically required, no amount of engineering could have removed them. Instead, DDIM cut 1000 to 50 by changing only the sampler, and consistency and distillation methods have since pushed image models to **one to four steps** — landing almost exactly where this exact-denoiser toy says the arithmetic allows. The step count was always a tax on approximation error, and the field spent three years paying it down. + +:::note[What this widget cannot show you] +Because its denoiser is exact rather than learned, this simulator demonstrates the one-step failure faithfully and the thousand-step *necessity* not at all. The blur at T=1 is real and it is the point. The flatness after T=4 is an artifact of having an exact posterior, and it is reported here rather than papered over. +::: + +## The Other Thing This Denoiser Reveals + +Look at the gallery of finished samples. Every one is a training image, reproduced pixel for pixel — the measured distance is 0.0032, which is zero plus floating-point dust. That is not a bug in the sampler. It is what an *exact* empirical posterior must do: the distribution it was handed is 128 delta spikes, so the only thing it can possibly sample is one of those 128 spikes. It has perfectly memorized its training set and learned nothing else. + +The instinct is that this is a sharpness problem — blur the posterior a little and the model should start producing shapes between the training images. It does not work, and the reason is worth more than the fix would have been. Widening the kernel by a factor of eight, and separately flooring the denoiser's effective noise level, both leave the distance at **0.0032, unchanged to four decimals.** In 576 dimensions the squared distance from a noisy image to the *wrong* training image runs to the hundreds, so the softmax saturates no matter how much you divide it down. Smoothing delays the collapse onto a single training image; it never prevents it. + +Which is the actual argument for the neural network, and it is not about scale or speed. **Generalization is not a knob you can turn on an exact posterior — it requires a different function class.** A network cannot represent 128 delta spikes; it has finite capacity, so it is forced to interpolate between the training points, and that forced interpolation *is* the generalization. The approximation is not a compromise we accept for tractability. It is the only reason the model produces anything new. + +It is also, uncomfortably, why real diffusion models sometimes reproduce training data verbatim. Memorization is not an exotic failure bolted onto the method; it is the method's exact solution, and everything that makes a model creative is the network failing, productively, to reach it. + +## References + +- Sohl-Dickstein et al., *Deep Unsupervised Learning using Nonequilibrium Thermodynamics.* ICML 2015. [arXiv:1503.03585](https://arxiv.org/abs/1503.03585) — invokes Feller's result that small-step diffusions have reverse processes of the same functional form. +- Ho, Jain & Abbeel, *Denoising Diffusion Probabilistic Models.* NeurIPS 2020. [arXiv:2006.11239](https://arxiv.org/abs/2006.11239) — the 1000-step ancestral sampler. +- Song, Meng & Ermon, *Denoising Diffusion Implicit Models.* ICLR 2021. [arXiv:2010.02502](https://arxiv.org/abs/2010.02502) — the η parameterization used by the sampler switch; 1000 steps to 50 without retraining. +- Song et al., *Consistency Models.* ICML 2023. [arXiv:2303.01469](https://arxiv.org/abs/2303.01469) — one- and few-step generation, the evidence that the step count was approximation error rather than arithmetic. +- Carlini et al., *Extracting Training Data from Diffusion Models.* USENIX Security 2023. [arXiv:2301.13188](https://arxiv.org/abs/2301.13188) — memorization in deployed models. diff --git a/src/content/docs/diffusion/02-diffusion/predict-the-noise.mdx b/src/content/docs/diffusion/02-diffusion/predict-the-noise.mdx new file mode 100644 index 0000000..78f24b8 --- /dev/null +++ b/src/content/docs/diffusion/02-diffusion/predict-the-noise.mdx @@ -0,0 +1,75 @@ +--- +title: Why Predict the Noise +description: The model could predict the image, the noise, or the score — they are the same object. Why the perverse-looking choice is the one that works. +sidebar: + order: 3 +--- + +import ThreeTargets from '../../../../components/visualizations/ThreeTargets'; + +We want images. The network is handed a noisy image and asked for a number, and the sane request would seem to be *give me the clean image*. Instead, every diffusion model you have used asks for something that sounds like the opposite: **predict the noise that was added.** Then it subtracts that prediction and keeps what is left. + +This looks like a detour, and readers reasonably suspect it is a mathematical convenience with no real content. It is not. But the first thing to establish is that the choice cannot possibly be about information. + +## They Are the Same Object + +Given the noisy image $x_t$ and the noise level, any one of these three quantities determines the other two exactly: + +$$ +\hat{x}_0 = \frac{x_t - \sqrt{1-\bar\alpha_t}\,\hat\varepsilon}{\sqrt{\bar\alpha_t}}, +\qquad +\hat\varepsilon = \frac{x_t - \sqrt{\bar\alpha_t}\,\hat{x}_0}{\sqrt{1-\bar\alpha_t}}, +\qquad +\hat{s} = -\frac{\hat\varepsilon}{\sqrt{1-\bar\alpha_t}} +$$ + +Predicting the clean image, predicting the noise, and predicting the score $\nabla_{x_t}\log q(x_t)$ are one prediction in three costumes. A network that is perfect at any one of them is perfect at all three. Whatever the choice is buying, it is not information. + +What it changes is **conditioning**: how big the target is, whether its size depends on $t$, and — the part that actually decides the matter — how an error of a given size propagates into the thing we care about. + + + +
+**What it models.** One image at one noise level, showing all three candidate targets side by side, plus the quantity that separates them. The chart is the error amplification: if the network's prediction is off by one unit, how far off is the reconstructed clean image? For $x_0$-prediction that factor is 1 by definition. For $\varepsilon$-prediction it is $\sqrt{1-\bar\alpha}/\sqrt{\bar\alpha}$. For score-prediction it is $(1-\bar\alpha)/\sqrt{\bar\alpha}$. The dashed vertical line marks where the $t$ slider is. + +**Knobs.** The $t$ slider sweeps the noise level; the thumbnails change the image. + +**Try this.** Park $t$ in the middle, around 600, and note the three chips sit within about 1.5× of each other — through the mid-range no parameterization has a real advantage. Now drag $t$ down toward 0 and watch the green $\varepsilon$ curve dive while the amber $x_0$ line stays pinned at 1×: by $t = 100$ a unit error in the predicted noise costs **0.17** of that in the image, and by $t = 20$ only **0.042**. Then drag the other way, past 900, and watch the same green curve shoot *above* the amber one — ×6.4 at $t = 900$ and ×316 at $t = 999$. $\varepsilon$-prediction is not uniformly better; it is dramatically better exactly where it matters and worse where it does not. Also watch the RMS-score readout as you approach $t = 0$: it passes 150, which is why the purple curve wins on paper and loses in practice. +
+ +## The Three Reasons, In Order of Weight + +**1. The target has the same distribution at every noise level.** $\varepsilon \sim \mathcal{N}(0,\mathbf{I})$ — zero mean, unit variance, always. Not approximately, and not on average over the dataset: exactly, at every $t$, by construction. A single network with one set of weights has to cover the entire range from nearly-clean to pure static, and asking it for a target whose scale and shape never move is the difference between one regression problem and a thousand differently-scaled ones. This is the sense in which *the thing being predicted is normal, so it is easier to predict* — the network never has to learn what scale its own output should be, because the answer is always the same. + +Compare the alternatives. $x_0$ is bounded and fine, but its *relationship* to the input changes drastically with $t$. The score is worst: $\hat{s} = -\hat\varepsilon/\sqrt{1-\bar\alpha}$, so as $t \to 0$ the target diverges. Watch the RMS-score readout climb past 150 near $t = 1$. Regressing onto an unbounded target is a bad idea for reasons that have nothing to do with diffusion. + +**2. Predicting the noise is a free skip connection.** This is what the chart shows, and it is the most practical of the three. + +Near $t = 0$ the input is *almost* the answer. The correct behavior is to change almost nothing. Under $\varepsilon$-prediction, "change nothing" is spelled `output ≈ 0` — the easiest thing a neural network can say, and the thing an untrained one says by default. Under $x_0$-prediction, "change nothing" requires the network to re-emit all 576 pixels of the input faithfully through every layer, and any sloppiness shows up directly as a corrupted image. + +The chart makes the consequence exact. At $t = 100$ the amplification factor for $\varepsilon$-prediction has fallen to 0.17, at $t = 20$ to 0.042, and at $t = 1$ to 0.006: mistakes made in the low-noise regime are multiplied by nearly zero before they reach the image. And the low-noise regime is where the final quality of a sample is decided — it is the last thing that happens before you look at the output. $\varepsilon$-prediction puts the network's least reliable behavior exactly where it costs the least. + +**3. The geometry is asymmetric.** $\varepsilon$ is isotropic: it fills all 576 dimensions equally, with no structure to learn and no manifold to fall off. Images do the opposite — they sit on a thin, curved, complicated sliver of that space. Asking the network to *emit* a point on that manifold is a harder demand than asking it to identify a direction in the ambient space, and the second request has a well-conditioned answer everywhere. + +:::note[This is a choice, not a law] +$\varepsilon$-prediction wins overwhelmingly at low noise and actively *loses* at high noise — the green curve crosses above the amber one around $t \approx 750$ and reaches ×316 by $t = 999$. That crossing is why the picture kept evolving. **v-prediction** interpolates between $\varepsilon$ and $x_0$ so that neither end degenerates, and it is what most modern high-resolution and distilled models train on. The chart above is the argument for it in one picture: a target that is well-conditioned across the *whole* range beats one that is superb on half of it and unusable on the other. +::: + +## What Falls Out for Free + +Because $\hat\varepsilon$ and the score differ only by a constant factor at each noise level, a network trained with the plain noise-prediction loss + +$$ +L_\text{simple} = \mathbb{E}_{t,x_0,\varepsilon}\big\lVert \varepsilon - \varepsilon_\theta(\sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\varepsilon,\ t) \big\rVert^2 +$$ + +is, without anyone intending it, a **score estimator** — it has learned $\nabla_x \log q(x_t)$ up to scaling, at every noise level at once. That is what connects this construction to the separate line of work on score matching and Langevin dynamics, and it is why the two literatures turned out to be describing the same algorithm from opposite directions. + +It is also why the loss is so plain. No adversary, no discriminator, no equilibrium to balance — just a regression onto a target you generated yourself and therefore know exactly. That property is the one the next chapter will contrast against GANs, where the target is a moving object produced by a second network actively trying to defeat the first. + +## References + +- Ho, Jain & Abbeel, *Denoising Diffusion Probabilistic Models.* NeurIPS 2020. [arXiv:2006.11239](https://arxiv.org/abs/2006.11239) — the $\varepsilon$-parameterization and $L_\text{simple}$; reports noise prediction outperforming direct image prediction. +- Song & Ermon, *Generative Modeling by Estimating Gradients of the Data Distribution.* NeurIPS 2019. [arXiv:1907.05600](https://arxiv.org/abs/1907.05600) — the score-matching route to the same algorithm, and why a single noise scale fails. +- Salimans & Ho, *Progressive Distillation for Fast Sampling of Diffusion Models.* ICLR 2022. [arXiv:2202.00512](https://arxiv.org/abs/2202.00512) — introduces v-prediction and the conditioning argument for it. +- Dieleman, *Perspectives on diffusion* (2023). [sander.ai](https://sander.ai/2023/07/20/perspectives.html) — the equivalence of the parameterizations, and why an MSE-trained network necessarily outputs a conditional expectation. diff --git a/src/content/docs/diffusion/03-real-systems/conditioning.mdx b/src/content/docs/diffusion/03-real-systems/conditioning.mdx new file mode 100644 index 0000000..5d2ee7d --- /dev/null +++ b/src/content/docs/diffusion/03-real-systems/conditioning.mdx @@ -0,0 +1,69 @@ +--- +title: Getting a Prompt Into the Loop +description: How a sentence becomes a vector, how that vector reaches every denoising step, and why conditioning is a much smaller change than it looks +sidebar: + order: 2 +--- + +import GuidanceDial from '../../../../components/visualizations/GuidanceDial'; + +Everything built so far samples from $p(\text{image})$ — it produces *an* image from the training distribution, and you get no say in which. What people actually want is $p(\text{image} \mid \text{"a fox reading a newspaper"})$. + +The surprise is how little has to change. Conditioning is not a new algorithm; it is one extra argument threaded through the one the model already had. + +## The Denoiser Gets One More Input + +The network was $\varepsilon_\theta(x_t, t)$: given a noisy image and its noise level, predict the noise. Conditioning makes it $\varepsilon_\theta(x_t, t, c)$, where $c$ encodes the prompt. The loss is the same regression it always was: + +$$ +L = \mathbb{E}_{t, x_0, \varepsilon, c}\big\lVert \varepsilon - \varepsilon_\theta(\sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\varepsilon,\ t,\ c) \big\rVert^2 +$$ + +Train on (image, caption) pairs instead of images, hand the caption to the network alongside the noisy image, and the model learns to denoise *differently depending on what it was told*. There is no adversary to rebalance and no new objective — which is a direct dividend of [the loss being a plain regression](../../02-diffusion/predict-the-noise/). Conditioning a GAN was a research programme; conditioning a diffusion model is an extra function argument. + +## Turning a Sentence Into a Vector + +$c$ comes from a text encoder, and the reason a *contrastive* one is used is worth pinning down. + +CLIP is trained on hundreds of millions of image–caption pairs with one objective: put an image and its true caption close together in a shared space, and push mismatched pairs apart. It never generates anything. What it produces is a text embedding that already **lives in the same space as image content** — a geometry where "golden retriever" sits near pictures of golden retrievers. + +That is exactly the property the diffusion model needs, and the reason you cannot swap in any language model. A text encoder trained only on text has a geometry organised around *linguistic* similarity. CLIP's is organised around *visual* similarity, and the denoiser is a visual model. + +(The choice is not settled. Imagen found that a large frozen text-only encoder — T5-XXL — beat CLIP for prompt fidelity, on the argument that raw language understanding matters more than a shared embedding space once the model is big enough. Both approaches ship in production systems today.) + +## Cross-Attention: How It Reaches Every Step + +The prompt vector is injected into the denoiser through **cross-attention** layers spliced through the U-Net. At each layer, the image features form the queries; the prompt tokens form the keys and values. Every spatial position asks "which words are relevant to me?" and pulls in the answer. + +Two consequences worth holding on to: + +**Conditioning is spatial.** Different regions attend to different words. This is not decoration — it is why the model can put the fox on the left and the newspaper on the right, and it is the mechanism that attention-editing tools reach into when they let you change one object in a scene. + +**Conditioning applies at every step, not just the first.** The prompt is re-consulted at every one of the fifty reverse steps. That matters given [the coarse-to-fine ordering](../../02-diffusion/forward-process/): early steps have only coarse structure available, so the prompt influences layout; late steps have detail, so the same prompt influences texture. One vector, read differently at each noise level, because the thing reading it is at a different scale each time. + +Here is conditional sampling running — the same exact denoiser as before, with its posterior restricted to the requested class: + + + +
+**What it models.** Conditional sampling on the sprite dataset, where the "prompt" is one of eight shapes. The conditional score is the exact posterior restricted to training images of that class; the unconditional score is the posterior over all of them. Both are computed in closed form. Set the guidance dial to 1.0 and this is ordinary conditional generation — the case this page is about; the [next page](../guidance/) is about what the dial does away from 1. + +**Knobs.** The prompt buttons choose the class. The guidance slider is the subject of the next page — leave it at 1.0 for now. + +**Try this.** With w = 1.0, pick a prompt and press Run. Every finished sample in the strip is that shape, and they differ in position, size and angle — the model is sampling *within* the condition rather than memorizing one answer to it. Now drag w to 0 and run again: the prompt is ignored entirely and you get whatever the unconditional model felt like. Measured over 24 runs, w = 0 lands on the requested class 25% of the time — barely above the 12.5% you would get by chance — and w ≥ 0.5 hits it 100% of the time. +
+ +## Why This Page Is Short + +Because conditioning genuinely is a small change, and it is worth noticing how much of that is inherited rather than invented. + +The forward process is untouched — noise does not care what the caption said. The parameterization is untouched. The sampler is untouched. The loss is the same squared error with one more variable in the expectation. All the machinery from chapter 2 keeps working, and the only new component is an encoder borrowed from a different field entirely. + +Compare that to the alternative history. Conditioning a GAN meant conditioning the *discriminator* too, and then keeping a two-player game balanced while both players juggled an extra input — a genuinely hard problem that produced a decade of papers. The asymmetry is not an accident. It is what you get when your objective is a fixed regression instead of a moving target. + +## References + +- Radford et al., *Learning Transferable Visual Models From Natural Language Supervision.* ICML 2021. [arXiv:2103.00020](https://arxiv.org/abs/2103.00020) — CLIP. +- Rombach et al., *High-Resolution Image Synthesis with Latent Diffusion Models.* CVPR 2022. [arXiv:2112.10752](https://arxiv.org/abs/2112.10752) — §3.3 introduces the cross-attention conditioning mechanism used here. +- Saharia et al., *Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.* NeurIPS 2022. [arXiv:2205.11487](https://arxiv.org/abs/2205.11487) — Imagen; the finding that a large frozen text-only encoder outperforms CLIP for prompt fidelity. +- Hertz et al., *Prompt-to-Prompt Image Editing with Cross-Attention Control.* ICLR 2023. [arXiv:2208.01626](https://arxiv.org/abs/2208.01626) — evidence that cross-attention maps really do carry the spatial word-to-region correspondence. diff --git a/src/content/docs/diffusion/03-real-systems/guidance.mdx b/src/content/docs/diffusion/03-real-systems/guidance.mdx new file mode 100644 index 0000000..bb762a3 --- /dev/null +++ b/src/content/docs/diffusion/03-real-systems/guidance.mdx @@ -0,0 +1,80 @@ +--- +title: Guidance — The Dial Everyone Turns +description: Classifier-free guidance extrapolates away from the unconditional prediction. Why it works, what it costs in diversity, and why the saturation artifacts are an approximation error rather than a law. +sidebar: + order: 3 +--- + +import GuidanceDial from '../../../../components/visualizations/GuidanceDial'; + +Conditional sampling works, and it is not enough. Trained honestly on (image, caption) pairs, a diffusion model produces images that are *consistent* with the prompt but often only loosely committed to it — the caption is one influence among many, and the model is free to interpret. Users wanted the prompt obeyed. + +The fix is the single most consequential trick in deployed diffusion models, it costs no extra training, and it is one line of arithmetic. + +## Extrapolate, Don't Interpolate + +The model can produce two predictions for the same noisy image: one with the prompt, one without. The difference between them + +$$ +\varepsilon_\theta(x_t, t, c) - \varepsilon_\theta(x_t, t, \varnothing) +$$ + +is the direction the prompt is pulling in — the part of the denoising step that exists *because* of the caption. Classifier-free guidance takes that direction and pushes further along it than the model asked: + +$$ +\tilde\varepsilon = \varepsilon_\theta(x_t, t, \varnothing) + w\,\big[\varepsilon_\theta(x_t, t, c) - \varepsilon_\theta(x_t, t, \varnothing)\big] +$$ + +At $w = 0$ the prompt is discarded. At $w = 1$ this is exactly conditional sampling — the terms cancel to $\varepsilon_\theta(x_t,t,c)$. Above 1 you are **extrapolating past the conditional prediction, away from the unconditional one**, amplifying whatever the caption contributed. Production systems default to $w$ between 5 and 8. + +The training cost is almost nothing: randomly drop the caption for ~10% of examples so the same network learns the unconditional prediction too. One network, two modes, no classifier — which is where the name comes from, since the earlier version of this idea needed a separately trained classifier to supply the gradient. + + + +
+**What it models.** The guidance formula above, with both scores computed exactly rather than by a network. The conditional score is the closed-form posterior restricted to training images of the requested shape; the unconditional one is the posterior over all 128. The strip collects finished samples so the diversity question can be answered by looking. "On-prompt" counts how many came back as the requested shape; "distinct" counts how many are different images. + +**Knobs.** w is the guidance scale — the subject of the page. The prompt buttons change the requested class. + +**Try this.** At w = 0 press Run and watch samples arrive from all over the dataset — measured, the requested class shows up 25% of the time, near the 12.5% chance rate. Move to w = 0.5 and it locks on: **100% on-prompt**, and it stays there for every higher setting. Then hunt for the cost. Fix one prompt, run a batch at w = 1, and count distinct images; do it again at w = 8. Measured over 32 seeds on a fixed prompt, the sampler reaches **13 of the 16 possible images at w = 1 and only 8 at w = 8**, with the single most frequent image rising from 16% to 25% of all draws. Obedience is bought with variety. +
+ +## What It Is Really Doing + +The clean way to read the formula is through the score. Since $\varepsilon$ and the score are [the same object](../../02-diffusion/predict-the-noise/), guidance is sampling from a modified distribution: + +$$ +\tilde{p}(x \mid c) \;\propto\; p(x)\,\Big[\frac{p(x \mid c)}{p(x)}\Big]^{w} \;\propto\; p(x)\,p(c \mid x)^{w} +$$ + +The likelihood term is raised to the power $w$. At $w = 1$ that is Bayes' rule and you are sampling the true conditional. Above 1 you are sampling a distribution **sharpened around the prompt** — mass concentrated where the caption is most strongly satisfied, and stripped from everywhere it is merely satisfied adequately. + +That immediately explains both observations. Prompt adherence rises because you are up-weighting exactly the images the caption explains best. Diversity falls because a sharpened distribution is by definition a narrower one, and the images it removes are the unusual interpretations. **This is not a side effect to be engineered away; it is the same operation viewed from the other end.** + +## The Artifacts Are Approximation Error + +Everyone who has used these models knows the high-guidance look: blown-out contrast, oversaturated colour, a scorched quality that appears somewhere past $w \approx 12$. The natural assumption is that this is what sharpening does at the limit. + +It is not, and this widget is a clean demonstration of why. Push $w$ to 15 here and the diversity collapse continues as expected — but the **distance from each sample to the nearest real training image stays at 0.003, unchanged from $w = 1$.** No saturation, no artifacts, no degradation of image quality at all. The samples get less varied and remain perfectly valid images. + +The difference is that this page's score is *exact*. Guidance extrapolates along a direction, and extrapolation is only as trustworthy as the thing being extrapolated. A learned $\varepsilon_\theta$ is accurate near the data it was trained on and increasingly wrong as you push away from it; multiply that error by $w = 15$ and the trajectory leaves the region where the network means anything. The burned-out look is the network's approximation error, amplified — not a property of the mathematics. + +Which is the same finding this track has now hit three times. The [step count](../../02-diffusion/one-step-vs-many/) collapsed to four steps once the denoiser was exact. Generalization [could not be bought](../../02-diffusion/one-step-vs-many/) by widening a kernel, because an exact posterior memorizes by construction. And guidance artifacts vanish once the score is exact. **Every pathology that survives in this toy is mathematical; every one that disappears was approximation error wearing a costume.** That distinction is worth more than any individual result on this page, because it tells you which problems the field can engineer away and which it cannot. + +:::note[Why the default is 7 and not 70] +Guidance scale is the one knob most users ever touch, and the useful range is narrow. Below ~3 the prompt is a suggestion. Above ~12 the extrapolation outruns the network's accuracy. The commonly shipped 7–8 is empirical: roughly the largest push that stays inside the region where the learned score is still trustworthy. Newer samplers — dynamic thresholding, rescaled guidance — mostly work by *keeping the extrapolated prediction in range*, which is exactly what you would design if you believed the artifacts were approximation error. +::: + +## Where This Track Ends + +You can now read a Stable Diffusion model card and know what each part is for. A **VAE** compressing 512×512×3 into a 64×64×4 latent. A **U-Net** predicting noise in that latent at each of ~50 steps. A **CLIP** text encoder producing a vector that reaches every step through cross-attention. A **scheduler** setting the noise levels. A **guidance scale** deciding how hard the prompt is pushed. Every one of those is a page in this track. + +And the through-line, if you want one sentence: diffusion won not because it was more powerful than the adversarial game, but because it replaced a moving target with a fixed one. Everything else — the conditioning, the guidance, the latents, the distillation down to four steps — was only buildable on top of a loss that stays still. + +## References + +- Ho & Salimans, *Classifier-Free Diffusion Guidance.* NeurIPS 2021 Workshop. [arXiv:2207.12598](https://arxiv.org/abs/2207.12598) — the method, the caption-dropout training trick, and the quality/diversity trade-off. +- Dhariwal & Nichol, *Diffusion Models Beat GANs on Image Synthesis.* NeurIPS 2021. [arXiv:2105.05233](https://arxiv.org/abs/2105.05233) — classifier guidance, the predecessor that needed a separate classifier. +- Saharia et al., *Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.* NeurIPS 2022. [arXiv:2205.11487](https://arxiv.org/abs/2205.11487) — dynamic thresholding, introduced specifically to fix high-guidance saturation. +- Lin et al., *Common Diffusion Noise Schedules and Sample Steps are Flawed.* WACV 2024. [arXiv:2305.08891](https://arxiv.org/abs/2305.08891) — guidance rescaling, and the case that these artifacts are fixable implementation faults rather than inherent. +- Dieleman, *Guidance: a cheat code for diffusion models* (2022). [sander.ai](https://sander.ai/2022/05/26/guidance.html) — the clearest available explanation of the sharpened-distribution reading. diff --git a/src/content/docs/diffusion/03-real-systems/index.mdx b/src/content/docs/diffusion/03-real-systems/index.mdx new file mode 100644 index 0000000..5c004e4 --- /dev/null +++ b/src/content/docs/diffusion/03-real-systems/index.mdx @@ -0,0 +1,67 @@ +--- +title: Latent Diffusion +description: Why Stable Diffusion denoises a 64×64 code instead of a 512×512 image, and what the autoencoder around it is really for +sidebar: + order: 1 +--- + +import LatentCompress from '../../../../components/visualizations/LatentCompress'; + +Everything so far has run in pixel space. That is the honest way to learn the method and a ruinous way to deploy it. + +Count the cost. A 512×512 RGB image is 786,432 numbers. Every reverse step pushes all of them through a large network, and there are dozens of steps per image. The first pixel-space diffusion models were extraordinary and almost unusable — DALL·E 2 and Imagen ran on datacentre hardware and generated in the tens of seconds, and no one was running them on a laptop. + +Latent diffusion is the change that put image generation on consumer GPUs, and the idea is one sentence: **don't denoise the image, denoise a compressed code of the image.** + +## Most Pixels Are Not Carrying Information + +The premise is that images are enormously redundant — which is the manifold argument from [chapter 1](../../01-the-problem/) with a practical edge on it. If real images occupy a thin surface inside pixel space, then the coordinates *on that surface* are far fewer than the pixels, and the pixel representation is mostly spending its capacity on correlations you could have predicted. + +The way to check is to compress and see what breaks: + + + +
+**What it models.** A real encoder–decoder pair on the sprite dataset: principal components, fitted here by power iteration. A linear autoencoder trained to convergence learns exactly this subspace, so this is a genuine bottleneck rather than an illustration of one. Left is the original 576 numbers, middle is what survives a round trip through k numbers, right is the residual: precisely what the compression discarded. + +**Knobs.** k is the size of the code. The chart tracks cumulative variance captured as k grows. + +**Try this.** Start at k = 1 and step up, watching which properties come back in what order. Identity and rough position arrive early — by k = 8 (52.2% of variance, mean reconstruction error 0.441 across the dataset — the readout shows the current image, so it will differ) the shape is recognisable and roughly in the right place, while the crisp edges are not. The *semantics* are cheap; the exact rendering is expensive. Then keep going and notice that it never gets truly clean: at k = 64 this encoder still holds only 92.8% of the variance at mean error 0.181, and the residual panel is still busy. That failure is the next section. +
+ +## Why This Encoder Is Bad, and Why That Is the Point + +The measured numbers above are unimpressive, and it would be easy to quietly not mention them: 68.6% of variance at k = 16, 92.8% at k = 64, with visible softening throughout. If images were as compressible as claimed, a 64-number code should be nearly perfect. + +The gap is not evidence against the manifold argument — it is evidence about **what kind of map you need to exploit it.** These sprites have four true degrees of freedom, but two of them are rotation and translation, and those are savagely non-linear in pixel space. Shift a shape two pixels right and *every* pixel changes; there is no small set of fixed directions whose weighted sum expresses "the same shape, moved." A linear encoder is forced to spend component after component approximating a motion that a convolution represents for free. + +Which is exactly why real latent diffusion does not use PCA. Stable Diffusion's encoder is a deep convolutional network — translation-equivariant by construction, non-linear, and trained with a perceptual loss so that its errors land where human vision is least sensitive. It achieves at 48× what this linear map cannot manage at 9×. + +So read the widget as a lower bound. The claim it establishes is the ordering — semantics come back first, exact rendering last, and the code that captures *what the image is* is far smaller than the code that captures *every pixel of it*. The claim it cannot establish is how good a properly-built encoder gets, because a linear one is the wrong tool for this data and the numbers say so. + +## What the Autoencoder Is Actually For + +Latent diffusion brackets the whole process between two learned maps: an encoder $\mathcal{E}$ from images to a small code, a decoder $\mathcal{D}$ back. Stable Diffusion's takes 512×512×3 and returns 64×64×4 — **786,432 numbers down to 16,384, a 48× reduction.** Training noises the code, not the image. Sampling denoises a code from scratch and decodes once at the end. + +The subtle part is what the two halves are each responsible for, because it is not a simple split of labour: + +- **The autoencoder handles perceptual detail** — texture, high-frequency structure, the exact placement of edges. It is trained with a perceptual loss and an adversarial term, and yes, that is a GAN, doing the one job GANs are genuinely excellent at: making a single deterministic output look sharp. Nobody is asking it to model a distribution, so none of [chapter 1's](../why-gans-broke/) objections apply. +- **The diffusion model handles semantics** — what is in the image, where, in what arrangement. It works entirely in the compressed space and never sees a pixel. + +Which is a real division of concerns, not just a speed hack. The 1/f argument from [the forward process](../../02-diffusion/forward-process/) said that noise destroys fine detail almost immediately and coarse structure last, so the vast majority of a pixel-space diffusion model's capacity is spent on frequencies that are decided in the last few steps anyway. Latent diffusion hands that band to a single-pass autoencoder that can do it in one shot, and lets the expensive iterative machinery work only on the part that actually needs iterating. + +:::caution[The bottleneck is a ceiling, not just a speedup] +Nothing the diffusion model does can recover detail the encoder threw away. If the autoencoder cannot represent small text, the system cannot generate small text, no matter how many steps you run or how good the prompt is — which is exactly why early Stable Diffusion was so bad at written words. Push k down to 2 in the widget and the diffusion model working in that space is *strictly* incapable of producing anything the decoder cannot express — 22% of the variance is the whole world it can ever draw from. + +That is why the compression is kept mild. 48× sounds aggressive but is deliberately conservative: it is roughly the point where the reconstruction stops being distinguishable, and going further trades capability for speed. +::: + +## What Is Left + +We now have a fast conditional-free generator of plausible images. What is missing is the thing everyone actually uses these models for: **saying what you want.** The [next page](../conditioning/) is how a sentence gets into the loop, and the [one after](../guidance/) is the dial that decides how hard the model listens. + +## References + +- Rombach, Blattmann, Lorenz, Esser & Ommer, *High-Resolution Image Synthesis with Latent Diffusion Models.* CVPR 2022. [arXiv:2112.10752](https://arxiv.org/abs/2112.10752) — the paper Stable Diffusion is built on; §4 covers the autoencoder and the choice of downsampling factor. +- Esser, Rombach & Ommer, *Taming Transformers for High-Resolution Image Synthesis.* CVPR 2021. [arXiv:2012.09841](https://arxiv.org/abs/2012.09841) — the perceptual + adversarial autoencoder that latent diffusion reuses. +- Alammar, *The Illustrated Stable Diffusion* (2022). [jalammar.github.io](https://jalammar.github.io/illustrated-stable-diffusion/) — the standard visual walkthrough of how the pieces fit together. diff --git a/src/content/docs/index.mdx b/src/content/docs/index.mdx index 803396d..9fdf057 100644 --- a/src/content/docs/index.mdx +++ b/src/content/docs/index.mdx @@ -28,6 +28,16 @@ From MDPs to modern policy optimization for LLM alignment. +## Image Generation & Diffusion + +How a model turns pure noise into an image it has never seen — what the generation task actually asks for, what GANs answered and why the field left them, and why destroying data and learning to undo it one small step at a time turned out to work. + +**Prerequisites**: basic probability (Gaussians, conditional expectation) and familiarity with neural networks. No prior generative-modeling background assumed. + + + + + ## Speculative Decoding How modern LLM serving generates 3–6× faster without changing a single output token — from the lossless acceptance rule to EAGLE, DFlash, and DSpark. diff --git a/src/content/docs/specdec/03-parallel-drafting/index.mdx b/src/content/docs/specdec/03-parallel-drafting/index.mdx index f0d3ce8..e68e094 100644 --- a/src/content/docs/specdec/03-parallel-drafting/index.mdx +++ b/src/content/docs/specdec/03-parallel-drafting/index.mdx @@ -14,7 +14,7 @@ DFlash's answer: draft the **entire block in a single forward pass** — with a ## Why a Diffusion Drafter -Diffusion LLMs generate by iteratively refining a block of masked tokens **in parallel** — all positions at once — rather than left to right. As standalone generators they still trail autoregressive models in quality, which kept them out of production. DFlash's reframe: in a speculative pipeline, *draft quality only sets the acceptance rate*. The lossless acceptance rule from [chapter 1](../01-fundamentals/rejection-sampling/) means a diffusion model's weaknesses cost speed, never correctness — while its parallelism attacks exactly the term ($c$) that the autoregressive drafters cannot. It is the rare case where a technology's flaw is priced at zero and its strength at full value. +[Diffusion](../../diffusion/02-diffusion/) LLMs generate by iteratively refining a block of masked tokens **in parallel** — all positions at once — rather than left to right. As standalone generators they still trail autoregressive models in quality, which kept them out of production. DFlash's reframe: in a speculative pipeline, *draft quality only sets the acceptance rate*. The lossless acceptance rule from [chapter 1](../01-fundamentals/rejection-sampling/) means a diffusion model's weaknesses cost speed, never correctness — while its parallelism attacks exactly the term ($c$) that the autoregressive drafters cannot. It is the rare case where a technology's flaw is priced at zero and its strength at full value. So each cycle becomes: one parallel draft pass proposes the block → one target pass verifies it. Two big matmuls, no token-by-token loop on either side. diff --git a/src/styles/custom.css b/src/styles/custom.css index a7f18dd..18e2a28 100644 --- a/src/styles/custom.css +++ b/src/styles/custom.css @@ -331,6 +331,128 @@ animation: viz-pulse 1.1s ease-in-out infinite; } +/* Canvas-rendered images (diffusion track). The frame follows the theme; the + pixels inside deliberately do not — see lib/PixelCanvas.tsx. */ +.viz-pixel { + margin: 0; + display: flex; + flex-direction: column; + align-items: center; + gap: 0.3rem; +} + +.viz-pixel-canvas { + display: block; + border: 1px solid var(--sl-color-gray-5); + border-radius: 4px; + background: #000; + image-rendering: pixelated; + max-width: 100%; +} + +.viz-pixel-label { + font-size: 0.72rem; + color: var(--sl-color-gray-3); + text-align: center; + line-height: 1.3; +} + +/* Thumbnail picker for choosing which training image to work on. */ +.viz-sprite-row { + display: flex; + flex-wrap: wrap; + gap: 0.25rem; + margin-top: 0.5rem; + justify-content: center; +} + +.viz-sprite-btn { + border: 1px solid transparent; + border-radius: 4px; + background: none; + padding: 2px; + cursor: pointer; + line-height: 0; +} +.viz-sprite-btn:hover { + border-color: var(--sl-color-gray-4); +} +.viz-sprite-btn.active { + border-color: var(--sl-color-accent); +} + +/* Row of side-by-side panels inside a widget; wraps on narrow screens. */ +.viz-panels { + display: flex; + flex-wrap: wrap; + gap: 0.9rem; + align-items: flex-start; + justify-content: center; +} + +.viz-panel { + flex: 1 1 11rem; + min-width: 0; +} + +/* Charts inside a panel cap their width so a wide container doesn't stretch + them into oversized boxes (the viewBox aspect would scale height with it). */ +.viz-panel-chart { + width: 100%; + max-width: 22rem; + height: auto; +} + +/* Horizontal strip of frames, e.g. steps along an interpolation path. */ +.viz-strip { + display: flex; + flex-wrap: wrap; + gap: 0.35rem; + align-items: flex-start; + margin-top: 0.3rem; +} + +/* Wrapping row of controls under a widget body. */ +.viz-controls-row { + display: flex; + flex-wrap: wrap; + gap: 0.6rem 1.1rem; + align-items: flex-end; + margin-top: 0.7rem; +} + +.viz-legend { + display: flex; + flex-wrap: wrap; + gap: 0.15rem 0.7rem; + margin-top: 0.2rem; + font-size: 0.7rem; + color: var(--sl-color-gray-3); + line-height: 1.35; +} +.viz-legend span { + display: inline-flex; + align-items: center; + gap: 0.28rem; + white-space: nowrap; +} +.viz-legend i { + width: 0.85rem; + height: 0; + border-top-width: 2px; + border-top-style: solid; + flex: none; +} + +.viz-panel-title { + font-size: 0.72rem; + font-weight: 600; + letter-spacing: 0.04em; + text-transform: uppercase; + color: var(--sl-color-gray-3); + margin-bottom: 0.3rem; +} + /* Structured figure captions (What it models / Knobs / Try this) */ .viz-caption { margin: -0.75rem 0 1.75rem;