Feature set 134.

.github/workflows/checks.yml

@@ -1,7 +1,7 @@
 name: Checks
 
-# The gate on every PR to main. Before this the only CI here was the weekly scraper,
-# so a PR could only be verified by reading it.
+# The gate on every PR to main. Before this there was no CI that ran on a PR at all,
+# so a change could only be verified by reading it.
 #
 # Format, build, test. Deliberately NOT path-filtered: this is the required status
 # check for main, and a required check that never runs leaves a docs-only PR pending

.github/workflows/images.yml

@@ -1,17 +1,16 @@
 name: Artwork
 
-# This was "Scrape Final Fantasy Data", on a Sunday-night cron. Both the name and the schedule
-# are gone with the wiki stages: the catalogue is curated by hand now, rows are added through the
-# dashboard, and an unattended weekly job that rewrites data nobody asked it to touch is exactly
-# what was retired.
+# Artwork only, and manual only. The catalogue itself is curated by hand through the dashboard —
+# no job here writes data on a timer, because an unattended job that rewrites rows nobody asked it
+# to touch is exactly what was retired.
 #
-# What is left runs on request only. This is also the only place the R2 and Gemini credentials
-# exist, which is why the file survives at all — `generate` spends real money per image, so it
-# wants a deliberate press of a button and a ceiling, not a timer.
+# This is also the only place the R2 and Gemini credentials exist, which is why the file survives
+# at all — `generate` spends real money per image, so it wants a deliberate press of a button and
+# a ceiling, not a schedule.
 #
-# Worth knowing about `images`: a row imported from the wiki through the dashboard still points
-# at the wiki's own CDN. That stage copies the art into our bucket and repoints the row, so it is
-# the natural thing to run after a batch of imports. It skips anything already hosted by us,
+# Worth knowing about `images`: a row added through the dashboard can still point at artwork hosted
+# somewhere else. That stage copies the art into our bucket and repoints the row, so it is the
+# natural thing to run after a batch of additions. It skips anything already hosted by us,
 # including hand-uploaded art, so it costs nothing when there is nothing to do.
 on:
   workflow_dispatch:
@@ -59,8 +58,8 @@ jobs:
         env:
           CONNECTION_STRING: ${{ secrets.CONNECTION_STRING }}
 
-          # Every stage here writes to the bucket, so unlike the old scrape these are required
-          # rather than optional — the tool exits with an error when they are missing.
+          # Every stage here writes to the bucket, so these are required rather than optional —
+          # the tool exits with an error when they are missing.
           # The account id is the same value under both names — the secret predates the R2
           # naming the tool reads it by.
           R2_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}

README.md

@@ -23,7 +23,7 @@ A free, open REST API for Final Fantasy data — characters, monsters, and games
 - **HybridCache** — stampede-proof L1/L2 caching out of the box
 - **Rate limiting** — 60 req/min anonymous, 600 req/min with an API key
 - **Interactive docs** at `/scalar/v1` (far nicer than Swagger UI)
-- **Auto-updating** — a GitHub Action scrapes the Final Fantasy Wiki every Sunday
+- **Hand-curated** — every row reviewed and edited through the dashboard, not bulk-imported
 
 ---
 
@@ -103,7 +103,7 @@ Full interactive docs at [`/scalar/v1`](https://moogleapi.com/scalar/v1).
 | Caching | `HybridCache` — L1 in-process + optional L2 Redis |
 | Docs | [Scalar](https://scalar.com) — replaces Swagger UI |
 | Rate Limiting | `PartitionedRateLimiter` (native .NET 10) |
-| Data pipeline | GitHub Actions scraper → Final Fantasy Wiki |
+| Artwork pipeline | GitHub Actions → Gemini → Cloudflare R2 |
 
 ### Project Structure
 
@@ -138,17 +138,26 @@ MoogleApi.sln
 │       ├── wwwroot/              ← Landing page + /games hub + four games
 │       └── Program.cs
 ├── scripts/
-│   └── MoogleAPI.Scraper/        ← Console app, runs in GitHub Actions
+│   └── MoogleAPI.Scraper/        ← Artwork tool, runs in GitHub Actions
 └── tests/
     └── MoogleAPI.Tests/
 ```
 ---
 
-## 🤖 Data Pipeline
+## 🤖 Data & Artwork
 
-A GitHub Action runs every Sunday at 2 AM UTC and scrapes the [Final Fantasy Wiki](https://finalfantasy.fandom.com) via the MediaWiki API. It upserts characters and monsters per game — no duplicates, no full reloads.
+The catalogue is curated by hand. Rows are added and edited through the private dashboard, one at
+a time, with a person deciding what belongs — there is no unattended job that rewrites the data on
+a timer, and no bulk import behind the current contents.
 
-Stages can be run individually with `--only=`: `games`, `characters`, `playable`, `monsters`, `cards`, `images`, `audit`, `generate`, `promote`. The `playable` stage reads each game's character navbox to mark which characters the player actually controls — the only source scoped to a single game, since the wiki has no playable-character category and the prose test answers for the whole compilation.
+What still runs on request is the artwork tool, dispatched manually from the **Artwork** workflow.
+Its stages are `images` (copy artwork into our own bucket and repoint the row), `audit` (classify
+what each image actually is), `generate` (replace it with an illustration in one house style) and
+`unpromote` (withdraw generated art and restore the original). `generate` costs money per image, so
+it takes an explicit ceiling with `--max` and never runs as part of an unnamed "all stages" pass.
+
+Artwork is served from Cloudflare R2 at `images.moogleapi.com`. Keys derive from the row id, which
+is what makes a move between domains a database pass rather than a re-upload.
 
 ---
 
@@ -169,7 +178,9 @@ allowlist with `ApiKeys__Keys__0`, `ApiKeys__Keys__1`, … With none set, everyt
 
 ## 📜 Disclaimer
 
-MoogleAPI is a fan project and is not affiliated with or endorsed by Square Enix. All Final Fantasy names, characters, and related marks are trademarks of Square Enix Co., Ltd. Data is sourced from the community-maintained [Final Fantasy Wiki](https://finalfantasy.fandom.com).
+MoogleAPI is a fan project and is not affiliated with or endorsed by Square Enix. All Final Fantasy names, characters, and related marks are trademarks of Square Enix Co., Ltd.
+
+The catalogue was originally seeded from the community-maintained [Final Fantasy Wiki](https://finalfantasy.fandom.com) and is maintained by hand from there on. That attribution stays while any of it remains: the wiki's text is CC BY-SA, which requires credit regardless of how much editing has happened since.
 
 ---
 

cloudflare/RATE-LIMITING.md

@@ -0,0 +1,93 @@
+# Rate limiting
+
+Two layers, doing different jobs. The edge one is the one that protects costs; the app one is the
+one that knows who is keyed.
+
+## Why the app limiter alone was not enough
+
+Railway terminates connections at its own load balancer, so `Connection.RemoteIpAddress` inside the
+app is one of a small pool of internal `100.64.0.0/10` addresses — never the caller. Measured
+against three months of production request logs on 2026-08-05, the ten busiest "clients" were
+`100.64.0.3` through `100.64.0.9`, and the distinct-address count sat at 21–24 a day whether the day
+served 56 requests or 1,730. Real visitors scale with traffic; a fixed pool of load balancers does
+not.
+
+The consequences were that no caller could be isolated (an abuser's requests scattered across ~20
+buckets), legitimate callers shared buckets with abusers, and the effective anonymous ceiling was
+roughly 60/min × proxy nodes × replicas — which is where the ~290/min measured in production came
+from.
+
+## What is fixed in the app
+
+`ClientIpResolver` now resolves the caller from `CF-Connecting-IP`, but **only** on requests that
+carry the shared secret the Worker injects. That condition is the whole design: Cloudflare appends
+to a caller-supplied `X-Forwarded-For`, so its first entry is attacker-controlled, and Railway also
+answers on its own `*.up.railway.app` hostname where `CF-Connecting-IP` can simply be typed in. A
+spoofable partition key would be worse than a coarse one — it would let one caller mint unlimited
+fresh windows. Without the secret, the resolver falls back to the peer address: imprecise, never in
+the attacker's favour.
+
+Rejections are also recorded now. `UseRateLimiter` short-circuits ahead of the logging middleware,
+so three months of logs contain zero 429s — which was never evidence the limiter wasn't firing.
+
+### Setup — both sides, or it silently degrades
+
+```bash
+# 1. Generate a secret
+openssl rand -hex 32
+
+# 2. Cloudflare Worker
+cd cloudflare/maintenance-worker
+wrangler secret put EDGE_SECRET     # paste it
+
+# 3. Railway → Variables (same value)
+Edge__Secret = <the same value>
+```
+
+If the two disagree the app stops trusting forwarded addresses and quietly goes back to limiting on
+the load balancer. Nothing breaks and nothing warns, so treat rotation as a two-sided change.
+
+**Verifying it took**, once both are deployed: hit the site through Cloudflare a few times, then
+check that `/stats` shows a "Busiest Clients" list that grows with real traffic rather than sitting
+at ~20 fixed hashes. Those hashes are the tell.
+
+## What still needs doing at the edge — not applicable from this repo
+
+The app limiter is per-process, and Railway runs several replicas, so its ceiling is still
+multiplied by replica count. More importantly, a request it rejects has already cost Railway compute
+and possibly a Neon query. Blocking at Cloudflare is what actually protects spend.
+
+In the dashboard: **Security → WAF → Rate limiting rules → Create rule**
+
+| Field | Value |
+|---|---|
+| Rule name | `api-anonymous` |
+| If incoming requests match | `(http.request.uri.path contains "/api/")` |
+| Characteristics | IP |
+| Period | 1 minute |
+| Requests | 120 |
+| Action | Block (or Managed Challenge) |
+| Duration | 1 minute |
+
+Notes on the numbers. 120/min is deliberately looser than the app's 60 — the edge rule is the
+backstop against abuse, and the app is what draws the anonymous/keyed distinction. Set the edge
+below the app limit and the app's tiers stop meaning anything.
+
+To exempt keyed callers, add to the rule expression:
+
+```
+(http.request.uri.path contains "/api/" and not any(http.request.headers["x-api-key"][*] in {"key-one" "key-two"}))
+```
+
+That places live credentials in a dashboard rule, which is a real trade — it is why this is written
+as optional rather than recommended. The alternative is to leave keyed callers subject to the edge
+rule too and raise its ceiling.
+
+Free plans allow a limited number of rate-limiting rules; check the allowance on the current plan
+before designing around several.
+
+## Costs worth knowing
+
+Images serve from R2, which has no egress fees, so image bandwidth is not the exposure. Railway
+compute and Neon are — which makes the list and search endpoints the ones worth protecting. Both
+are already fronted by `HybridCache`.

cloudflare/maintenance-worker/worker.js

@@ -22,8 +22,10 @@ const FAILURE_STATUSES = new Set([502, 503, 504, 521, 522, 523, 524, 525, 526]);
 const RETRY_DELAY_MS = 1500;
 
 export default {
-  async fetch(request) {
-    let response = await tryOrigin(request);
+  async fetch(request, env) {
+    const proxied = withEdgeSecret(request, env);
+
+    let response = await tryOrigin(proxied);
 
     // Only replay methods that are safe to run twice. A retried POST could double
     // a daily-guess submission, which is worse than showing the maintenance page.
@@ -32,7 +34,7 @@ export default {
     if (response === null || FAILURE_STATUSES.has(response.status)) {
       if (isReplayable) {
         await sleep(RETRY_DELAY_MS);
-        const second = await tryOrigin(request);
+        const second = await tryOrigin(proxied);
         if (second !== null && !FAILURE_STATUSES.has(second.status)) return second;
         response = second;
       }
@@ -43,6 +45,33 @@ export default {
   },
 };
 
+/**
+ * Stamps the request with the shared secret that tells the origin this call really came
+ * through Cloudflare, so it can believe CF-Connecting-IP and rate-limit the actual caller.
+ *
+ * Without this the origin has no usable identity for a request: Railway terminates at its
+ * own load balancer, so the peer address the app sees is one of about twenty internal
+ * 100.64.x addresses shared by everybody. X-Forwarded-For can't fill the gap either —
+ * Cloudflare appends to whatever the caller sends, so its first entry is caller-controlled.
+ *
+ * The header is always set or deleted, never passed through: Railway answers on its own
+ * *.up.railway.app hostname too, and a caller who could forge this on that path would be
+ * choosing their own rate-limit bucket. Overwriting here means anything inbound is discarded.
+ */
+function withEdgeSecret(request, env) {
+  const proxied = new Request(request);
+
+  if (env && env.EDGE_SECRET) {
+    proxied.headers.set('X-Moogle-Edge', env.EDGE_SECRET);
+  } else {
+    // No secret configured: strip it rather than forward a caller's own value. The origin
+    // then falls back to the peer address, which is imprecise but not attacker-chosen.
+    proxied.headers.delete('X-Moogle-Edge');
+  }
+
+  return proxied;
+}
+
 async function tryOrigin(request) {
   try {
     return await fetch(request);

cloudflare/maintenance-worker/wrangler.toml

@@ -17,3 +17,13 @@ zone_name = "moogleapi.com"
 
 # No observability, KV, or bindings on purpose — this Worker is pure pass-through
 # and adding state would put something else in the critical path of every request.
+#
+# One secret, deliberately not a var and so not written here:
+#
+#   wrangler secret put EDGE_SECRET
+#
+# It is stamped onto every proxied request as X-Moogle-Edge, and it is what lets the origin
+# believe CF-Connecting-IP — see withEdgeSecret in worker.js. The same value goes to Railway
+# as Edge__Secret. If the two ever disagree the app simply stops trusting forwarded addresses
+# and rate-limits on the load balancer again: degraded, not broken, and silent, so treat
+# rotating it as a two-sided change.

scripts/MoogleAPI.Scraper/ImageStore.cs

@@ -50,7 +50,7 @@ public record ImageStoreOptions(
 /// Copies artwork into Cloudflare R2, re-encoded on the way in.
 /// </summary>
 /// <remarks>
-/// Storing the wiki's originals verbatim would be roughly 4.6 GB across the library and would
+/// Storing the source originals verbatim would be roughly 4.6 GB across the library and would
 /// serve megabyte PNGs to phones. Resizing to a sane bound and re-encoding as WebP lands the
 /// whole set near 0.5 GB while still being sharper than the 400px thumbnails half the rows
 /// currently point at.
@@ -120,7 +120,7 @@ public async Task<bool> EnsureBucketAsync(CancellationToken ct)
     {
         try
         {
-            // No Referer: the wiki's CDN rejects any request that carries one.
+            // No Referer: the source CDN rejects any request that carries one.
             using var response = await http.GetAsync(OriginalOf(sourceUrl), ct);
             if (!response.IsSuccessStatusCode)
             {
@@ -150,7 +150,7 @@ await _s3.PutObjectAsync(new PutObjectRequest
             return key;
         }
         // A missing image is never fatal to a run — including when the source is not a URL at
-        // all. Not every row's provenance comes from the wiki now: the dashboard records a
+        // all. Not every row's provenance is a remote URL: the dashboard records a
         // hand-upload by writing what it did into ImageSourceUrl, and a forced re-copy hands
         // that straight to HttpClient, which rejects a non-absolute address before any request
         // leaves the process. Uncaught, one such row would take the whole image stage down with

scripts/MoogleAPI.Scraper/Program.cs

@@ -6,16 +6,14 @@
 using MoogleAPI.Scraper.Scrapers;
 using MoogleAPI.Web.Infrastructure.Data;
 
-// This was a wiki scraper. It is now an artwork tool, and it keeps the name only because
-// renaming a project is noisier than the clarity would be worth.
+// The artwork tool. It keeps the project name only because renaming one is noisier than the
+// clarity would be worth.
 //
-// The stages that built the catalogue out of Final Fantasy Wiki — games, characters, playable,
-// monsters, cards, repair — are gone. They finished. What they produced has since been corrected
-// and curated by hand, and every one of them matched rows by wiki page name rather than by id,
-// so a re-run could only undo that work: deleted rows returned, renamed rows arrived a second
-// time, and hand-scored popularity was overwritten on sight. New rows are added through the
-// dashboard now, one at a time, with the wiki available as something to import from rather than
-// as the authority.
+// The stages that first populated the catalogue are gone. They were a seeder: they ran, they
+// finished, and what they produced has since been corrected and curated by hand. They also
+// matched rows by page name rather than by id, so a re-run could only undo that work — deleted
+// rows returned, renamed rows arrived a second time, and hand-scored popularity was overwritten
+// on sight. Rows are added through the dashboard now, one at a time, by a person.
 //
 // What remains is the part that has not finished, because it is about pixels rather than facts:
 // copying art into R2, classifying what each picture actually is, and paying Gemini to replace

scripts/MoogleAPI.Scraper/Scrapers/ImageReverter.cs

@@ -81,8 +81,8 @@ public async Task RevertAsync(CancellationToken ct = default)
     /// Where a row's artwork should point once the generated version is withdrawn.
     /// </summary>
     /// <remarks>
-    /// The copy the image stage made of the wiki's original, which promotion never touched — it
-    /// only re-pointed the column, so the file is still at its own address. The scraped wiki URL
+    /// The copy the image stage made of the original, which promotion never touched — it only
+    /// re-pointed the column, so the file is still at its own address. The recorded source URL
     /// is the fallback, and it is a poor one: that CDN blocks any request carrying a Referer,
     /// which is the whole reason the images were copied here. Better than a row with no picture.
     /// Null means neither exists, and the caller must leave the row alone.

scripts/MoogleAPI.Scraper/Scrapers/ImageScraper.cs

@@ -8,7 +8,7 @@ namespace MoogleAPI.Scraper.Scrapers;
 /// Copies every piece of artwork the API points at into our own bucket.
 /// </summary>
 /// <remarks>
-/// Until this runs, the API serves URLs on the wiki's CDN — which hotlink-blocks any request
+/// Until this runs, the API serves URLs on somebody else's CDN — which hotlink-blocks any request
 /// carrying a Referer, meaning every consumer has to remember to suppress it, and any change
 /// on their side breaks every image at once. Once copied, the URLs are ours and stable.
 /// <para>