Skip to content

Feature set 134. - #61

Merged
jackfperryjr merged 1 commit into
mainfrom
features-134
Aug 6, 2026
Merged

Feature set 134.#61
jackfperryjr merged 1 commit into
mainfrom
features-134

Conversation

@jackfperryjr

Copy link
Copy Markdown
Owner

Rate limiting was partitioning on the load balancer

Connection.RemoteIpAddress behind Railway is not the caller. Recovered from three months of
production IpHash values by hashing 4.7M candidate addresses through the same
SHA256(...)[..16] the logger uses — six of the ten busiest "clients" were:

41299CDA2B4E7CCC -> ::ffff:100.64.0.3
8FF6F9580DEB0B71 -> ::ffff:100.64.0.4
74D2322EC22F06E8 -> ::ffff:100.64.0.5
...

100.64.0.0/10 is carrier-grade NAT space — Railway's internal load balancer pool. Not Cloudflare
edge addresses either (all 15 published ranges were hashed; no hits). The aggregate agrees: distinct
addresses per day sat at 21–24 whether the day served 56 requests or 1,730.

So no caller could be isolated, legitimate callers shared buckets with abusers, and the anonymous
ceiling was ~60/min × proxy nodes × replicas — the source of the ~290/min measured in production.

ClientIpResolver resolves CF-Connecting-IP, but only when the request carries a secret the
Worker injects. That condition is the design, not belt-and-braces: Cloudflare appends to a
caller-supplied X-Forwarded-For so its first entry is attacker-controlled, and Railway answers on
its own *.up.railway.app hostname where CF-Connecting-IP can simply be typed in. A spoofable
partition key is worse than a coarse one — it lets one caller mint unlimited fresh windows. Without
the secret it falls back to the peer address. Seven tests cover the bypasses.

429s are now recorded. UseRateLimiter short-circuits ahead of the logging middleware, which is
why the logs contain zero of them — that zero was never evidence the limiter wasn't firing.
Reordering the pipeline was rejected as the fix: it would have exempted every static asset from rate
limiting. OnRejected writes through the extracted RequestLogWriter instead.

Stats: ranges and metrics

Presets (24h / 7d / 30d / All) plus custom dates. Buckets switch hourly → daily past three days, and
the response echoes the window actually measured, because a chart whose axis silently changed unit
is a chart that lies about its shape. Very large ranges are trimmed to their most recent slice and
say so.

New: keyed vs anonymous, top failing paths, per-endpoint p95 (endpoints under five requests omitted),
busiest clients. The clients panel carries the caveat that pre-fix rows counted infrastructure.

Verified against production data by booting locally with a temporary auth bypass, since the endpoint
is owner-only — that bypass is reverted and not in this diff. It caught a real bug: model binding
returns Kind=Local for an ISO string ending in Z, and Npgsql refuses non-UTC for timestamptz,
so every custom range 500'd while the default 24 hours worked. Fixed and re-verified across eight
range shapes including reversed, empty, zone-less, and all-time.

Two findings worth your eye, from the new panels: 125 404s on /api/v1/games (something is
calling a versioned path that doesn't exist), and /api/arena/roster at 3.3s p95, the slowest
endpoint in the catalogue.

Language

Drops the framing of this as a wiki-scraping project — README, workflow headers, tool header. The
seeder is gone and the catalogue is curated by hand. Attribution to the wiki stays: its text is
CC BY-SA, which requires credit regardless of how much hand-editing has happened since. References
that describe live behaviour (the Referer-blocking CDN, the MediaWiki thumbnail URL the regex
actually parses) were generalised, not deleted.

Needs you

cloudflare/RATE-LIMITING.md has the steps I can't do from here: wrangler secret put EDGE_SECRET,
the matching Edge__Secret on Railway, and the edge rate-limiting rule. Until both sides of the
secret are set, this changes nothing
— the resolver falls back to today's behaviour, silently.

🤖 Generated with Claude Code

Rate-limit the caller, not the load balancer.

Connection.RemoteIpAddress behind Railway is one of ~20 internal 100.64.x
addresses, so every anonymous caller shared about twenty buckets and none
of them was anybody. Recovered from production logs by hashing candidate
ranges against the stored IpHash values: the ten busiest "clients" were
100.64.0.3 through .9.

ClientIpResolver reads CF-Connecting-IP, but only on requests carrying a
secret the Worker injects — Cloudflare appends to a caller-supplied
X-Forwarded-For, and Railway answers on its own hostname where the header
can just be typed in, so a spoofable key would be worse than a coarse one.

Rate-limit rejections are now recorded. UseRateLimiter short-circuits
ahead of the logging middleware, so the logs held zero 429s.

Stats: date ranges, and metrics worth the trip.

The page was fixed at 24 hours. It now takes any range, switches from
hourly to daily buckets past three days, and reports the window it
actually measured. Adds keyed-vs-anonymous split, top failing paths,
per-endpoint p95, and busiest clients.

Also drops the language describing this as a wiki-scraping project. The
seeder is gone; attribution stays, because the wiki's text is CC BY-SA.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jackfperryjr
jackfperryjr merged commit 82f87d4 into main Aug 6, 2026
1 check passed
@jackfperryjr
jackfperryjr deleted the features-134 branch August 6, 2026 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant