contextforge-org · Lang-Akshay · Jul 30, 2026 · Aug 7, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -1,5 +1,18 @@
 # AGENTS.md
 
+## Start with the wiki
+
+At the start of each task, check `_context/wiki/index.md` to decide
+whether wiki context is needed before acting. Don't read the wiki in
+full. Use the index and follow links only when they are relevant to
+the task.
+
+## Update the wiki
+
+After completing a task, offer to update the wiki if the task yielded durable knowledge that could benefit future work, then wait for user approval. This includes new processes, architecture decisions, or insights that go beyond the immediate task.
+
+---
+
 Guidance for agents working on `contextforge-gateway-rs`.
 
 This repo is the Rust dataplane part of ContextForge. It must stay compatible

diff --git a/Makefile b/Makefile
@@ -1,17 +1,17 @@
-.PHONY: help docker-prod testing-up testing-down
+.PHONY: help docker-prod compose-up compose-down
 
 help: ## Show this help
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
 
 docker-prod: ## Build production Docker image (dataplane:latest) from docker/Dockerfile
 	docker build -t dataplane:latest -f docker/Dockerfile .
 
-testing-up: ## Launch testing stack: nginx, control plane, redis, postgres, pgbouncer, dataplane, fast_time_server
+compose-up: ## Launch stack: nginx, control plane, redis, postgres, pgbouncer, dataplane, fast_time_server
 	@docker image inspect dataplane:latest >/dev/null 2>&1 || { \
 		echo "Image dataplane:latest not found. Run 'make docker-prod' first."; \
 		exit 1; \
 	}
 	docker compose -f docker/docker-compose.yml up -d nginx control-plane redis postgres pgbouncer data-plane fast_time_server register_fast_time
 
-testing-down: ## Tear down the testing stack
+compose-down: ## Tear down the stack
 	docker compose -f docker/docker-compose.yml stop nginx control-plane redis postgres pgbouncer data-plane fast_time_server register_fast_time
diff --git a/_context/wiki/architecture.md b/_context/wiki/architecture.md
@@ -0,0 +1,75 @@
+# Architecture
+
+## Middleware Stack Order
+
+Tower layers execute outside-in. A request reaches MCP handlers with these extensions already set:
+
+```
+TCP/TLS listener
+  -> HttpMetricsLayer
+  -> TraceLayer
+  -> /contextforge-rs nested router
+  -> CORS layer
+  -> virtual_host_id_layer       → inserts VirtualHostId           (400 on path mismatch)
+  -> claims_layer                → inserts ContextForgeClaims      (401 on bad/missing JWT)
+  -> session_id_layer            → inserts SessionId if present
+  -> user_config_store_layer     → inserts UserConfig              (400 no config, 500 store error)
+  -> virtual_host_config_layer   → rejects unknown vhost           (404 "Server not found")
+  -> /servers/{virtual_host_name}/mcp RMCP service
+```
+
+MCP handlers read typed extensions — they never parse headers, paths, or Redis keys directly.
+
+## Pipeline Shape
+
+```
+downstream request
+  -> virtual host extraction → JWT validation → session extraction
+  -> user config lookup → MCP handler validation
+  -> request plugin hooks
+  -> backend MCP call (concurrent via join_all for initialize/list)
+
+upstream response
+  -> response plugin hooks → merge/namespace/passthrough
+  -> metrics, tracing, logging → downstream response
+```
+
+Order is invariant: auth/config before backend selection; request plugins before upstream; response plugins before returning.
+
+## Module Boundaries (`contextforge-gateway-rs-lib`)
+
+| Module | Owns |
+| --- | --- |
+| `common.rs` | CLI config shape, JWT claims, Redis config validation, `reqwest::Client` construction |
+| `layers/` | HTTP request extension extraction, request-bound validation |
+| `gateway/` | MCP server behavior, initialize fanout, list merging, prefixed routing, backend service state |
+| `gateway/session_store/` | Local and Redis user session storage |
+| `user_config_store/` | `UserConfigStore` trait, Redis-backed store |
+| `transports/` | Downstream TCP and TLS listener setup |
+| `tools.rs` | Local bootstrap helpers (`with_tools` feature only) |
+
+## State Ownership
+
+| State | Owner | Lifetime |
+| --- | --- | --- |
+| CLI `Config` | Binary startup + `Gateway` | Process |
+| JWT decoders | `ContextForgeGatewayAppState` | Process |
+| User config | `RedisUserConfigStore` (LRU + Redis) | Request-path consumed; control-plane authored |
+| Request identity / VirtualHostId | Request extensions | One HTTP request |
+| Downstream session id | RMCP + `SessionId` extension | MCP session |
+| Backend RMCP services | `BackendTransports` map | Local process, per principal/backend/session |
+| Local user session mapping | `LocalUserSessionStore` | Local LRU, 50k entries, 1 hour |
+| Plugin manager | `CpexRuntimeRegistry` | Process, reloadable |
+
+> **Session rule:** backend MCP services are local process state. Sticky routing required for load-balanced deployments.
+
+## Executor Shapes
+
+| `--single-runtime` | Shape |
+| --- | --- |
+| `true` (default) | One multi-thread Tokio runtime, `--number-of-cpus` workers. All connections share one `BackendTransports`. |
+| `false` | One OS thread per CPU, each with its own current-thread Tokio runtime and own `BackendTransports`. `SO_REUSEPORT` spreads connections — no session affinity. **Stateful MCP sessions need `--single-runtime true`**. |
+
+## Lock Design
+
+Locks guard maps of handles, not I/O. Backend calls, Redis reads, and plugin hooks run outside any gateway lock. `borrow_transports()` clones `Arc<RunningService>` so the lock is not held across calls.
diff --git a/_context/wiki/config.md b/_context/wiki/config.md
@@ -0,0 +1,154 @@
+# Configuration Reference
+
+## Minimum Required Flags
+
+```
+--redis-address   --redis-port   --redis-mode
+```
+
+Plus at least: `--address` or `--tls-address`, `--token-verification-public-key` or `--token-verification-secret`.
+
+## Key CLI Flags (env var: `CONTEXTFORGE_GATEWAY_RS_*`)
+
+| Flag | Env suffix | Default | Note |
+| --- | --- | --- | --- |
+| `--address` | `ADDRESS` | — | Plain HTTP listener |
+| `--tls-address` | `TLS_ADDRESS` | — | Requires cert + key |
+| `--server-certificate` | `TLS_SERVER_CERTIFICATE` | — | With `--tls-address` |
+| `--server-private-key` | `TLS_SERVER_PRIVATE_KEY` | — | With `--tls-address` |
+| `--token-verification-public-key` | `TOKEN_VERIFICATION_PUBLIC_KEY` | — | RSA (RS256/384/512) |
+| `--token-verification-secret` | `TOKEN_SECRET` | — | HMAC (HS256/384/512) |
+| `--redis-address` | `REDIS_HOSTNAME` | **required** | |
+| `--redis-port` | `REDIS_PORT` | **required** | |
+| `--redis-mode` | `REDIS_CONNECTION_MODE` | **required** | `plain-text` \| `tls` \| `mtls` |
+| `--user-config-cache-expiry-seconds` | `USER_CONFIG_CACHE_EXPIRY_SECONDS` | `60` | `0` = no cache |
+| `--upstream-connection-mode` | `UPSTREAM_CONNECTION_MODE` | HTTPS-only | `plain-text-or-tls` for local HTTP backends |
+| `--number-of-cpus` | `GATEWAY_CPUS` | host CPU count | Tokio worker threads |
+| `--single-runtime` | `SINGLE_RUNTIME` | `true` | `false` = multi-runtime (no session affinity) |
+| `--runtime-plugins-enabled` | `RUNTIME_PLUGINS_ENABLED` | `false` | Enables CPEX hooks |
+| `--enable-open-telemetry` | `ENABLE_OPEN_TELEMETRY` | `false` | OTLP traces |
+| `--enable-otel-metrics` | `ENABLE_OTEL_METRICS` | `false` | OTLP metrics |
+
+## JWT Claims (validated by `claims_layer`)
+
+| Claim | Required value |
+| --- | --- |
+| `iss` | `mcpgateway` |
+| `aud` | `mcpgateway-api` |
+| `exp` | present, not expired |
+| `sub` | → selects Redis user config key |
+
+Optional: `token_use`, `iat`, `teams`, `scopes`, `user.full_name`.
+
+> **No revocation:** a leaked token is valid until `exp`. Rotate the signing key and restart to invalidate all outstanding tokens.
+
+## UserConfig Shape (from `contextforge-gateway-rs-apis`)
+
+```
+UserConfig
+  virtual_hosts: HashMap<String, VirtualHost>
+
+VirtualHost
+  backends: HashMap<String, BackendMCPGateway>   ← map key = routing prefix
+
+BackendMCPGateway
+  name: String
+  url: Url
+  transport: STREAMABLEHTTP | SSE | STDIO         ← only STREAMABLEHTTP used today
+  passthrough_headers: Vec<String>                ← snapshotted at initialize; session-scoped
+  add_headers: HashMap<String, String>            ← injected after passthrough
+  remove_headers: Vec<String>                     ← stripped after add
+  tool_name_aliases: HashMap<String, String>      ← downstream_alias → upstream_original
+  allowed_tool_names: Vec<String>                 ← model exists, NOT currently enforced
+  allowed_resource_names: Vec<String>             ← model exists, NOT currently enforced
+  allowed_prompt_names: Vec<String>               ← model exists, NOT currently enforced
+```
+
+**Header apply order:** `passthrough_headers` → `add_headers` (override passthrough) → `remove_headers` (applied last).
+
+**`passthrough_headers` is session-scoped.** Values are snapshotted from the `initialize` request and baked into the backend transport for the session lifetime. Post-`initialize` calls (tool calls, list calls) reuse those headers. Request-scoped propagation requires per-request transport reconstruction (future work).
+
+**Protected headers** — silently skipped in all three phases (passthrough/add/remove):
+
+| Category | Headers |
+| --- | --- |
+| Body-framing | `Content-Length`, `Content-Type` |
+| Hop-by-hop | `Connection`, `Keep-Alive`, `Proxy-Authenticate`, `Proxy-Authorization`, `Proxy-Connection`, `TE`, `Trailer`, `Trailers`, `Transfer-Encoding`, `Upgrade` |
+| RMCP-reserved | `Mcp-Session-Id`, `Accept`, `Last-Event-Id` |
+| Gateway-managed | `Host` (set from backend URL host + port; never overridden by config) |
+
+Redis storage: `MessagePack(User::new(sub))` → `MessagePack(UserConfig)`.
+
+Schema: `schemas/user_config.json`. Regenerate after any struct change:
+```bash
+cargo run -p contextforge-gateway-rs-apis
+```
+
+## Plugin Config (Redis key: `ContextForgeGatewayRuntimePluginConfig`)
+
+```
+RuntimePluginConfigDocument
+  version: 1
+  cpex: CpexConfig
+```
+
+Supported: `cmf.tool_pre_invoke`, `cmf.tool_post_invoke` only.  
+Rejected: routing-based selection, plugin dirs, global policies, other hook types.  
+Reload watcher: 10-minute interval. Invalid reload → runtime marked failed.
+
+## Startup Validation (fails fast)
+
+| Invalid combo | Reason |
+| --- | --- |
+| `--tls-address` without cert or key | Rustls needs both |
+| Same address for `--address` and `--tls-address` | Cannot bind same socket twice |
+| `--redis-mode tls` without trust bundle | Required |
+| `--redis-mode mtls` without trust bundle + client cert + key | All three required |
+| mTLS upstream without cert and key | reqwest identity cannot be built |
+| HTTP backend URL with default upstream mode (HTTPS-only) | Calls fail before reaching backend |
+
+## Upstream Connection Modes
+
+| Mode | Behavior |
+| --- | --- |
+| omitted / `tls-only` | HTTPS backends only (safe default) |
+| `plain-text-or-tls` | HTTP or HTTPS (use for local Compose backends) |
+| `plain-text-or-m-tls` | HTTP or HTTPS + client identity |
+| `mtls-only` | HTTPS + client cert/key required |
+
+## Logging Env Vars
+
+| Var | Default | Controls |
+| --- | --- | --- |
+| `RUST_LOG` | `debug` | Console filter |
+| `RUST_FILE_LOG` | `debug` | File filter |
+| `RUST_TRACE_LOG` | `info` | OTLP span filter (`debug` for local trace verification) |
+
+
+## Telemetry Debugging Notes
+
+> **`RUST_TRACE_LOG=debug` is required for trace export.** The default (`info`) drops HTTP spans before they reach the OTLP exporter — nothing arrives at the trace backend.
+
+Metrics are pushed by a `PeriodicReader` every **30 seconds**. Allow ~35s after the first request before data appears downstream.
+
+**Stable log prefixes for grepping** (use these to scope log searches by boundary):
+
+| Prefix | Boundary |
+| --- | --- |
+| `claims_layer` | JWT validation failures |
+| `user_config_store_layer` | Config lookup / Redis errors |
+| `virtual_host_config_layer` | Unknown virtual host |
+| `AuthorizedCallValidator::validate` | Post-session MCP validation |
+| `initialize:` | Backend session creation |
+| `call_tool` | Tool routing and backend invocation |
+
+**Debugging by symptom:**
+
+| Symptom | Where to look |
+| --- | --- |
+| `401` | `claims_layer` logs: missing/invalid token, unsupported algorithm, no decoder key |
+| `400` config error | `user_config_store_layer` logs + Redis content for the JWT subject |
+| `404 Server not found` | `virtual_host_config_layer` debug: requested vhost id vs caller's config |
+| MCP routing errors | `AuthorizedCallValidator::validate` debug, then `call_tool`/`read_resource`/`get_prompt` warns |
+| Backend failures | `initialize:` warns for failed backends; routed-call warns name the failing backend |
+| Plugin problems | CPEX pipeline error logs; invalid reload marks runtime failed |
diff --git a/_context/wiki/deployment.md b/_context/wiki/deployment.md
@@ -0,0 +1,63 @@
+# Deployment
+
+## Checklist
+
+1. Front door routes only `/contextforge-rs` to the dataplane.
+2. JWT verification key/secret in place and rotated with the control plane's signing key.
+3. Redis reachable; TLS/mTLS across trust zones; write access restricted to the control plane; `DATAPLANE_PUBLISHER=true` on the control plane.
+4. Upstream connection mode matches backend URL schemes.
+5. One replica per `Mcp-session-id` (single replica or sticky routing).
+6. `with_tools` feature **disabled** in the production build.
+7. Telemetry export pointed at the collector.
+8. System limits raised: `nofile 65535`, TCP tuning (`tcp_fin_timeout=15`, widened local port range).
+
+## Health Endpoint
+
+**`/contextforge-rs/health` is a `with_tools` bootstrap helper only.** Production builds compile it out. Use TCP-level liveness checks or the exported metrics until a real health endpoint exists.
+
+## nginx Front-Door Routing
+
+Reference `docker/nginx.conf` split:
+- `location ^~ /contextforge-rs` → proxies to the gateway.
+- All other traffic (UI, management, SSE, legacy MCP) → control-plane.
+- Upstream retries on `error timeout http_502/503/504`: 2 tries, 10-second window. Non-idempotent MCP `POST` bodies are not re-sent after they reached an upstream — only connection-stage failures retry.
+
+## Session Affinity And Failover
+
+Backend MCP sessions are **local process state** — see [routing.md](routing.md).
+
+- >1 replica requires sticky routing by `Mcp-session-id`. The reference nginx config does not provide this; safe shapes today are a single replica or a front door with stickiness.
+- On restart or failover, all sessions are lost. Design clients to treat session-not-found as "reinitialize", not "retry".
+
+## Redis Availability
+
+- Redis is required at startup and on every uncached config lookup.
+- Connection manager retries 1,000 times (rather than failing fast).
+- In-process cache (default 60s) rides out short Redis blips for warm subjects.
+- A cold subject during a Redis outage fails at `user_config_store_layer` → `400` until Redis returns.
+
+## Images
+
+- CI publishes `ghcr.io/<owner>/contextforge-gateway-rs:<version>` (Cargo package version as tag).
+- **No `latest` tag — always pin the version.**
+- Builder: `rust:1.96.1` in `docker/Dockerfile`.
+
+## Config Propagation Delay
+
+```text
+worst-case staleness = publisher interval + user-config cache expiry
+```
+
+Both default to ~60s. For functional tests, shorten the publisher interval and disable the cache. For throughput benchmarks, keep both at 60s.
+
+
+## Security Posture
+
+| Concern | Current state |
+| --- | --- |
+| JWT revocation | None. A leaked token is valid until `exp`. Rotate the key and restart to invalidate. |
+| CORS | Wide open (any origin, method, header). Bearer-token based + cookie-free → no CSRF risk, but expect tightening as policy work lands. |
+| Local bootstrap routes | `/contextforge-rs/admin/tokens/{user}`, `/admin/userconfigs/{user}`, `/health` are **outside auth middleware — unauthenticated by design.** Only exist with `with_tools`. Production builds must not enable `with_tools`. |
+| Redis trust | Whoever can write Redis controls routing (arbitrary backend URLs receive caller traffic) AND which registered plugin hooks execute on payloads. Protect with TLS/mTLS and restrict write access to the control plane. |
+| Downstream TLS | Optional. Plain HTTP is acceptable only behind a trusted front door on a private network. Identity is always the bearer JWT, not mTLS. |
+| Plugin code | Fully trusted, in-process. Redis config activates compiled-in factories only — it cannot inject new Rust code. |
diff --git a/_context/wiki/failure-modes.md b/_context/wiki/failure-modes.md
@@ -0,0 +1,58 @@
+# Failure Modes
+
+**Rule:** failures come from the layer that owns the missing fact. Identity/config failures are HTTP responses before MCP handling; routing/backend failures are JSON-RPC errors.
+
+## HTTP Layer (middleware, before MCP)
+
+| Failure | Response | Layer |
+| --- | --- | --- |
+| Path doesn't match `/servers/{id}/mcp` | `400` | `virtual_host_id_layer` |
+| Missing `Authorization` / non-`Bearer` scheme | `401` | `claims_layer` |
+| JWT undecoded, unsupported algorithm, no key | `401` | `claims_layer` |
+| Expired token, wrong issuer/audience | `401` | `claims_layer` |
+| No user config for `claims.sub`, or claims absent | `400` | `user_config_store_layer` |
+| Config store error (not missing) | `500` | `user_config_store_layer` |
+| Virtual host id absent from caller's config | `404` `{"detail":"Server not found"}` | `virtual_host_config_layer` |
+
+## MCP Validation (defense-in-depth, normally unreachable)
+
+| Failure | JSON-RPC error |
+| --- | --- |
+| Missing session id / config / vhost / claims extension | Internal error (`Routing problem...`) |
+| Virtual host absent from user config | `RESOURCE_NOT_FOUND` `No configuration` |
+
+## Routing
+
+| Failure | Behavior |
+| --- | --- |
+| Prefixed name doesn't start with backend name + `-` | Internal error |
+| No backend entry matches split name | Internal error (`got no responses from backends`) |
+| Backend entry exists but no running service | Internal error (backend failed during initialize) |
+| More than one backend entry matches | `INVALID_REQUEST`; session backend entries cleaned up |
+| Undecodable pagination cursor | `-32602 Invalid params` |
+
+## Backend Session
+
+| Situation | Behavior |
+| --- | --- |
+| Backend unreachable during `initialize` | Stored with no running service; initialize still succeeds |
+| Backend unreachable during routed call | Call returns internal error; other backends unaffected |
+| Gateway process restart | All session state lost; clients must re-run `initialize` |
+| Request lands on wrong gateway node | List returns empty; routed calls fail — need sticky routing |
+
+## Plugins
+
+| Failure | Behavior |
+| --- | --- |
+| Plugin denies call/response | Becomes MCP error to caller |
+| Soft plugin error | Logged; call proceeds |
+| Invalid plugin config on reload | Runtime marked failed; plugin calls return internal MCP error until valid config applied |
+
+## Config Store (Redis)
+
+| Failure | Behavior |
+| --- | --- |
+| Redis connection loss | Connection manager retries (1,000 configured) |
+| User config missing | `400` from `user_config_store_layer` |
+| Redis `GET` error | Reported as missing → `400` |
+| Undecodable config / key encoding failure | `500` |