System-III-Router

A learned regulator that decides how hard to think, built from Critique of Agent Model (arXiv:2606.23991, Xing, Deng, Hou).

Cumulative pseudo-regret over a 2,000-task stream (lower is better). The Configurator (learned, System III) hugs the oracle floor; every fixed deliberation depth pays a standing regret tax. Regenerate: python scripts/run_demo.py.

The idea

The paper's central distinction is agentic vs agentive: today's agents are agentic — competence lives in external scaffolding (fixed tools, fixed workflows) — not agentive, where the agent regulates itself. Its most concrete proposal is System III, the Configurator: a learned controller that, each step, decides how much to deliberate instead of always running a fixed depth. By Theorem 3, fixed-depth deliberation is provably wasteful — easy problems need none, hard ones need a lot — which is the gap a learned regulator fills.

This repo makes System III runnable on math reasoning. The "arms" the regulator chooses among are deliberation modes, mapped onto the paper's cognitive stack:

mode	paper	what it does	cost
direct	System I (Actor)	answer, no scratchpad	cheapest
cot	System II (light)	one chain of thought	medium
plan_verify	System II (Simulative Planner)	decompose → solve → self-verify	most

It is, underneath, a cost-aware contextual bandit: reward = correctness − λ·tokens, metric = cumulative regret over a streaming task sequence. The Configurator keeps per-difficulty success/cost statistics and learns "easy → think less, hard → think more". That online statistic update is the fast loop of the paper's fast-slow learning (Theorem 1): revising the self-model within the round is what drives regret below any fixed policy.

Quickstart (offline, free, no API key)

pip install -r requirements.txt        # or just: the demo needs only matplotlib
python scripts/run_demo.py             # 2,000-task synthetic stream

strategy                              accuracy  avg_reward  avg_tokens  cum_regret
Fixed: direct (System I only)           64.4%       0.632          25       286.5
Fixed: cot (System II light)            83.0%       0.751         158        42.6
Fixed: plan+verify (always deep)        90.1%       0.665         472       208.2
Configurator (Thompson, learned)        86.3%       0.770         186         8.3   <- learned
Oracle (zero-regret floor)              87.2%       0.776         194         0.0

>> Learned self-regulation cuts cumulative regret by +80.5% vs the best fixed depth (cot).

Read the two stories in that table:

• Regret. The learned Configurator (8.3) nearly matches the oracle (0.0) and beats every fixed depth — the paper's claim that self-regulation must be learned, not fixed, reproduced in miniature.
• Efficiency. It reaches near-plan_verify accuracy (86% vs 90%) at ~40% of the tokens (186 vs 472), and beats cot on both accuracy and reward. "Deliberate only when it pays" is the whole point.

How the code maps to the paper

Module	Role in the C→A→F→C loop	Paper concept
src/tasks.py	Context: task → difficulty key	the context the regulator conditions on
src/modes.py	the three deliberation depths	System I / II machinery
src/experience.py	per-(difficulty×mode) Beta + token stats	fast self-model update (Thm 1)
src/configurator.py	Action: pick a depth	System III, the Configurator
src/providers.py	run the chosen depth (mock or live)	the solver
src/grading.py	Feedback: correctness − λ·tokens	the reward
src/loop.py + src/eval.py	streaming regret evaluation	performance / efficiency / growth
src/plan_decoder.py	Action by generation: text → deliberation plan, no featurizer	SkillComposer core (2606.32025)
src/plan_configurator.py + src/bottleneck.py	the featurizer-bottleneck experiment	—

Run live against a 9router gateway on Vi-GSM8K

cp .env.example .env, fill in OPENAI_API_KEY, OPENAI_BASE_URL, SYS3_SOLVER_MODEL, then:

python scripts/run_live.py 30                         # Vi-GSM8K from the Hub
python scripts/run_live.py 30 --jsonl data/vi.jsonl   # a local export
python scripts/run_live.py 30 --hf openai/gsm8k       # plain GSM8K

One fixed solver model answers every task; the Configurator only changes how hard it thinks. With no difficulty labels there is no oracle regret live — the result is the accuracy-at-lower-token-cost frontier on real problems.

Does routing actually pay? (cached A/B)

scripts/run_cached_eval.py runs every (task, mode) once, caches the outcomes, then replays the Configurator over the cache for free — recovering a true oracle and real regret, and letting you A/B featurizers at zero API cost:

SYS3_SOLVER_MODEL=cc/claude-haiku-4-5 python scripts/run_cached_eval.py 24

What 24 matched Vi-GSM8K tasks show — the answer depends on solver strength:

solver	best fixed	learned Configurator	verdict
haiku (weak)	cot regret 5.96	cot_steps feat → 5.28	routing beats fixed
opus (strong)	cot regret 0.74	every feat → 0.74	routing can't fire

For the weak solver the hard tail genuinely needs plan_verify, and an execution-grounded difficulty feature finds it. For the strong solver, no surface feature predicts which problems it one-shots — so a hand-featurized bandit is stuck, and only a learned policy (see train/) can capture the gain. The dataset's own difficulty flag is a curation filter (near-constant), so it does not help — a clean negative result.

The featurizer is the bottleneck — a generative fix (SkillComposer)

The cached A/B hints at it; this makes it explicit. A bandit's ceiling is often not the learning rule but the featurizer that turns a task into the key it conditions on. src/plan_decoder.py ports the core of SkillComposer (arXiv:2606.32025) to the depth axis: instead of featurize → bucket → bandit, a tiny constrained autoregressive decoder (~a few thousand params, numpy, trained by SFT) generates the deliberation plan straight from the task text — no hand-built featurizer.

python scripts/run_plan_decoder.py

On a diagnostic stream where difficulty is signalled by a content word while digit-count and length are held constant, count-based featurizers go blind — they bucket every task as "easy":

strategy                                accuracy  avg_tokens  cum_regret
Bandit (featurizer=proxy, blind)          84.0%         151        47.9
Bandit (featurizer=oracle label)          87.2%         182        17.1
PlanDecoder (SFT, reads text)             87.5%         180         0.0   <- learned
Oracle (zero-regret floor)                87.5%         180         0.0

Two readings:

• Bottleneck. Same bandit, swap the featurizer (blind proxy → oracle label) and cumulative regret jumps 2.8× (47.9 → 17.1). The learner never changed — the featurizer was the ceiling.
• Fix. A decoder that reads the text recovers oracle routing (0.0) with no featurizer at all: feature engineering replaced by generation. On the original stream (where the proxy is already decent) it still closes the residual 1.6× gap without regressing — so this isn't a cherry-picked stream.

The same decoder emits genuine multi-step plans (subset + count + order), which is where it earns its keep on super-agent's skill axis; on the single depth choice here the plan is length 1. A frozen pretrained encoder (Qwen3-Embedding in the paper) is the drop-in upgrade for messy real text.

Roadmap — toward a trained Configurator (the agentive step)

The bandit above is the externalized, training-free System III. The paper wants the regulator inside the model. The next build keeps everything here as the eval harness and adds a learned policy:

1. SFT the mode head. Use the oracle labels (cheapest mode that solves each problem) as targets: input = problem, output = <mode> token + solution.
2. GRPO the regulator (your existing Qwen3-4B pipeline). One policy emits the mode then the answer; reward = correctness − λ·tokens — the same objective, now shaping the weights instead of a Beta posterior. The model learns to spend test-time compute only where it lifts accuracy.
3. Measure growth. Re-run this regret harness on held-out Vi-GSM8K; the win is the learned policy moving from the cot-baseline point toward the oracle frontier as training proceeds.

Tests

python -c "import pytest, sys; sys.exit(pytest.main(['-q']))"

25 tests lock in the behaviors, including the central claim (test_learned_beats_every_fixed_depth_on_regret — the learned Configurator posts lower cumulative regret than any fixed depth) and the featurizer-bottleneck fix (test_featurizer_is_the_bottleneck_and_plandecoder_fixes_it).