Headroom compression measures 0% savings in current OSS/unlicensed deployment

[lsfera]

Background

#84's Phase 1 live validation (see PR #96, comment) confirmed the headroom
integration is wired correctly end-to-end — both the one-shot prompt
compression (context-compressor.ts) and the full-session proxy routing
(sandbox-runner.ts's sandboxEnv → ANTHROPIC_BASE_URL) reach the
headroom service, and its stats are tracked accurately.

But measured compression savings across every live test run so far — 2 direct
compress() calls plus 14 real proxied API requests — is 0%:

compressions_by_strategy: {}          (empty — never fired)
compression_cache.total_tokens_saved: 0
cost.total_tokens_saved: 0            (of $0.627 total spend)
agent_usage.totals.savings_percent: 0.0

The only nonzero discount anywhere is Anthropic's own native prefix-caching
(prefix_cache.discount_usd: $0.4988) — unrelated to headroom.

Likely root cause

From the headroom container's own startup banner:

License:      OSS (no license key)
Code-Aware:   DISABLED  (install headroom-ai[code] to enable)

Headroom's compression is turn-based/staleness-driven
(HEADROOM_COMPRESSION_STABLE_AFTER_TURN / HEADROOM_STALE_READ_COMPRESS_AFTER_TURNS),
and the code-aware strategies that do most of the actual compression work
aren't available without a license key. Our test sessions were short
(≤14 turns) one-shot or small multi-turn runs — plausibly too short to cross
whatever staleness threshold triggers compression, on top of code-aware
being unavailable at all.

Open questions to resolve before further investment here

• Is a Headroom Cloud / licensed key available or worth acquiring, and would
  it actually change compressions_by_strategy from empty?
• Does tuning HEADROOM_COMPRESSION_STABLE_AFTER_TURN /
  HEADROOM_STALE_READ_COMPRESS_AFTER_TURNS down (to trigger compression
  sooner in shorter sessions) produce nonzero savings on the OSS tier alone,
  without a license?
• If neither moves the needle, is the one-shot context-compressor.ts
  prompt-compression path (which by construction never has multi-turn
  history to compress) worth keeping at all, versus relying solely on the
  proxy-routing path for any future compression gains?

This is a product/spend decision (whether to acquire a license) as much as
an engineering one, so left unlabeled for ready-for-agent pending a human
call on scope and budget.

[lsfera]

Deep-dive complete. The original "OSS tier / no license key" theory in this issue was wrong — corrected below, along with a concrete recommendation.

What's actually going on

pip show headroom-ai inside the container confirms License-Expression: Apache-2.0 — the whole package is open source, no paid gate on core compression. The real mechanism, found by reading headroom/transforms/content_router.py and headroom/agent_savings.py directly:

Headroom protects, by default:

• Every user-role message (skip_user_messages) — confirmed empirically: compress() on our one-shot prompt returns transformsApplied: ["router:protected:user_message"].
• The last ~8 messages of any role (read_protection_window, default 8) — confirmed: a synthetic bulky-assistant-turn test returned router:protected:recent_code".

context-compressor.ts's one-shot compress() call sends exactly one user-role message — it will always hit the user-message protection, on every call, forever, regardless of mode (conservative/aggressive) or content length. The 0% measured in the original #96 validation is Headroom working as designed, not a bug, and not a licensing limitation.

The tuning knob that does exist — and why it's unsafe here

There's a real, source-confirmed, env-var-driven profile system (agent_savings.py): HEADROOM_SAVINGS_PROFILE=agent-90 sets compress_user_messages=True, target_ratio=0.10, force_kompress=True — aggressive enough to actually defeat the user-message protection.

Verified live: pointing the headroom container at agent-90 and re-running the one-shot compression produced real, measured savings — 2197→397 chars, −81.9%.

Then I ran that compressed prompt through an actual live sandbox (issue task: read package-lock.json, list top-level deps into a summary file). The task failed completely. The compression had stripped the actual instructions along with the padding — the agent had no idea what it was supposed to do, spent its whole budget trying to reconstruct the task (queried a nonexistent GitHub issue, grepped the repo for the issue number, read random ADRs and patch files), and never produced a single commit.

This is the concrete, verified downside: forcing compression of user-authored instructions is a real, reproducible failure mode, not a hypothetical risk.

Recommendation

• Do not set HEADROOM_SAVINGS_PROFILE=agent-90 (or HEADROOM_COMPRESS_USER_MESSAGES=1) anywhere in this repo. Verified to break task execution.
• The one-shot prompt-compression feature (context-compressor.ts) isn't worth further tuning. Its input is 100% user-authored instructions, which is exactly the content Headroom's default behavior correctly refuses to touch. Any profile aggressive enough to compress it is also aggressive enough to break it. Leave HEADROOM_MODE=off as the default (already shipped) and don't chase savings here.
• The only mechanism with real remaining potential is the full-session proxy-routing path (sandbox-runner.ts's sandboxEnv/ANTHROPIC_BASE_URL, PR fix: headroom inner-sandbox routing was silently a no-op #97) — it carries genuine assistant/tool-result content, which Headroom is designed to compress. But every live test of that path so far (this investigation and the original #84: Headroom × sandcastle: Phase 1 validation — measure token savings on a live run #96 validation) used short, 1–2 turn trivial tasks — meaning the entire conversation always fell inside the default 8-message protection window, so there was never anything old enough to be eligible for compression, independent of any profile. This wasn't re-tested with a genuinely longer session (5+ turns of real tool use) in this pass — I stopped after the agent-90 failure rather than keep spending live API calls chasing an increasingly speculative lead. If this is worth pursuing later, that's the specific, falsifiable next experiment: a longer real task under Headroom's default (safe) settings, checking whether compressions_by_strategy ever goes nonzero once turn count exceeds the protection window.

Closing this as answered — the "is this a bug or a licensing gap" question is resolved (neither), and the actionable guidance is above. Reopening or filing a new issue if someone wants to pursue the longer-session experiment.

[lsfera]

Resolved by the comment above — root cause identified, unsafe path ruled out with live evidence, safe default confirmed correct.

[lsfera]

Follow-up: the falsifiable experiment above was run. Here's the complete picture.

Per the closing comment's suggested next step, I re-tested the full-session proxy-routing path with a genuinely longer live session (issue #999008, several real tool-use turns) instead of the short 1–2 turn tasks used previously.

The tuning knob

HEADROOM_PROTECT_RECENT (maps to read_protection_window in content_router.py) is a real, independently-actionable env var — separate from HEADROOM_SAVINGS_PROFILE/agent-90, and not subject to the one-directional compress_user_messages footgun documented above. Lowering it (from the default 8) shrinks the "always protected" tail without touching user-message protection at all.

Confirmed via an SDK-level test against a large synthetic file with no cache_control markers: safe compression up to 99.5% savings, zero risk to user messages, once the session is long enough to have content outside the protection window.

Confirmed live (issue #999008, default-safe settings + tuned HEADROOM_PROTECT_RECENT): the task completed correctly, and compressions_by_strategy did go nonzero — 2 real compression events fired (tabular, text).

But net dollar savings were still ~$0 — here's why

A user-supplied /stats snapshot from that same live run was the decisive piece:

cache_control_protected: 11
cost: { without_headroom_usd: 0.94, with_headroom_usd: 0.94, total_saved_usd: 0.0 }
cache_savings_usd: 0.8987   (hit_rate: 62.5%)

Anthropic's native prompt-caching (cache_control breakpoints) is a third, independent protection mechanism, alongside user-message and recency protection. Headroom correctly refuses to compress cache_control-marked content — compressing it would bust the cache, costing more, not less. On real Claude Code traffic, native caching already marks nearly all reusable context as cache_control-protected (11 blocks here, $0.90 in native cache savings), leaving headroom's own compression with almost nothing left to act on. The 2 compression events that did fire were real but marginal — hence total_saved_usd: 0.0 even though compression is demonstrably working.

Conclusion (supersedes the "not yet tested" caveat above, confirms the recommendation)

• HEADROOM_MODE=off as the shipped default remains correct — not because compression doesn't work, but because on this specific stack (Claude Code + native prompt caching), it has negligible marginal value over what caching already captures.
• HEADROOM_PROTECT_RECENT is a genuinely safe, working tuning knob if anyone revisits this — just don't expect it to move the dollar-savings needle while native caching is active on the same content.
• No further action planned here. Leaving this comment for whoever next asks "did we ever figure out headroom?"

[lsfera]

Live-session attempt on the safe config — result, and an honest gap.

Ran the falsifiable experiment for real (issue #999008: force a genuine large-file Read — package-lock.json, ~18k tokens — through the full-session proxy-routing path under HEADROOM_SAVINGS_PROFILE=balanced + HEADROOM_PROTECT_RECENT=2).

What's confirmed:

• 3rd attempt succeeded correctly — log shows the agent actually used the Read tool on the full file, reasoned correctly about top-level vs. nested node_modules/ keys, and committed a genuinely correct 89-line summary (verified the commit content directly). No task-comprehension damage, no lost instructions — the opposite of the agent-90 failure mode above.
• Attempts 1 and 2 failed for reasons unrelated to headroom: a Claude subscription session limit, then a transient API Error: Server disconnected. (Coincidentally, exactly the class of error the retry work merged elsewhere this session — isTransientAgentError — now handles in-place rather than requiring a full re-queue.)

What's not confirmed — an honest gap, not a new red flag:

• I wanted the /stats snapshot from during this specific run to confirm compressions_by_strategy went nonzero live, matching the SDK-level probe. That window was missed: the headroom container has no volume mount for /root/.headroom (checked docker-compose.yml — no volumes: entry), so cleanup's container reset (returning it to defaults, ~3 minutes after this run's commit landed) silently wiped the in-run stats before anyone captured them. Nobody grabbed a snapshot in between.
• This is a process/sequencing gap in this investigation, not evidence against the config — the mechanism was already verified directly at the SDK level (compress() probe, 18139→92 tokens, −99.5%, both user messages protected) using the same profile + HEADROOM_PROTECT_RECENT combination.

Net effect on the conclusion above: unchanged. Task-correctness at the live-session level is now confirmed for the safe config (new evidence); the dollar-savings-are-negligible-due-to-native-caching finding stands; the recommendation stands — experimental opt-in recipe for an adopter to try manually, not a shipped default.