#84: Headroom × sandcastle: Phase 1 validation — measure token savings on a live run

[lsfera]

Closes #84

Implemented autonomously by the AFK orchestrator in an isolated, git-isolated sandbox.

Commits: 4f3335a

[lsfera]

AI review: changes-requested

The PR delivers useful infrastructure (unit tests for context-compressor.ts, proxy-injection tests in sandbox-runner.test.ts, env-at-call-time refactor, and savings logging) but does not satisfy the core mandate of issue #84, which is Phase 1 validation via a live run. Three of four acceptance criteria are unmet: AC1 requires evidence that conservative mode completed a real issue without regressions; AC2 requires raw, measured token savings numbers (not just logging instrumentation); AC3 requires documented results from an aggressive-mode run including any over-compression observations. Only AC4 (default HEADROOM_MODE=off confirmed) is met by the unit tests. Additionally, a test comment incorrectly states that headroom-ai is not installed in the test environment when it is in fact listed as a runtime dependency in package.json.

---

.sandcastle/context-compressor.test.ts:103 — The comment says "headroom-ai is not installed in the test environment", but headroom-ai is a listed dependency in package.json and will be present after npm install. The assertion (result instanceof Promise) is valid regardless — async functions always return a Promise before any dynamic import resolves — but the stated justification is wrong and will mislead future readers. Update the comment to explain that async functions always return a Promise synchronously, so the assertion holds whether or not the import succeeds.

(Posted manually — gh REST rejects REQUEST_CHANGES reviews from a PR's own author; see the durable-fix discussion for this.)

[lsfera]

Phase 1 validation results (live, #84)

Ran two real live sandbox issues through SandboxRunner.runIssue() — one with HEADROOM_MODE=conservative, one aggressive, using this branch's fixed code.

AC1 — conservative completes without regressions: PASS. Live run finished in 1 commit, exactly the requested file, no scope drift.

AC3 — aggressive tested, over-compression documented: PASS (no scope drift observed). The aggressive-mode issue included explicit negative constraints ("don't touch other files, don't run tests, don't add deps") specifically to probe whether compression could strip guardrail text. It didn't — all constraints held, 1 clean commit, correct file only.

AC2 — token savings measured (raw numbers): measured, and the number is 0%. Both compress() calls returned the prompt byte-for-byte unchanged:

[context-compressor] mode=conservative 2563→2563 chars (−0, −0.0%)
[context-compressor] mode=aggressive   3465→3465 chars (−0, −0.0%)

This isn't a fluke of these two runs — queried headroom's own /stats afterward and across all 14 real API requests logged this session (spanning this validation plus the earlier #97 proxy-routing verification):

compressions_by_strategy: {}          (empty — never fired, ever)
compression_cache.total_tokens_saved: 0
cost.total_tokens_saved: 0            (of $0.627 total spend)
agent_usage.totals.savings_percent: 0.0

The only nonzero discount anywhere is prefix_cache.discount_usd: $0.4988 — Anthropic's own native prompt-caching, unrelated to headroom's compression engine.

Root cause (from headroom's own startup banner):

License:      OSS (no license key)
Code-Aware:   DISABLED  (install headroom-ai[code] to enable)

Headroom's compression is turn-based/staleness-driven (HEADROOM_COMPRESSION_STABLE_AFTER_TURN / HEADROOM_STALE_READ_COMPRESS_AFTER_TURNS), and the code-aware strategies aren't available without a license. A one-shot prompt with no accumulated turn history, sent to a proxy whose main compression strategy is disabled, has nothing for it to compress.

Conclusion: the integration is wired correctly end-to-end (prompt reaches headroom, live session traffic reaches headroom, stats are tracked accurately) — but in this repo's current unlicensed/OSS deployment, headroom compresses nothing. Filing a follow-up issue to decide whether to pursue a licensed tier or tune the turn-threshold settings before revisiting. Merging this PR on the strength of AC1/AC3 passing and AC2/AC4 being honestly measured (0% and off-by-default, respectively) rather than left as an estimate.