Guilty Until Proven

The Problem

Every context-compaction method makes the same two promises: zero quality loss, and an 80% smaller bill. But most benchmarks look nothing like your real work, and the savings ignore caching.

Our last post used minmax-bench to price the bill; here we add its other half: quality. It replays your own ~/.claude sessions and grades each step of the trajectory, not just the tokens. We frame it as three lies, each a flaw in how compaction is usually evaluated. Every experiment compares the same arms: vanilla (Claude Code out of the box), control (vanilla again, no compression), condense, and headroom.

The honest verdict on ourselves: past ~20 turns condense holds quality for 62% fewer tokens and 45% less cost, while tools advertising 80%+ cuts deliver ~15% of tokens and at best 14% of the bill. Every number here is one 200-turn session, not a ceiling; the 72% in the anatomy of a token bill came off a 938-turn one, and depth is what separates the two.

Lie #1: “your agent won’t notice”

Proving no quality loss is pointless on a benchmark that looks nothing like your workload: single-shot Q&A, RAG, and friends. An evaluation has to be faithful to the task it stands in for. Ours is long coding sessions, so we run only the long tasks from terminal-bench 2, the closest existing match.

This is the minimum gate any compaction engine must clear, and it is a weak one. Even vanilla runs flip tasks between solved and unsolved [1], and across 14 tasks the 95% intervals overlap almost entirely, so condense and headroom scoring above vanilla sits inside the noise, not above it. To rank harnesses we have to read the trajectory itself.

The obvious way to do that is to ask did the compressed agent take the same next action as the original run? It fails its own control. With no compression at all, a control run agrees with its own original only 10–46% of the time; a perfectly good run scores badly because two steps swapped order.

This is expected: LLM nondeterminism sends agents down a different, equally valid route each time. A single trajectory is a coin flip [2], and freezing temperature to 0 would only make the test unrealistic.

Replay every step, anchored against the floor

So we freeze the real session’s history up to each step, let each arm propose the next action, and score it against control as the noise floor: next-step prediction instead of next-token prediction:

Grade the intent

A judge rates each replayed step good, degraded, or bad by its progress toward the task, a faithful metric for agentic trajectories [3][4], so a different-but-correct action still gets full marks:

As hypothesized, a run can wreck exact match and still preserve the goal. LLM judges carry known biases and are unreliable alone, so we calibrate the way current research suggests: controlled rubric, normalized score [1][5]. Most importantly the judge catches compression’s biggest failure mode: amnesia: the model re-reading what it already knew, paying twice in latency and cost [6]. Those re-fetches are marked ↻ redundant and down-ranked.

Lie #2: “80% saved”, yet tokens aren’t dollars

Every compressor’s headline number is tokens removed. But rewriting history breaks the cache, which tokens you cut (input or output) changes the bill entirely, and savings mean nothing if the model then re-fetches what it forgot (our cost-focused post has the full accounting). headroom makes the gap plain: over the same windows as our own numbers, it trims a few tokens while the bill goes up.

The savings grow with the session

For condense, the deeper the session, the more dead history there is to remove:

Same curve the cache-aware cost study finds by chain depth, and the early “loss” is pennies: the opening ten steps cost $1.30 against $1.06, 23% more but a 24-cent gap. It breaks even around step 20 and ends at $18.53 against $33.61: 62% of tokens, 45% of cost, about $15 back on one session.

Which is why 45% here and the 72% in the anatomy of a token bill are the same result. That one ran 938 turns, deep enough to sit near the structural ceiling; this one is 200. That post’s own depth table brackets sessions this size at 47–55%, and we land at 45%, the same curve, read further down it. Quote either number as the savings and you are quoting a session length, not a product.

Lie #3: “you won’t feel it”

Quality holds and the bill drops, but there is a third thing you actually feel: the wait. Proxies love to quote a tiny number here (“under 100ms of added latency”) clocked on the proxy alone. That is not the number you feel. What you feel compounds over every turn of the session, so minmax-bench clocks it end-to-end.

Over the same 192-step window control averages 20.7s per step and condense 20.7s: flat, the overhead cancelling against a model reading 62% fewer tokens. Early on, though, condense is genuinely slower (+56% at step five, +21% ten steps in) because the overhead lands before there are enough dead tokens to offset it.

All axes, one comparison

A harness comparison is only honest holding quality, tokens, cost, and latency all at once:

The existing tools we ran through the same harness hold quality too. On quality, everyone ties. The bill is where the methods separate: ~15% of tokens and at best 14% saved, negative on several replays, against condense’s 62% / 45%. Cutting tokens is not the same as cutting the bill.

The verdict

Every method is guilty until proven, and the proof has to survive all four tests at once: a benchmark shaped like your real work, a judge that measures the right thing, a bill that counts the cache, and a clock that runs end to end. Held to that, condense keeps the agent on task for 62% fewer tokens and 45% less money at the same round-trip latency, while the tools promising 80%+ mostly hand back the cache tax instead. Honest, not flashy: lower than we first thought, which is the point.

And no row here is a ceiling. Sessions differ, models differ, the judge is still being calibrated; 45% is what one 200-turn session gave, where 938 turns gives 72% and a short chat costs a little more than it saves. That spread is exactly why we’re shipping the benchmark, not just the numbers: the only savings figure worth anything is the one measured on your sessions.

What’s next. The harness already flags work that is on us: condense re-fetches a little more than vanilla: 17 redundant calls to its 14. We want the benchmark user-contributed, more robust, and harder to game, so it stays impartial as it grows. We’ll keep turning it on every compression tool we can find, and on ourselves, and publishing whatever it says.

Debunk us

minmax-bench is open source and measures every proxy on equal footing. It reads your own ~/.claude sessions, replays them through control, condense, and headroom at once, and prints every table above, including the columns where condense loses.

shellgit clone https://github.com/condense-chat/minmax-bench
cd minmax-bench

# incremental: teacher-forced per-step replay of your own sessions,
# judged goal-based against the no-compression control floor
uv run minmax-bench quality incremental --arms condense,headroom --judge goal

# full trajectory: real tasks end to end in a sandbox, outcome + milestone
# coverage (spends real API credit, needs Docker)
uv run minmax-bench quality run --arms condense,headroom --milestones

Point it at a session that actually gets long; compaction has nothing to do on a 4k-token chat.

Run it on your own sessions and post the numbers, flattering or not. If it debunks a compaction method, ours included, the harness is doing its job. Tag us on X and we’ll reply either way.

Method. Two modes. Full trajectory: each arm (vanilla = no proxy; condense, headroom = live proxies) runs the real terminal-bench task end to end in a Harbor sandbox; solved = the task’s own verifier returns reward 1. Solve numbers are the 14 long tasks (author budget ≥30 min: dna-assembly, regex-chess, install-windows-3.11, train-fasttext, schemelike-metacircular-eval, …) on Opus-4-8: vanilla 23/34 (68%), condense 11/14 (79%), headroom 10/14 (71%); trial counts differ because vanilla runs k+1. Short tasks solve ~100% in every arm and never cross the compaction threshold. Milestone coverage: a judge extracts each task’s subgoals and scores how many the run reaches; condense matches vanilla where measured. Incremental: teacher-forced per-step replay: at each recorded decision point of a real Claude Code session the same prefix is sent through each arm (control = api.anthropic.com, no compression) and the replayed next action is scored. Same-action = exact structural match of the next tool call/answer against the original. Goal-quality = an LLM judge (rubric identical across arms) rating each step good/degraded/bad toward the task; a ↻-redundant re-fetch of already-surfaced context is down-ranked one notch. Control’s good-rate is a calibration check (target ≥90%; the clean 45-step run lands 93%, the deep run 58%, flagged noisy). Context, cost, and latency = averages and totals over the 192 steps every arm completed, control as baseline; latency is the mean wall-clock round-trip per replayed step (the proxy’s own overhead included). Deep run: one 200-step Opus-4-8 session (control vs condense: 46%/24% same-action, 58%/63% goal, 260k→99k context, $33.61→$18.53, 20.7s→20.7s latency). Depth trend: a 45-step Opus-4-8 session (~113k context, −27% tokens / −7% cost) and a 58-step Sonnet-5 session (~127k, −31% / −14%). Existing-tool numbers: headroom on the live cost bench (replay 202f98bd, claude-haiku-4-5, ~1,573 points) saved 15.1% of tokens and 13.8% of the bill, its kompress mode 11.7% / 1.8%; on three teacher-forced replays and one further cost run its bill came in 24–84% above control. Single runs, live proxies, real provider usage. The cache-aware production cost study is the anatomy of a token bill. All reproducible from the commands above.

References

1. Lù, Kazemnejad, Meade, Patel, Shin, Zambrano, Stańczak, Shaw, Pal, Reddy. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. 2025. arXiv:2504.08942.
2. Yao, Shinn, Razavi, Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. 2024. arXiv:2406.12045.
3. Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, Cobbe. Let’s Verify Step by Step. ICLR 2024. arXiv:2305.20050.
4. Zhuge, Zhao, Ashley, Wang, Khizbullin, Xiong, Liu, Chang, Krishna, Schmidhuber et al. Agent-as-a-Judge: Evaluate Agents with Agents. ICML 2025. arXiv:2410.10934.
5. Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, Zhang, Gonzalez, Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685.
6. Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. arXiv:2307.03172.