Prompt compression shortens what you send a model while trying to keep the answer the same. Token pruning (LLMLingua, Selective Context) deletes low-information tokens; summarization rewrites the text shorter. Most results come from QA and RAG, and code is less forgiving: one dropped token can break an identifier or a diff. For coding agents, judge it by tasks solved and total cost, not compression ratio.
Three families
Only two of them work with closed models like Claude or GPT. Soft prompts need model weights.
| Family | Examples | Output | Closed APIs |
|---|---|---|---|
| Token pruning (extractive) | LLMLingua, LLMLingua-2, Selective Context | Subset of original tokens | Yes |
| Summarization (abstractive) | LLM-written summaries, agent compaction | New, shorter text | Yes |
| Soft prompts | Learned compressed embeddings | Vectors, not text | No, needs model internals |
How token pruning works
A small model scores tokens and the least informative ones go. The output is a strict subset in original order, so nothing is invented, but it reads like telegraph shorthand.
- 1Selective Context
Drops the least surprising words and phrases, betting the target model can infer them.
- 2LLMLingua
Adds a budget controller that compresses examples harder than instructions and the question.
- 3LongLLMLingua
Makes scoring aware of the question, for long contexts.
- 4LLMLingua-2
An encoder classifies each token keep or drop, trained on labels from a larger model. Faster and less task-bound.
Summarization
This is what /compact does to a whole session. It reads cleanly but can lose an exact error, a path or why an approach was rejected, and a large model makes it slow and costly. Smaller dedicated models fix the cost: Adeline 1 kept 94.2% of atomic facts at 90.2% fewer tokens on agentic coding traces, within a point of its Claude Opus teacher.
Why code is fragile
Prose is redundant. Code is not, in four ways.
- Identifiers are atomic. Drop one token of
getUserByEmailand the agent may call a function that does not exist. - Diffs need markers. A dropped
-line or shifted hunk header no longer applies. - Whitespace can be syntax. In Python and YAML, indentation is meaning.
- Errors are needles. A scorer tuned on prose may rate the one failing line as unremarkable.
Compress around code, not through it. For tool output, see how to compress test logs, diffs and grep results.
Measure these, not the ratio
- 1Task success
Same tasks, several runs each, graded against an uncompressed control, as in our quality evaluation.
- 2Cost with caching
Rewriting the prefix forces cache writes, which cost more than input. See caching vs compression.
- 3Rate control
Ask for 50% and check you get 50%. Threshold settings make budgets hard to enforce.
Published model results
On LongBench v2, Helene 1 on auto tied LLMLingua-2 for the top score (49.3%) while removing 1.7 times as many tokens. On code and structured data it beat the uncompressed control.
On Lost in Compression, Helene 1.1 removed 50.5% when asked for 50%, with no novel tokens and source order kept on every document. In condense, Helene trims the newest turn and Adeline packs finished loops; run it with dense claude, dense codex or dense opencode (coding agent docs).
Frequently asked questions
What is LLMLingua?
A family of prompt compression methods from Microsoft Research that uses a small model to find and remove the least informative tokens. LLMLingua-2 reframes this as token classification trained on distilled data, which is faster and task-agnostic.
Can I use LLMLingua with Claude or GPT models?
Yes. It runs locally as a Python library before any API call, so you compress the prompt yourself and send the result.
Does prompt compression work on code?
It can, but identifiers, paths, diff markers and indentation all carry meaning. Leave code, patches and errors intact and compress the prose and stale history around them.
What compression ratio should I use?
Pick the most aggressive ratio that does not reduce tasks solved on your own work. Check that the tool delivers the ratio you ask for, since many do not.
Is summarization better than token pruning?
They fail differently: pruning keeps wording but leaves fragments, summarization reads cleanly but can paraphrase details away. Many systems prune fresh input and summarize finished history.