A model with cheaper tokens can still cost more if it needs extra turns and more context to finish the same job. So measure cost per completed task, not just price per token. That was the lesson we wanted people to take from a breakfast discussion with Earlybird Venture Capital in London on 28 October.
Before the event, Akash Bajwa, a Principal at Earlybird, laid out the problem. The common promise is that moving to open source shrinks your inference bill. Akash's point was that in practice it often doesn't. The output tokens look cheaper, but open models use more tokens, and cost per task tends to come out higher.
A coding agent sends its whole history back to the model on every turn, so an extra turn re-reads everything that came before it. A task's cost depends on how many turns it takes and how much context each turn carries. Neither is on a price list.
You can see this in how a bill splits. In the anatomy of a token bill, cache reads were 67.7% of spend across twelve real coding sessions. Output was 13.2%. Those figures come from that post, not from the London discussion, and they don't compare open and frontier models. They do show how much of a bill an output price leaves out.
A made-up example
We invented these numbers and rounded them to keep the sums easy. They aren't from the event, and they aren't from a real model.
| Measure | Model A | Model B |
|---|---|---|
| Price per million tokens | $1 | $2 |
| Turns to finish the task | 20 | 10 |
| Tokens per turn | 150,000 | 100,000 |
| Total tokens | 3 million | 1 million |
| Cost of the task | $3 | $2 |
Model A is half the price per token, and the task costs 50% more. Swap the turn counts and Model A wins easily. The token price can't tell you which case you're in.
How to measure it
The session was planned around benchmarks like cost per PR and cost per work unit. Any unit works if it means the work got done.
- Pick a unit of work
- A pull request, a ticket, a finished job. It has to be something you'd call done.
- Count the whole run
- Every turn, retry and failed attempt goes in the total, not just the attempt that worked.
- Keep the job the same
- Give each model the same tasks and the same definition of done. Divide the total spend by the tasks it finished.
Once you have that number, you can work on it. Compaction, gateway routing and model routing were all on the agenda in London. It's worth checking the agent too. When we dug into Axiometa's agent together, we found avoidable steps and failed calls.
Thanks to Akash and to Anna Andersson, Earlybird's Head of Community & Events, for hosting our CTO, Povilas Nagrockas-Bandzaitis, and for a thoughtful discussion.