On 30 July we sponsored the London AI Cost Hack, an evening at All Saints Church near King's Cross. Kaashif Hymabaccus hosted it. Merge and Banyan Flashcards sponsored it with us, and Banyan brought the pizza.
The tagline was Lowest Cost Wins. The winner wasn't the lowest cost.
The challenge was PatchGuard: build an AI reviewer that stops dangerous changes before they merge. Each case was a pull request with its diff, CI logs, migrations and earlier discussion. The reviewer had to return a risk level, findings with evidence, a verification plan, and a decision to approve, request changes or block.
All model calls went through Merge Gateway, and the spend it recorded was the official cost. condense was optional. People could run their coding agent through it, or call models through its proxy API.
Entries ran against 10 public cases and 10 hidden ones, which were longer and noisier. The listed rule was that the cheapest passing solution wins. The listed prize was £100 and a very firm handshake from a partner at a venture capital fund.
The results
Four entries made the published leaderboard. Three passed all 10 public cases. On the hidden set, none passed more than 6.
| Cost | Hidden-set score | Entry |
|---|---|---|
| $0.0006 | 27 | The cheapest |
| $0.0017 | 59 | Alexander Procelewski, the winner |
| $0.0039 | 32.5 | |
| $0.43 | 64.5 | The highest score |
Kaashif wrote that Alexander's solution didn't top the leaderboard on cost or on quality, but “no-one else was even close to winning on both metrics.”
You can see that in the numbers. Compared with Alexander's entry, the cheapest one saved about a tenth of a cent and scored 32 points lower. The top scorer gained 5.5 points and spent about 250 times as much.
It's a small sample: one evening, four entries, 10 hidden cases. But sorting by cost alone wouldn't have picked this winner, and neither would sorting by quality.