An AI gateway is one endpoint in front of model providers that handles routing, keys, budgets, rate limits and logs. It forwards each request as it is, so the request costs the same. Coding agent cost comes from huge, repeated context, so a smaller bill also needs something that shrinks the request.
What an AI gateway does
One endpoint replaces a handful of provider keys. LiteLLM is the best-known open-source option; managed gateways offer the same without a server.
- 1Routing
One API over several providers, with fallbacks and load balancing.
- 2Key management
Provider keys stay on the server. Revoke one person without rotating all.
- 3Metering and budgets
Spend per key, user, team or project, with limits and alerts.
- 4Rate limits and policy
Which models a team may use, how fast, sometimes redaction.
- 5Logging
Request records for debugging, audit and cost reports.
Proxy vs gateway
Every gateway is a proxy, not the reverse. A gateway saves money by saying no; it never makes an allowed request smaller.
| Need | Routing gateway | Compression proxy |
|---|---|---|
| One endpoint for many providers | Yes | Usually limited to supported routes |
| Central provider keys | Yes | Varies |
| Spend per person or team | Yes | Often, since it sees every request |
| Budgets and caps | Yes | Varies |
| Smaller requests | No, forwards as is | Yes, rewrites context |
| Lower cost per task | Only by blocking or rerouting | Yes, fewer tokens per turn |
Why agents need more
Agents re-send their whole context every turn, so requests never repeat and gateway caching misses. Provider prompt caching discounts those tokens but still bills them, and expires after about five idle minutes.
How agents connect
Both kinds plug in the same way: change the base URL. Check what a third party stores first, using our proxy safety checklist.
| Agent | Where to set it | Notes |
|---|---|---|
| Claude Code | ANTHROPIC_BASE_URL | Messages format; gateway key in ANTHROPIC_AUTH_TOKEN. Also Bedrock and Vertex direct. |
| Codex | ~/.codex/config.toml | Custom model provider with base URL and wire API |
| OpenCode | opencode.json | Per-provider base URL and headers |
What a coding team needs
Gateways are easy to over-buy. The two can stack, but every hop adds latency and another vendor that sees your code.
| You are | First pain | What fits |
|---|---|---|
| Solo on a subscription | Plan runs out | No gateway; shrink the context |
| Small team on API keys | Who spent what | Per-developer keys or telemetry, then budgets |
| Many apps, many vendors | Keys, audit, fallbacks | Routing gateway such as LiteLLM |
| Agent bill is the problem | Cost per task | Compression proxy such as condense |
Where condense fits
condense compresses earlier turns of long conversations and leaves code, patches and errors untouched. Short conversations pass through uncompressed.
- Formats. Anthropic Messages, OpenAI Responses and Chat Completions, plus Bedrock and Vertex AI and Gemini on Vertex on your own cloud account.
- Team controls. Organizations with per-member budgets, the enterprise dashboard, service keys with caps and budget webhooks.
- Control group. API traffic splits send a share uncompressed.
- Not a universal router. For dozens of vendors, use LiteLLM.
- Trust. Provider keys never stored, no training, 7-day deletion, zero retention on request. See the security page.
Frequently asked questions
What is the difference between an LLM proxy and an LLM gateway?
Both sit on the request path. A gateway is a control plane for routing, keys, budgets and logs; a proxy is anything in the path, including tools that change the request.
Can I use LiteLLM with Claude Code?
Yes. Set ANTHROPIC_BASE_URL to the LiteLLM proxy and give Claude Code a LiteLLM key. Anthropic's LLM gateway docs cover the setup.
Does an AI gateway reduce token costs?
Only indirectly, through budgets and routing. Agent turns are almost never identical, so gateway response caching rarely hits.
Do I need a gateway for a single developer?
Usually not. A gateway pays off when several people share provider keys and need per-team limits.
Is it safe to route coding agents through a proxy?
It can be. Check storage, retention, training use and where requests are processed before sending code through it.