Cutting AI-agent token costs — without a proxy you don’t need yet
There’s a wave of tools that promise to slash your LLM bill with compression and caching — caveman.so is a good example. They can genuinely help. But for a tool-calling agent, most of the savings come from doing four boring things in the right order — and reaching for the proxy last. Here’s how we think about it.
1. Token cost isn’t the model — it’s what you resend
Every agent turn re-sends the same three things: the system prompt, the tool schemas, and the growing message history. For a tool-calling agent those input tokens dwarf the actual answer. So “make it cheaper” is rarely “pick a cheaper model” — it’s “send fewer tokens, at the right price.” There are four levers, and only some are worth pulling.
2. The four levers
In rough order of bang-for-effort:
- Trim what you resend — the cheapest win: a leaner system prompt, tighter tool descriptions, and a capped message history. No dependency, no risk.
- Context compression — byte-safe compression of large blocks before the call (tools like caveman.so report ~65% fewer output tokens). Powerful on long contexts, but be careful compressing anything the model must parse exactly.
- Prompt caching — provider-native caching of the stable prefix (system + tools). Big win where the provider supports it (Anthropic, OpenAI). Not universal.
- Model routing — send each task to the cheapest model that still passes your evals. This is the lever we leaned on hardest.
3. Where compression (and caveman) fits
caveman.so packages the compression idea well — a compression engine that runs as an open-source skill, a local proxy, or a managed gateway, plus caching hints and model routing. It shines when you’re pushing long contexts through an expensive model at volume. Two cautions for agent builders: provider-native caching only helps if your provider offers it, and compressing a tool-calling turn can corrupt the exact JSON the model needs to emit — so compress read-only context, never tool arguments. For us it’s deferred, not dismissed: at our current model+volume it isn’t worth the failure surface yet.
4. The levers, side by side
Same goal — fewer tokens at the right price. What each lever costs you to adopt, when it pays off, and where we landed.
| Lever | How | When it pays off | Our call |
|---|---|---|---|
| Trim what you resend | Shorter system prompt + tool schemas; cap message history | Always — input tokens dominate every turn | Done: history capped, lean tool defs |
| Context compression | Byte-safe compression of large blocks (logs, JSON, tables) — e.g. caveman.so | Long contexts on expensive models | Deferred — risky for tool-call JSON |
| Prompt caching | Provider-native cache of the stable prefix | Provider supports it (Anthropic, OpenAI) | N/A — Cerebras has no prompt cache |
| Model routing | Send each task to the cheapest model that passes your evals | Wide task mix, quality gates in place | Done: fast open model on Cerebras |
“Trim” and “route” are cheap and safe; “compress” and “cache” are powerful but conditional. Pull them in that order.
Per-turn economics
Why the proxy can wait: on a fast, cheap open model the token cost is already tiny (turn ≈ ~12K in + ~1K out over a few tool steps).
| Model | Price in / out ($/1M) | ~Cost / turn | Note |
|---|---|---|---|
| gpt-oss-120b (Cerebras) — ours | ~$0.35 / $0.75 | ~$0.006 | fast + cheap; token cost is a rounding error |
| Claude Haiku | ~$1.00 / $5.00 | ~$0.017 | excellent tool-use; caching helps a lot |
| Kimi K2 (before) | ~$0.15 / $2.50 | ~$0.004 | cheap tokens, but slow → we moved off it |
| Frontier 1T self-host | GPU-time | see other post | never breaks even at our volume |
Approximate early-2026 ballparks — verify current provider pricing.
5. What we actually did — and how we price it
We pulled the two cheapest levers first: trim the prompt/tools and cap history, then route to a fast, cheap open model (gpt-oss-120b on Cerebras — see our self-hosting post for why). At ~$0.006 a turn, token cost became a rounding error, so heavier machinery like compression or a caching proxy would cost more (latency, dependency, risk) than it saves. We meter it honestly: every Ask TorqueX turn deducts credits equal to exactly what the model cost us — pass-through token pricing, no markup ($1 = 1 credit). You pay for the tokens you use, at cost.
6. The takeaway
Before you bolt a compression/caching proxy onto your agent, do the boring things first: shorten the prompt, trim the tools, cap the history, and route to a fast cheap model. Measure the real per-turn cost. Reach for compression and caching when your context is genuinely large and your model is genuinely expensive — not by default. Optimise the tokens and the price; add infrastructure last.
Agents that are fast, cheap, and honest about it
torqueX runs a team of AI agents on a fast open model and bills you at pass-through token cost — $1 = 1 credit, no markup. Start with the Outreach Agent and grow into the full team.