Why we didn’t self-host our LLM on RunPod or Vast.ai
Our AI agent was replying in ~18 seconds. The internet’s advice was to self-host the model on rented GPUs. We ran the numbers first — and chose not to. Here’s the latency and cost breakdown for anyone building an interactive agent.
1. The symptom: a ~18-second reply
Our in-app agent, Ask TorqueX, was taking around 18 seconds to answer. The advice we kept hearing was the obvious one: “self-host the model on RunPod or Vast.ai and you’ll cut the latency.” It sounds right — your own GPU, no shared queue, no middleman. So we ran the numbers before renting anything. Self-hosting was the wrong lever, and here is the math.
2. Latency isn’t where the GPU lives
The mental model “slow API → rent a faster GPU” is wrong for an agent. Per-turn latency is two things multiplied together:
- Inference speed — how many tokens per second the model actually generates.
- Agentic round-trips — how many sequential model calls a turn takes. Our agent reads status, sets a goal, previews a plan, then acts: several calls, one after another.
- A slow model × N steps = ~18s. A faster GPU running a model that’s confused about its tools just gets you confused, faster — it does not fix either factor.
3. The comparison
Same job — the Ask TorqueX orchestrator — across hosting options. Latency is the number that matters for a chat product; cost/turn is what matters for the bill.
| Approach | Where | Price in / out ($/1M) | ~Cost / turn | Gen speed | ~Turn latency | Tool-use |
|---|---|---|---|---|---|---|
| Kimi K2 — before | OpenRouter | ~$0.15 / $2.50 | ~$0.004 | ~30–50 tok/s | ~15–20s | Medium |
| Gemini 2.0 Flash | OpenRouter / Google | ~$0.10 / $0.40 | ~$0.0016 | ~150–250 tok/s | ~3–6s | High |
| Claude Haiku | Anthropic | ~$1.00 / $5.00 | ~$0.017 | ~80–130 tok/s | ~4–8s | Highest |
| Llama-3.3-70B | Groq | ~$0.59 / $0.79 | ~$0.008 | ~275 tok/s | ~2–5s | Medium |
| gpt-oss-120b — now | Cerebras | ~$0.35 / $0.75 | ~$0.006 | sub-1s TTFT | ~4s ✓ | High |
| zai-glm-4.7 | Cerebras | ~$2.25 / $2.75 | ~$0.03 | sub-1s TTFT | ~4.2s ✓ | High |
| gemma-4-31b | Cerebras | not listed | — | — | ~7s ✓ | Medium |
| Self-host 70B | 1× H100 (RunPod / Vast) | GPU-time | see below | ~40–80 tok/s | ~8–15s | Medium |
| Self-host 1T MoE | 8× H100 (RunPod / Vast) | GPU-time | see below | ~30–60 tok/s | ~15–25s | Medium |
Approximate early-2026 ballparks — verify current provider pricing. Turn ≈ ~12K in + ~1K out tokens over ~3 tool steps. gemma-4-31b has no public per-token price; Qwen (Qwen3-32B / 235B) is Cerebras dedicated-only (reserved-capacity, custom pricing), not self-serve.
4. Why self-hosting loses at our scale
GPU rental is billed by time and runs always-on — not per token. Two things break it for a bursty, single-user product: there’s no request batching (so tokens/sec is often worse than an optimised hosted API), and a frontier model needs a small cluster.
| Rig | Rate | If always-on | Verdict |
|---|---|---|---|
| 1× H100 — fits a quantized ~70B | ~$2.5–4 / hr | ~$1,800–2,900 / mo | Break-even vs Gemini Flash needs hundreds of thousands of turns/mo |
| 8× H100 — a Kimi-class 1T MoE | ~$20–32 / hr | ~$14k–23k / mo | Never, at our volume — and single-stream latency is no better |
5. What we’re doing instead
The fix wasn’t the metal — it was the model. Switching the orchestrator from Kimi K2 to a fast, reliable hosted model collapses the wall-clock and removes the tool-confusion in one move, with nothing to operate:
- Fastest + cheapest, no new key — a fast general model like Gemini 2.0 Flash: ~4× faster than Kimi and cheaper per turn.
- Most reliable tool-use — Claude Haiku: fast and excellent at calling the right tool, for a few cents a turn.
- Open weights but blazing — Cerebras. In our own test the same “send cold outreach mail” turn ran in ~4 seconds on Cerebras (gpt-oss-120b) — sub-second per step, with a few sequential tool calls — and it used the tools correctly, versus ~18s on Kimi K2. That’s the real “fast open model” answer, and it fixed the tool-confusion too — not renting GPUs. (We A/B’d zai-glm-4.7 on Cerebras as well — ~4.2s and equally correct; we shipped gpt-oss-120b for the small speed edge.)
6. The takeaway
Before you rent GPUs to fix agent latency, check whether a faster hosted model solves it — it almost always does, and for a fraction of the cost. Self-hosting only pays off at high, steady throughput where batching amortises a full-time GPU. For an interactive product with bursty, single-user traffic, a hosted endpoint is faster and orders of magnitude cheaper. Optimise the model and the number of steps first; reach for your own silicon last.
Agents that run the work — not just chat about it
torqueX is an Agent Orchestrator: set the goal, it runs a team of agents end-to-end. The Outreach Agent is live today — start there and grow into the full team as more agents ship.