LLM Caching: Prompt Cache vs KV Cache (2026)
When an interviewer asks how you’d cut an LLM’s token bill, they are testing whether you understand caching — and which kind. Two different mechanisms hide under the word: prompt (prefix) caching, which lets the provider skip re-processing the unchanged front of your prompt, and the KV cache, the attention key/value memory the model reuses while it generates. Knowing when each applies — and how to lay out a prompt to earn prefix caching — is the LLM-cost interview.
The highest-leverage move is ordering, because prefix caching only reuses tokens up to the first one that changes — one early volatile segment busts the cached prefix for everything after it. Concrete, verifiable example: in the puzzle below a 2,400-token prompt is authored with a volatile segment first, so every call recomputes all 2,400 tokens. Reorder the stable context to the front and the cached prefix grows to 1,400 tokens — you recompute only 1,000, 58% fewer billed tokens, on every single call.
That gap is the interview. Caching is not automatic; you earn it by structuring the prompt so the stable parts — system prompt, tool definitions, few-shot examples, retrieved docs — sit ahead of the volatile parts, like the user’s latest turn. This guide explains prompt cache vs KV cache, the ordering rule, and a free interactive that scores your prompt layout against the exact optimum.
Why this question shows up in AI-engineering interviews
Token cost is the line item every AI team watches, and caching is the biggest lever — so it gets probed constantly, usually disguised:
- “How would you reduce this agent’s token cost?”
- “What’s the difference between prompt caching and the KV cache?”
- “We turned on prompt caching and saw no savings — why?”
- “How do you structure a prompt to maximise cache hits?”
Each tests whether you know caching is structural, not a switch. The savings depend on how the prompt is laid out and on which cache you mean — and a confident, specific answer here signals you’ve actually run an LLM system at cost.
Prompt cache vs KV cache
| Prompt (prefix) cache | KV cache | |
|---|---|---|
| Caches | The provider’s processed prefix of your prompt | The attention key/value tensors for tokens already in context |
| Reused across | Separate requests that share an identical prefix | Decode steps within one generation (and a shared prefix on one server) |
| You control it by | Ordering: stable content first, volatile last | Mostly automatic — you benefit by reusing the same context |
| Saves | Re-processing (prefill) work, and billed prompt tokens | Recomputing attention over prior tokens each step |
| Busted by | The first changed token — everything after re-processes | Eviction, or a new context with no shared prefix |
The short version: the KV cache is mostly the model’s job, and you benefit from it automatically by keeping a conversation’s context stable. The prompt (prefix) cache is your job — you earn it by ordering the prompt so the unchanging parts come first. The puzzle below drills that ordering.
The rule: stable context first
Prefix caching reuses the prompt only up to the first token that differs from last time. So the layout that wins is simple in principle and easy to get wrong in practice:
- Put stable content first — system prompt, tool/function definitions, few-shot exemplars, retrieved documents. Anything identical across calls belongs in the cached prefix.
- Put volatile content last — the user’s latest message, timestamps, per-call IDs, anything that changes. One volatile segment near the front re-processes everything after it.
- Mind the dependencies. Sometimes a stable segment must legitimately follow a volatile one, so you can’t cache every static piece — order for the largest cacheable prefix the constraints allow.
- Don’t confuse the two caches. You don’t hand-tune the KV cache; you do design the prompt layout that the prefix cache rewards.
Put together: float every stable segment to the front, sink every volatile one to the back, and respect the ordering constraints — the largest cacheable prefix is the cheapest prompt. That is exactly what the puzzle scores.
The trap: “caching is automatic”
The common wrong answer is “I’d just turn caching on.” Prefix caching only helps if the prompt is structured for it; with volatile content up front it saves nothing. In the puzzle, the as-authored order recomputes 2,400 tokens every call; the same segments reordered recompute only 1,000 — the caching was always available, the ordering unlocked it. Naming that — and that one early volatile token busts the prefix — is the senior answer.
Practice it: score your own prompt layout against the optimum
Describing the rule is not the same as finding the best order under real constraints — which is what the interview tests. Below is the live Prompt Cache puzzle from The Arena. Reorder the segments to grow the cached prefix; The Arena computes the order with the fewest recomputed tokens by exhaustive search and scores yours as a percentage of it. It starts in the as-authored order — see how much you can save. No login required.
What “good” looks like
When you can find the largest cacheable prefix under ordering constraints, you can answer “how would you cut token cost?” with a concrete mechanism, not a buzzword. Cutting the as-authored 2,400 tokens toward the optimal 1,000 means you’ve internalised that caching is earned by prompt layout — and you can explain why one early volatile segment is so expensive.
Frequently asked
What is the difference between prompt caching and KV caching?
Prompt (prefix) caching is provider-side reuse of the processed front of your prompt across separate requests that share an identical prefix — you control it by ordering stable content first. The KV cache is the attention key/value memory the model reuses across generation steps (and for a shared prefix on one server); it is mostly automatic. One is earned by prompt layout; the other you benefit from by keeping context stable.
How do you reduce LLM token costs with caching?
Lay the prompt out so stable content — system prompt, tool definitions, few-shot examples, retrieved docs — comes first and volatile content — the user’s latest turn, timestamps, IDs — comes last, so the prefix cache reuses the unchanged front. In the worked example, reordering a 2,400-token prompt cuts recomputed tokens from 2,400 to 1,000 — 58% fewer, every call.
Why is my prompt cache not working?
Almost always because something volatile sits near the front of the prompt — a timestamp, a session id, or the user’s message — and prefix caching stops at the first token that changes. Everything after that volatile segment re-processes. Move volatile content to the very end of the prompt.
What is a KV cache in an LLM?
The key and value tensors the attention mechanism computes for each token. During generation the model reuses them across decode steps instead of recomputing attention over the whole sequence, and a server can reuse them for a shared prompt prefix — which is what makes prefix caching fast.
New scored, provably-optimal interview drills, now and then — no spam, unsubscribe anytime.
More interview guides
41 puzzles — RAG tuning, tool routing, agent orchestration, prompt/KV caching, memory compaction — each scored against a provably optimal answer.
Explore the puzzles →