RAG Chunking Strategy: Choosing Chunk Size (2026)

Why this question shows up in AI-engineering interviews

Chunking is the highest-leverage knob in a RAG system, so it shows up constantly — usually disguised:

• ▹“What chunk size would you use, and why?”
• ▹“Your retrieval recall is low — what do you tune first?”
• ▹“How do you trade off chunk size against top-k and reranking?”
• ▹“This RAG pipeline is over its latency SLA — where do you cut?”

Every one is the same tuning problem: recall is a function of chunk size, how many chunks you retrieve (top-k), and how deeply you rerank — but each of those costs latency, and you have a budget. Maxing all three is never the answer because it blows the SLA. Interviewers ask because real pipelines live here, and “I’d use 512-token chunks” with no reasoning is the junior answer.

The mental model that gets you to optimal

Treat recall as a function you maximise under a latency constraint. Three knobs move it, with different shapes — and chunk size is the one with a true interior optimum:

• ▹Chunk size has a sweet spot, not a direction. Too small fragments a coherent passage across chunks; too large packs each chunk with off-topic text that dilutes the embedding and wastes tokens. There is an interior best — here it is 256 tokens, below the 512-token default and well below the 1024 max.
• ▹Top-k has diminishing returns. The first few retrieved chunks carry most of the recall; each extra one adds less and costs latency. Past the knee you are paying for noise.
• ▹Rerank buys recall but costs latency. A shallow rerank often recovers more recall per millisecond than a deeper top-k — but only up to a point, and only if the budget has room.
• ▹The latency budget binds. Because every knob costs latency and the SLA is fixed, you cannot max all three. The skill is spending the budget where it buys the most recall — which is exactly what the puzzle scores.

Put together: find the chunk-size sweet spot first, then spend the remaining latency budget on whichever of top-k or rerank buys more recall per millisecond. That is the configuration the puzzle scores you against.

The trap: “bigger chunks, retrieve more”

The most common wrong answer pairs big chunks with high top-k — “I’ll use 512- or 1024-token chunks and retrieve 12 to be safe.” It feels thorough and it blows the latency budget while diluting every chunk. In the puzzle, that instinct lands the 512-token default at 65.7% recall, while the 256-token sweet spot with a light rerank reaches 84.3% inside the same SLA. Naming the sweet spot — and that it is a tuning problem, not a default — is what separates a senior answer.

Practice it: score your own configuration against the optimum

Describing the tradeoff is not the same as finding the sweet spot under a budget — which is what the interview tests. Below is the live RAG Pipeline puzzle from The Arena. Tune chunk size, top-k, and rerank depth; The Arena computes the best feasible recall by exhaustive search and scores yours as a percentage of it. It starts on the common 512/top-5 default — see how much you can gain. No login required.

What “good” looks like

When you can find the recall sweet spot under a latency budget, you can answer “what chunk size?” with a method instead of a number — and defend it. Beating the 512-token default’s 65.7% means you’ve internalised that chunk size is tuned, not defaulted, and that the budget decides where the extra recall comes from.