LLM-as-a-Judge: Building an Eval Set (2026)
When an interviewer asks how you’d measure whether your LLM feature actually works, they are testing evals — increasingly the AI flavour of the system-design round, and the home of LLM-as-a-judge. A big part of the answer is choosing a representative eval set: a small set of test cases that covers your failure modes. The instinct to pick the ‘biggest’ tests is a trap.
Concrete, verifiable example. In the puzzle below there are 8 bug-classes and 8 candidate tests, and you may keep 3. Choosing for marginal coverage covers 7 of 8 classes; picking the 3 tests with the largest individual coverage — the obvious move — covers only 6, because they overlap. A second test that hits the same bug-classes as the first adds nothing.
That gap is the interview. Choosing an eval set is maximum-coverage: maximise the union of failure modes your cases exercise, under a budget on how many you can run or label. A senior answer picks for what each new test adds, not for its raw size. This guide gives you the model, and a free interactive scored against the exact optimum.
Why this question shows up in AI-engineering interviews
You can’t ship an LLM feature you can’t measure, so eval design is rising fast — disguised as:
- “How would you evaluate this LLM feature?”
- “Design an LLM-as-a-judge to score these outputs.”
- “You can only run 20 test cases — which ones?”
- “How do you know your eval set is representative?”
Each rewards thinking about coverage under a budget. Evals (and LLM-as-a-judge) cost tokens and labelling time, so you can’t run everything — you choose a small set that exercises as many distinct failure modes as possible. Interviewers ask because a non-representative eval set gives false confidence: green dashboards over untested behaviour.
The mental model that gets you to optimal
Picking the eval set is the maximum-coverage problem. Three ideas get you to the widest coverage:
- Optimise for marginal coverage. The value of a test is the new failure-modes it adds over the ones you’ve already covered — not its size in isolation. A huge test that overlaps an existing one adds little.
- Beware overlapping ‘big’ tests. The two tests with the largest individual coverage often hit the same classes; two smaller, disjoint tests can cover more of the space together.
- Cover failure modes, not volume. Ten variations of the same easy case measure one thing; one case per distinct failure mode measures many. Breadth of coverage beats count of cases.
Put together: greedily add the test that covers the most still-uncovered failure modes, and stop at your budget — and check that no two of your picks are redundant. That is the eval set the puzzle scores.
The trap: “pick the biggest tests”
The default wrong answer is to rank tests by how much each covers and take the top few. They overlap. In the puzzle, that greedy-by-size pick covers 6 of 8 bug-classes, while choosing for marginal coverage reaches 7 of 8 with the same budget — the extra class is one the ‘big’ tests all happened to skip. Naming marginal coverage, and that overlap is wasted budget, is the senior answer.
Practice it: choose the eval set with the widest coverage
Describing coverage is not the same as choosing a set under a budget. Below is the live Eval Set puzzle from The Arena. Pick your tests; The Arena computes the set that covers the most bug-classes and scores yours as a percentage of it. No login required.
What “good” looks like
When you choose evals for marginal coverage, you can defend an eval set as representative instead of hoping it is — and your LLM-as-a-judge measures real failure modes, not duplicates. Beating the greedy-by-size 6/8 toward the optimal 7/8 means you’ve internalised that overlap is wasted budget.
Frequently asked
How do you build a good eval set for an LLM?
Choose a small set of cases that together cover as many distinct failure modes as possible — maximum coverage under a budget on how many you can run or label. Pick each test for what it adds over the ones you already have, not for its raw size. In the worked example the best 3 cover 7 of 8 failure-classes versus 6 for greedy-by-size.
Is LLM-as-a-judge a system design interview question?
Increasingly, yes — evaluating LLM outputs is the AI flavour of system design. You design the judge (rubric, calibration, bias controls) and, crucially, the eval set it runs on: a representative set of cases that covers your failure modes under a budget.
How many test cases do you need to evaluate an LLM?
Fewer than you’d think, if they’re chosen well. What matters is coverage of distinct failure modes, not volume — one case per failure mode beats ten variations of the same easy case. Pick for marginal coverage until added cases stop covering anything new.
Why is my eval set not catching regressions?
Usually because it’s redundant — many cases exercise the same few behaviours while whole failure modes go untested. Re-pick for marginal coverage so each case adds a distinct failure mode, and include any real failure you’ve seen in production.
New scored, provably-optimal interview drills, now and then — no spam, unsubscribe anytime.
More interview guides
41 puzzles — RAG tuning, tool routing, agent orchestration, prompt/KV caching, memory compaction — each scored against a provably optimal answer.
Explore the puzzles →