Guardrail Latency Budgets: Which Checks Are Worth Running?

Why this question shows up in AI-engineering interviews

Every production LLM app now runs safety checks in the request path, so the latency question is a staple — disguised as:

• ▹“Your guardrails add 300ms — which do you cut?”
• ▹“How many safety checks are enough?”
• ▹“Design the moderation pipeline for this feature.”
• ▹“Would you add another jailbreak classifier here?”

Each is the same question: which checks earn their milliseconds? Interviewers ask because both failure modes are expensive in production — an under-guarded app ships harm, and an over-guarded one adds serial latency to every single request for coverage the target never asked for. The senior signal is treating the stack as a budget, not a virtue.

The mental model that gets you to optimal

Model each check as (catch rate, latency cost) and assume independent failures. Four ideas get you to the cheapest clearing stack:

• ▹Compound the misses, not the catches. A stack’s combined catch is 1 − the product of the misses. Three checks at 72%, 67%, and 65% miss 0.28 × 0.33 × 0.35 of unsafe outputs — a 96.8% catch from three moderate checks.
• ▹Each added check works on leftovers. The second check only sees what the first missed, so its marginal contribution shrinks: a 65% check added to a 90% stack adds ~6.5 points, not 65. Raw catch rate overstates every check after the first.
• ▹Stop at the bar. The target is a constraint, not a score to maximise — combined catch above it buys nothing. The question for every check is whether the stack still clears the target without it; if it does, the check is pure latency.
• ▹Buy marginal catch per millisecond. When you must add a check, add the one with the best marginal catch (on what’s still missed) per unit of latency — which is not always the strongest one.

Put together: find the cheapest subset whose misses multiply below 1 − target, and run nothing else. That subset — not the full stack — is what the puzzle scores you against.

The trap: “run everything, to be safe”

Running every check feels responsible and costs 80% more latency than the target requires — on every request, forever. The compounding walk above is the argument: 3 checks already clear 96.3%, so the remaining 4 checks spend 43ms buying catch the target never asked for. Naming that — coverage past the bar is pure latency — is the senior answer.

The puzzle simplifies on purpose: it treats checks as independent (real failure modes correlate — two jailbreak classifiers overlap far more than their rates multiply), runs them serially (production stacks parallelise some checks, which moves cost from sum toward max), and ignores false positives (a check that blocks good traffic has a cost beyond milliseconds). The selection logic survives all three: estimate marginal catch, price each check, stop at the bar.

Practice it: build the cheapest clearing stack

Describing compounding is not the same as choosing under a real target. Below is the live Guardrail Gate puzzle from The Arena. Pick checks until the combined catch clears the 96.3% bar; The Arena computes the cheapest clearing subset by exhaustive search and scores your latency as a percentage of it. No login required.

What “good” looks like

When you can clear the bar at 54ms instead of 97ms, you can answer “which checks do you cut?” with a method: compound the misses, compute each check’s marginal catch per millisecond, and drop everything the target doesn’t need. That is a latency budget argument — the kind that survives follow-ups.