Retry vs Fallback: The Expected-Cost Math of Flaky LLM Calls
When an interviewer asks how you’d handle flaky LLM providers, they are testing whether you think in expected cost. A retry chain is tried in order and stops at the first success — so you pay a fallback’s price only on the calls that reach it. That one fact breaks the intuitive answer (“use the reliable premium model”) more often than candidates expect.
Concrete, verifiable example. In the puzzle below, 6 providers each have a success rate and a cost, and a failed run — nothing succeeds — costs a 74 penalty. The single most reliable call, provider-0 at 84% success for 21, is the best one-call policy: 32.8 expected. The optimal chain instead leads with provider-5 — 74% success for 16 — and keeps provider-0 as the first fallback, for an expected cost of 22.7: 69% of the premium-only bill. The premium call doesn’t disappear; it becomes the backup you pay for only 26% of the time.
That gap is the interview. Retry-and-fallback design is ordering arithmetic: expected cost is the sum over attempts of (probability you reach the attempt) × (its cost), plus the penalty weighted by the chance everything fails. This guide gives you the math, the ordering rule, and a free interactive that scores your chain against the exact optimum (exhaustive search over ordered subsets).
Why this question shows up in AI-engineering interviews
Provider flakiness is a fact of production LLM systems, so reliability design is probed constantly — disguised as:
- “Your provider has a 5% error rate — what does your client do?”
- “When do you fall back to a different model, and in what order?”
- “Cheap unreliable model or expensive reliable one?”
- “What does a failed request actually cost you?”
Each is the same calculation: attempts have costs and success probabilities, failure has a price, and the policy that wins is the one with the lowest expected cost — not the one that feels safest. Interviewers ask because the instinct to lead with the most reliable call quietly pays premium prices on requests a cheap model would have cleared.
The mental model that gets you to optimal
Write the expected cost down and the strategy falls out. Four ideas get you to the best chain:
- You pay an attempt only if you reach it. Expected cost = Σ P(reach attempt i) × costᵢ + P(all fail) × penalty. The chance of reaching attempt i is the product of every earlier attempt’s failure rate — fallbacks deep in the chain are nearly free.
- Cheap and likely-to-work goes first. The first attempt is paid on 100% of calls, so its cost dominates. A 16-cost call at 74% success in front means the 21-cost premium is only paid on the 26% of calls that need it.
- Order by cost ÷ success. For a fixed set of attempts, the expected cost is minimised by trying them in ascending cost-per-success-probability — the adjacent-swap exchange argument, and exactly the order the optimal chain below follows.
- The penalty decides the depth. Deep fallbacks are cheap insurance against the all-fail penalty: whether a sixth attempt is worth adding depends on the penalty it hedges, not on how often it runs.
Put together: include the attempts whose insurance value beats their cost, and order them by cost ÷ success. That policy — not the premium one-shot — is what the puzzle scores you against.
The expected-cost math, line by line
| Attempt | Chance you reach it | Cost if reached | Adds to expected |
|---|---|---|---|
| 1. provider-5 | 100.0% | 16 | 16.00 |
| 2. provider-0 | 26.0% | 21 | 5.46 |
| 3. provider-2 | 4.2% | 13 | 0.54 |
| 4. provider-4 | 2.2% | 19 | 0.42 |
| 5. provider-1 | 0.7% | 17 | 0.12 |
| 6. provider-3 | 0.3% | 12 | 0.03 |
| (every attempt fails) | 0.2% | 74 penalty | 0.12 |
Those lines sum to 22.69 — the optimal chain’s expected cost, and the number the puzzle grades against. Read the shape: the first attempt contributes almost everything, the premium fallback adds its cost × 26%, and everything deeper is pennies. Compare the premium-only policy: 21 on every call plus the penalty on the 16% it fails — 32.8 expected, 45% more than the chain.
The puzzle simplifies on purpose: attempts are independent (real providers fail together — a hard prompt or an upstream outage defeats the whole chain at once), each provider is tried once (real policies retry the same endpoint with backoff), the penalty is a flat number, and latency isn’t modelled (a deep chain costs wall-clock time even when it saves money). The expected-cost skeleton survives every one of those refinements — you just re-estimate the probabilities.
Practice it: build the chain against the optimum
Describing expected value is not the same as ordering real attempts under a penalty. Below is the live Retry & Fallback puzzle from The Arena. Build your chain; The Arena computes the minimum-expected-cost policy by exhaustive search over ordered subsets and scores yours as a percentage of it. No login required.
What “good” looks like
When you can build the 22.7-expected chain instead of defaulting to the 32.8 premium call, you can answer the reliability question with arithmetic: what each attempt costs, who reaches it, and what failure is worth. Explaining why the premium model belongs second — paid only when the cheap try fails — is the senior answer.
Frequently asked
What is a retry-fallback strategy for LLM calls?
An ordered chain of providers tried until one succeeds, designed to minimise expected cost: cheap likely-to-work attempts first, reliable expensive ones as fallbacks, and a penalty term for total failure. In the worked example the best chain expects 22.7 per call versus 32.8 for calling only the most reliable provider.
Should you call the most reliable LLM provider first?
Usually not. The first attempt is paid on every call, so leading with the premium provider pays its price even on requests a cheap model would clear. Lead with the cheapest attempt whose success rate is decent, and let the reliable provider be the fallback — you pay it only when reached.
How do you order a retry chain optimally?
For a fixed set of attempts, sort ascending by cost ÷ success probability. Swapping two adjacent attempts changes expected cost by exactly the difference this rule captures, so the sorted order is provably minimal. Which attempts to include at all depends on the all-fail penalty they insure against.
Is a cheap flaky model worth trying before an expensive reliable one?
Often, yes — that is the whole gap in the worked example. A 74%-success attempt at cost 16 resolves most calls cheaply, and the 21-cost reliable model backs it up on the 26% that remain. The chain expects 22.7; the reliable-only policy expects 32.8.
New scored, provably-optimal interview drills, now and then — no spam, unsubscribe anytime.
More interview guides
41 puzzles — RAG tuning, tool routing, agent orchestration, prompt/KV caching, memory compaction — each scored against a provably optimal answer.
Explore the puzzles →