Agent Orchestration: 2026 System Design Interview
When an interviewer asks you to “design a production agent” or “orchestrate these tools,” they are testing whether you can reach a goal for the least total cost — not the fewest tool-calls. It is min-cost planning over a graph of tool dependencies, and the instinct to make one big do-it-all call is usually the expensive answer.
Here is a concrete, verifiable example. In the puzzle below you must reach safety_checked starting from user_query, choosing among 10 tools that each consume some facts and produce one at a cost. The single do_it_all_llm call reaches the goal in one hop — and costs 31. The cheapest plan instead chains 5 focused tools for a total of 18 — 42% less. Fewest calls is not cheapest; the one-shot costs 72% more.
That gap is the whole question. Orchestration is shortest-path planning: every tool is an edge with a cost, and you want the cheapest route to the goal — which a chain of cheap, specialized tools usually wins, even though it is more calls. This guide gives you the model, and a free interactive that scores your plan against the exact optimum (computed by Dijkstra over the fact-states).
Why this question shows up in AI-engineering interviews
Agent orchestration is fast becoming the AI-flavored system-design round. It wears several disguises:
- “Design a production agent that answers a user question end-to-end.”
- “Here are your tools — how do you orchestrate them?”
- “When would you use one big model call versus a chain of smaller ones?”
- “This agent is too slow and too expensive — how do you bring its cost down?”
Every one of these is the same underlying problem: each tool consumes some facts and produces another at a cost, and you want to reach the goal fact as cheaply as possible. Tools are edges, facts are nodes, and “orchestrate the agent” means find the cheapest path. Interviewers ask it because production agents live or die on cost and latency — and the lazy one-shot call is exactly the answer that blows both.
The mental model that gets you to optimal
Model it as a graph. Each fact you can hold is a node; each tool is a directed edge that turns the facts it requires into the fact it produces, at a cost. “Reach the goal” means find the cheapest path from what you start with to the goal fact — that is Dijkstra, not “call the smartest model once.” Four ideas get you there:
- Count cost, not calls. The metric is total cost — tokens, latency, dollars — not how many tools you invoke. A 5-call plan at cost 18 beats a 1-call plan at cost 31 every time.
- Decompose into cheap, specialized steps. A focused tool that does one thing is usually far cheaper per result than a do-everything call. Chaining them is the win, not the overhead.
- Use a shortcut only when it is genuinely cheaper. Some skip-ahead tools are worth it — in the puzzle, a cheap alternate tool jumps straight to the finished answer. The skill is comparing a shortcut’s cost to the chain it replaces, not assuming fewer hops means cheaper.
- Prune dead branches. A tool that produces a fact the goal doesn’t need is pure cost. Only fire what sits on a cheapest path to the goal.
Put together: lay out the tool graph, then find the minimum-cost path to the goal — exactly what Dijkstra does, and exactly what the puzzle scores you against.
The trap: “just call the big model”
The single most common wrong answer is the one-shot do-it-all call — “I’d hand the model the query and the tools and let it figure it out.” It is one hop, so it feels efficient, but a call priced to produce the final answer in one shot is priced for that generality. In the puzzle, do_it_all_llm costs 31 against the chain’s 18 — you pay 72% more for the convenience. Interviewers probe exactly this: can you justify decomposition on cost, or do you reach for the biggest hammer?
Practice it: score your own plan against the optimum
Describing the approach is not the same as building the plan under a cost budget — which is what the interview tests. Below is the live Agent Orchestration puzzle from The Arena. You fire tools to turn the facts you hold into the goal fact; The Arena computes the provably cheapest plan by shortest-path search and scores yours as a percentage of it. No login required.
What “good” looks like
When you can consistently find the cheapest plan, you can defend an agent design on cost in the interview — not just draw boxes. Matching the optimum means you’ve out-reasoned the “just call the big model” instinct that costs 72% more here, and you can explain why a chain of cheap, specialized calls is the senior answer.
Frequently asked
Is agent orchestration a system design interview question?
Yes — increasingly the AI-flavored system-design round. Instead of designing a URL shortener, you design how an agent reaches a goal using tools, judged on cost, latency, and reliability. At its core it is minimum-cost path planning over a tool-dependency graph.
Should an AI agent use one big tool call or many small ones?
Whichever is cheaper in total, not whichever uses fewer calls. A do-everything call is priced for its generality; a chain of cheap, specialized tools often reaches the same goal for far less. In the worked example the one-shot call costs 31 versus the optimal chain’s 18 — 72% more.
How do you minimize an AI agent's cost?
Model tools as edges in a graph with costs, then take the cheapest path to the goal: decompose into specialized steps, use a shortcut only when it genuinely undercuts the chain it replaces, and never fire a tool that produces a fact the goal doesn’t need.
What algorithm solves agent orchestration optimally?
Treat the set of facts you hold as a state and each tool as a transition with a cost; the cheapest plan is the shortest path through that state graph, found with Dijkstra (uniform-cost search). The puzzle here computes exactly this to grade your plan.
New scored, provably-optimal interview drills, now and then — no spam, unsubscribe anytime.
More interview guides
41 puzzles — RAG tuning, tool routing, agent orchestration, prompt/KV caching, memory compaction — each scored against a provably optimal answer.
Explore the puzzles →