Multi agent AI systems are distributed systems in disguise. Agent orchestration patterns, state passing, failure handling, cost, and when one agent wins.
I spent two weeks building an elegant multi-agent system, and it took me two hours to realize I had reinvented a broken distributed system with worse observability and a higher bill. A planner agent, a researcher, a coder, a reviewer, a tester — each with its own prompt, its own tools, its own personality. On the whiteboard it looked like a tidy little engineering org: clean boxes, clean arrows, everyone in their lane. Then I ran it against a real task, and it behaved exactly like a distributed system nobody had bothered to design as one: partial failures, stale state, retries that made things worse, and a bill that quietly tripled.
That was the moment it clicked. I hadn't built an "AI agent swarm." I'd built a small, badly-supervised distributed system where every node is non-deterministic, expensive, and occasionally lies to its neighbors. Everything I already knew from Firebase, from message queues, from years of debugging why one service stalled the whole request — all of it came back. The multi-agent hype skips this part entirely, so this post is the part they skip: how to design LLM agent orchestration as the distributed-systems problem it actually is.
The pitch is irresistible if you've ever managed people. You don't write one giant prompt; you hire a team. A planner breaks the work down. Specialists execute. A reviewer checks the output. It maps perfectly onto how humans ship software, which is exactly why it feels right and exactly why it's a trap.
Human org charts work despite enormous communication overhead because humans share context cheaply. I can lean over to a teammate and say "you know that thing from standup," and three words carry an hour of shared background. Agents have none of that. Every scrap of context between agent A and agent B has to be serialized into text, shoved through a model, and reconstituted on the other side — lossily, every single time. You're not building a team. You're building a network of processes that can only talk by mailing each other essays, where the postal service occasionally rewrites the letter.
Once you frame it that way, the questions stop being "what's a good prompt for the reviewer" and start being the questions I've been answering for a decade:
Those are distributed-systems questions. The word "agent" is a costume. If you have ever debugged a race condition between two microservices, you already have most of the mental model you need for a multi-agent system.
My first working version had three agents for a code-generation feature: a "spec writer," a "code writer," and a "critic." It produced worse output than a single prompt I'd written in an afternoon. Not slightly worse — noticeably, embarrassingly worse.
The failure was pure telephone game. The spec writer produced a reasonable spec. The code writer interpreted 90% of it correctly and quietly dropped an edge case. The critic reviewed the code against its own re-imagining of the spec, not the original, and confidently approved it. Three lossy hops, three chances to drift, and no single place that held the ground truth. Each agent was locally reasonable and the system was globally wrong.
So I collapsed it. One agent, one big prompt, all the context in one place. It got better immediately, because there were zero serialization boundaries to lose information across. That's the first lesson, and it's the unglamorous one: an agent boundary is a network boundary, and every network boundary is a place where information goes to die. Don't add one for aesthetic reasons.
But single-agent has a real ceiling. As the task grew, the one prompt started thrashing — too many responsibilities, context window bloated with irrelevant tool output, the model losing the thread on a long run. Context rot is real: past a certain length, cramming more into one window measurably degrades the model's attention on the parts that matter. So I split it again. This time not by job title, but by two rules I actually trust:
Two agents for reasons. Not five for vibes. Splitting for a concrete, nameable capability versus splitting because an org chart looks impressive is the single most useful filter I apply when I design one of these systems now.
Once you accept you're doing distributed coordination, you can stop inventing and use patterns that are decades old. Three orchestration patterns cover almost everything I've built.
Agents run in a fixed sequence, output of one feeding the next. It's a Unix pipe with a language model in each stage.
research → draft → critique → revise
Dead simple, easy to reason about, easy to log. The weakness is the weakness of any pipe: it's only as strong as its most confused stage, and errors compound downstream. Great when the task genuinely has distinct stages. Terrible when it doesn't and you're just pretending it does. If you can't name what each stage uniquely contributes, you don't have a pipeline — you have a single prompt that you accidentally cut into pieces.
One orchestrator agent holds the goal and delegates to worker agents, deciding dynamically who runs next based on results. This is the "manager" pattern everyone reaches for.
┌─────────────┐ │ supervisor │ └──┬───┬───┬──┘ │ │ │ search code test
It's flexible and it's where the real cost and complexity live, because the supervisor is a control loop making non-deterministic routing decisions. It can loop forever. It can delegate to the wrong worker. It can misread a worker's result and declare victory early. In distributed-systems terms, you've built a scheduler whose logic is a probability distribution. Powerful, but treat it with the suspicion you'd treat any scheduler you can't fully predict — and give it a hard iteration cap so a confused routing decision can't turn into an infinite, billable loop.
All agents read from and write to a shared, structured state — the "blackboard" — instead of messaging each other directly. Nobody talks point-to-point; they all collaborate through one visible source of truth.
I've come to like this most for anything non-trivial, because it fixes the telephone game. There's exactly one place the truth lives. An agent doesn't paraphrase another agent's output; it reads the actual artifact. In practice my blackboard is a plain JSON object (or a Firestore document) with typed fields, and agents get scoped read/write access to parts of it.
{ "goal": "add rate limiting to the upload endpoint", "plan": ["find the endpoint", "add a limiter", "write a test"], "findings": { "endpoint_file": "lib/api/upload.dart", "framework": "shelf" }, "artifacts": { "patch": null, "test": null }, "status": "in_progress"}The moment state is explicit and shared like this, half the coordination problems become visible instead of mysterious. It also gives you something to snapshot and diff, which — as I'll get to — is most of what makes these systems debuggable at all.
These three aren't mutually exclusive. My real systems are usually a supervisor that routes work, with a blackboard as the shared memory the supervisor and workers all read and write, and a short pipeline inside individual steps. Pick the pattern per boundary, not for the whole system.
This is where most agent systems quietly rot. The naive approach is to pass the entire conversation history from one agent to the next. It works in the demo and falls apart in production for the same reasons unbounded state always does: it grows without limit, it carries irrelevant noise, and every token of it costs money on every hop.
What's worked for me is treating inter-agent state exactly like an API contract between microservices.
Here's roughly what a boundary contract looks like in practice — a typed handoff instead of a dumped transcript:
{ "task_id": "gen-4821", "input_ref": "blackboard://findings/endpoint", "requirement": "reject >10 uploads/min per user with a 429", "constraints": ["no new dependencies", "keep existing tests green"], "expected_output": "unified_diff"}I learned the reference-not-blob rule the hard way. On one run, two agents each had their own copy of "the plan" in their context, they drifted apart over several turns, and they spent real tokens arguing past each other because neither knew the other was working from a different version. That's not an AI problem. That's a cache-invalidation problem wearing a hoodie. One authoritative copy on the blackboard, everyone else holds a reference — the same discipline that keeps a distributed cache honest.
Here's the thing the demos never show: agents fail softly. A microservice that dies returns a 500 and you know. An agent that fails returns a confident, well-formatted, completely wrong answer, and the next agent treats it as gospel. It's the difference between a crash and silent data corruption, and silent corruption is the one that ruins your weekend.
A single hallucinated fact early in a pipeline propagates through every downstream stage, getting more elaborate and more confident at each hop. One poisoned node, whole run poisoned. So I handle agent failure the way I handle failure in any distributed system, adapted for the fact that "failure" here often means "plausible nonsense."
None of this is exotic. It's circuit breakers, bounded retries, idempotency keys, and blast-radius control — the exact playbook for any system where a component can fail independently. The only twist is that here the failure mode is eloquent, which means your validators have to be dumb and deterministic on purpose. You cannot ask a language model "is this correct?" and trust the answer to catch the model's own confident mistakes; you need a check that doesn't share the generator's blind spots.
The demo shows a clean run and a slick result. It does not show the token meter. Let me show you the token meter, because this is the part that actually changed how I design these systems.
A single agent answering a question costs roughly one pass over its context. A multi-agent system multiplies that along several axes at once, and they compound:
On a recent internal build, I compared a tuned single agent against my "nice" five-agent version on the same batch of tasks. The multi-agent system used several times the tokens for output that was, at best, marginally better and sometimes worse. A multiple, not a rounding error. That kind of number should be in every agent-swarm demo and it is in exactly none of them.
There are real levers that pull this cost back down when you do need multiple agents — prompt caching to stop re-paying for a static system prompt on every hop, keeping the shared blackboard small and typed so re-reads are cheap, and using a smaller, faster model for mechanical worker steps while reserving the expensive model for the reasoning-heavy supervisor. But none of those change the shape of the problem: coordination is a tax, and you're choosing to pay it.
This maps cleanly onto a distributed-systems truth I already believed: coordination is not free, and it's usually the most expensive thing in the system. In microservices, the network calls and serialization often cost more than the actual work. In multi-agent, the coordination tokens often cost more than the tokens doing the real thinking. If you can't point to a concrete capability the extra agents buy you, you're paying a large tax for an org chart that looks good on a slide.
The rule I now apply: add an agent only when the marginal capability clearly exceeds the marginal coordination cost. Most of the time it doesn't.
The first time a multi-agent run went sideways in a way I couldn't explain, I opened the logs and found a wall of text: five agents having a long, meandering conversation with each other, none of which I wrote, all of it plausible, and somewhere in there was the bug. Debugging that felt like reading a Slack channel from a company I'd just been hired into, where everyone is slightly wrong and very polite about it.
This is genuinely harder than debugging normal distributed systems, and the reasons are the same reasons plus a nasty new one:
What actually made this tractable was, again, boring distributed-systems hygiene — the same observability discipline I'd bring to any production service:
Observability isn't a nice-to-have here. It's the difference between a system you operate and a system that operates you.
After all of this, my honest default is: reach for a single strong agent first, and make it as good as you can before you split anything. Most tasks that look like they need a team need a better prompt, better tools, and a bigger context budget — not an org chart.
Multi-agent earns its keep in a narrow set of cases, and they're worth naming precisely:
Notice what's not on the list: "because a team of specialists sounds more powerful." That instinct comes from human org design, where the coordination cost is hidden by cheap shared context. With agents, the coordination cost is the dominant cost. Every agent you add is a node you have to coordinate, fund, and debug — so the burden of proof is on the split, not on staying simple.
The best multi-agent system I've shipped has two agents and a lot of deterministic code around them. It's boring. It's cheap. It works. That's the whole goal.