Production LLM agents fail silently and confidently. Hard lessons on tool design, budgets, external state, observability, and cost from shipping a real agent.
The demo was flawless, and that should have scared me more than it did. I asked our support agent to look up a customer's last order, check the refund policy, and draft a reply. It planned, called three tools, and wrote a genuinely good answer in about eight seconds. My co-founder clapped. We shipped it to a small cohort that Thursday from our office in Dubai.
By Saturday it had told a customer their order was "on the way" when the order didn't exist, burned about $30 in tokens on a single conversation by looping on a tool that kept returning the same error, and quietly dropped half of a multi-step task because it "remembered" a value that was never there. None of these showed up in the demo. All of them were obvious in hindsight. That gap — between the demo that works and the deploy that survives contact with real users — is the whole job. A production LLM agent is maybe 20% the happy-path loop and 80% engineering around the ways it silently goes wrong. This is the anatomy of that 80%, written down while the bruises are still fresh.
If you're building agentic AI of any kind — a support bot, an internal ops assistant, a coding agent, a RAG-backed research tool, anything that plans and calls tools in a loop — the failure modes below are the ones that will actually page you at 2am. Not the ones in the tutorials.
Here is the uncomfortable thing about LLM agents: the demo is genuinely easy and the product is genuinely hard, and they look nearly identical for the first five minutes.
The demo works because you, the author, are an invisible part of the loop. You picked a clean question. You knew which tools existed. You'd have caught a wrong answer instantly because you already knew the right one. In production none of that is true. The user asks something malformed. The database is mid-migration and returns a stale row. A tool times out. And the model — which has no concept of "I am unsure, I should stop" unless you build one — keeps going, because the one thing these models are exceptional at is producing plausible next tokens no matter how bad the situation is.
That's the core failure mode that everything else in this post descends from: an agent fails silently and confidently. A crashed service throws an exception. A null pointer throws. An agent hands you a beautifully formatted, completely wrong answer and moves on. Your entire job as an agent engineer is to convert silent, confident failure into loud, cheap, recoverable failure. Every technique below is a variation on that one theme.
Strip away the frameworks — LangChain, LlamaIndex, whatever's trending — and an LLM agent is a while-loop around a model:
while (!done && steps < MAX_STEPS) { const decision = await model.decide(context); // plan if (decision.type === "final") return decision.answer; const result = await runTool(decision.tool, decision.args); // act context = append(context, decision, result); // observe steps++;}Clean on a slide. This plan-act-observe loop (the same shape as ReAct-style agents) has a specific way each of those three lines betrays you in production.
Plan betrays you by hallucinating capability. The model will happily "call" a tool that doesn't exist, or pass arguments in a shape you never defined, because from its point of view emitting getOrderById is the same kind of act as emitting the word "the". If your runtime blindly trusts the tool name, you crash. So the first rule: the model's output is untrusted input. Validate every tool call against a schema before you execute anything. I treat a malformed tool call exactly like a malformed HTTP request — reject it, return a structured error, and let the model try again on the next turn.
Act betrays you because tools fail, and the model can't see the failure the way you can. A 500 from an upstream API, a timeout, an empty result set — these are all normal in a real system. What matters is not that they happen but what the model observes about them, which brings us to the step that surprised me most.
Observe betrays you because the model believes whatever you put in the context. If a tool returns { "orders": [] } and your observation text says "Here are the orders:", the model will invent orders. The observation you feed back is ground truth as far as the model is concerned — there is no second source it can check against. Garbage or ambiguity in that channel becomes hallucination out. I now spend more time designing what goes back into the context than I do on the prompt itself, and it's not close.
This was my biggest mindset shift, and it's the one I'd hand to anyone starting an agent today. I'd been thinking of tools as functions I expose to the model. They're not. They're an API whose only consumer is a slightly unreliable junior engineer who never reads the docs — only the function signature and the error messages. Design for that consumer, not for yourself.
Name tools like you're writing for autocomplete. search_orders_by_email beats orders beats handleOrderQuery. The model picks tools largely off the name and the one-line description. Ambiguous names cause the model to reach for the wrong tool, and you won't find out until you read a trace three days later. Verb-plus-noun names that describe the effect are worth the extra characters.
Make arguments hard to get wrong. Fewer required args, strong enums instead of free-text wherever possible, and validation that returns a useful message. Compare these two failures:
{ "error": "Invalid input" }{ "error": "invalid_argument", "field": "status", "message": "status must be one of: pending, shipped, delivered, cancelled", "received": "in_transit"}The first sends the model into a guessing loop — it has no idea which of five arguments was wrong or why. The second lets it self-correct on the very next turn, because now the error message is the documentation. On a recent build, rewriting our tool errors into this shape cut retry loops on argument mistakes to nearly nothing. The model was never dumb; it was under-informed. This is the single highest-return change most agent codebases can make.
Return structured, minimal observations. Don't dump a 4,000-token JSON blob back into context and hope the model finds the field. Project it down to what actually matters for the decision at hand. Every token you return is a token the model has to reason over and a token you pay for twice — once now, and again on every subsequent turn, because it stays in the context window for the rest of the run.
Distinguish "no result" from "error" explicitly. An empty list is a valid, correct answer to "find refunds over $500." A timeout is not. If both come back as an empty-ish blob, the model can't tell "there are none" from "I don't know," and it will pick the more confident interpretation every single time. Make the difference impossible to miss:
{ "status": "ok", "count": 0, "results": [] }versus
{ "status": "error", "reason": "upstream_timeout", "retryable": true }That retryable flag matters too: it tells the model (and your loop guard) whether trying again could plausibly help, or whether it should stop and report. Encoding retryability in the observation is cheaper than any retry-policy config you'll bolt on later.
The $30 conversation taught me this one the expensive way. A tool kept returning a retryable-looking error, the model kept retrying with tiny variations, and nothing stopped it because I had assumed the model would eventually "give up." Models don't give up. They have no internal budget and no fatigue. You have to impose limits from the outside, and you need several kinds at once:
MAX_STEPS). When you hit it, you stop and return a graceful "I couldn't finish this" — never an unbounded loop.const seen = new Map<string, number>();function guard(tool: string, args: object) { const key = `${tool}:${stableStringify(args)}`; const n = (seen.get(key) ?? 0) + 1; seen.set(key, n); if (n >= 2) { throw new AgentLoopError( `Repeated call to ${tool} with identical args. ` + `The previous result did not change; stop and report what you have.` ); }}Note that the error message is written for the model — it explains why it was stopped and what to do instead, in the model's own idiom. It's caught by the loop, converted into an observation, and often the model recovers on its own with a sensible partial answer instead of crashing the run. Every agent needs a kill switch, and the kill switch should degrade gracefully into a helpful partial result, not just throw a 500 at the user.
Early on I let the conversation history be the state. The order ID from step two lives in the context, so surely the model will use it in step five. Sometimes it did. Sometimes it grabbed a different number from earlier in the conversation, or hallucinated a plausible-looking one, or silently dropped a step in a multi-part task because the context had gotten long and the relevant fact scrolled out of the model's effective attention.
The fix was boring and it worked: stop treating the context window as durable memory. The context is a scratchpad, not a database. Once I internalized that, a whole category of "why did it forget" bugs disappeared.
order_a" and let your code resolve order_a to the real ID it captured earlier. The model can't fat-finger a value it never has to type.const session = { verifiedOrderId: null as string | null, refundApproved: false,};// After a tool verifies the order, YOUR code writes it:session.verifiedOrderId = result.orderId;// The refund tool reads from session, not from the model's argument:function refund(_args, session) { if (!session.verifiedOrderId) throw new Error("no_verified_order"); return processRefund(session.verifiedOrderId);}This also closes a nasty security gap. If the model can't supply the ID to refund, then a confused — or deliberately prompt-injected — model can't refund an arbitrary order. State ownership and least-privilege turned out to be the same lever pulled from two angles. The model proposes an action against a key it can see; your code authorizes and executes against the real value it holds. That separation is the difference between an agent you can put in front of customers and one you can only demo.
When my logs said "agent responded" and nothing else, debugging a bad run meant guessing. You cannot debug what you can't see, and an agent run has a lot of hidden middle between the user's question and the final answer. The single highest-leverage thing I built was a proper trace of every run.
For each run I record a linear timeline of events:
{ "run_id": "r_9f2", "step": 3, "type": "tool_call", "tool": "search_orders_by_email", "args": { "email": "a@b.com" }, "result": { "status": "ok", "count": 0 }, "latency_ms": 412, "tokens": { "in": 1840, "out": 62 }, "cost_usd": 0.0071}With this, a "the bot lied about the order" report goes from an afternoon of guessing to two minutes of reading. I can see the empty result, see the observation I fed back, and see exactly where a vague message let the model invent. Almost every bug I fixed traced back to a bad observation or a poorly worded tool error — things that are invisible without the trace and obvious with it.
One concrete example. A support ticket came in claiming the agent had promised a refund it never processed. The old me would have re-run the conversation, failed to reproduce it, and closed the ticket as flaky. With the trace I read it in ninety seconds: step 4 called the refund tool, the upstream returned a 202 Accepted with no body, and my observation text said "refund processed" when it should have said "refund queued, not yet confirmed." The model didn't lie. I lied to it, and it dutifully passed my lie to the customer. That is the shape of nearly every agent bug I have shipped — a human mistake in the observation channel, laundered through a confident model into something that looks like the model's fault.
Two habits made traces pay off. Version your prompts and tool schemas so you can correlate a regression with a specific change instead of a vibe — a content hash on the system prompt is enough to answer "did this break when we edited the prompt?" And sample full traces in production, not just failures, so you learn what "normal" looks like before you're reading logs at 2am on a Friday. Cheap object storage is enough for this; I don't need a fancy observability vendor, and on our budget I'm not paying for one. If you later want LLM-specific tracing tooling, the trace schema above ports cleanly into most of them.
With a traditional API, cost and latency are ops concerns you tune after launch. With an agent they're product features, because both are variable and unbounded by default. Two users asking "where's my refund" can differ 10x in cost depending on how many loops the model takes. You can't price a plan or promise a UX around that unless you control it.
What actually moved the numbers, in rough order of impact:
I now put a hard cost-per-run target and a p95-latency target in the spec, right next to the feature description. If a feature can't hit them, it's not done — same as a screen that jank-scrolls isn't done, no matter how good it looks in a screenshot.
Short answer: probably not on day one. My production loop is a while-loop, a schema validator, a budget guard, and a trace logger — maybe a couple hundred lines. Boring, legible, and mine to debug. The heavy agent frameworks I tried hid exactly the layer I most needed to see: the raw tool call, the exact observation, the reason the loop ended. When production misbehaves, an abstraction you can't step through is a liability, not a convenience.
That's not framework-hatred — it's sequencing. Build the loop yourself first so you understand where the failure modes live. Then, if you're managing many tools or complex routing and you know precisely which layer you want help with, adopt a framework with your eyes open. Reaching for one before you've felt the failure modes just means debugging someone else's abstraction on top of your own bug.
The happy-path plan-act-observe loop is the easy 20%. The product is the other 80%, and it's ordinary engineering wearing an AI costume. Build the loop in an afternoon. Then spend the next month on the 80% that decides whether anyone can actually trust it in production.