Structured Output From LLMs: A Retry-Repair Loop Your Parser Never Sees Through

Reliable structured output from LLMs needs three layers: constrained decoding, JSON Schema validation you own, and a bounded retry repair loop. Here's the pattern.

The first time I wired an LLM into a real product feature at Shpper, I did the naive thing: prompt the model to "return JSON", jsonDecode the response, move on. It worked in the demo. Then it hit real traffic and I started getting FormatException at 2am because the model wrapped its JSON in a ``` `json ``` fence, or added a cheerful "Here's the data you asked for!" preamble, or trailed a comma before the closing brace. A model that's right 97% of the time is still wrong on thousands of requests a day. Reliable structured output isn't a prompting trick — it's a small pipeline, and the last stage is a repair loop your parser never sees through.

This is the pattern I reach for every time I need an LLM to hand back a typed object instead of prose: contact extraction, invoice parsing, classification against a fixed label set, turning a messy paragraph into a database row. The shape is identical every time, and once you internalize it you stop firefighting malformed JSON for good.

Why "return JSON" fails in production

The failure modes are boring and relentless, which is exactly why they're worth naming. Prompt-only JSON breaks in a handful of predictable ways:

• Markdown fences. The model wraps the object in a ``` `json `` block, so your raw string starts with backticks, not {`.
• Conversational preamble or trailer. "Sure! Here's the JSON:" gets prepended, or a "Let me know if you need anything else!" gets appended — either way the payload isn't parseable end to end.
• Trailing commas and single quotes. Valid-looking to a human, invalid to a strict JSON parser.
• Type drift. You asked for a string, you got a number. You asked for an array, you got a comma-joined string. Syntactically fine, semantically wrong.
• Missing or hallucinated fields. The model omits the one field you actually needed, or invents a confidence: 0.9 you never asked for.

None of these are exotic. They're the median Tuesday. The mistake is treating them as bugs to squash one by one instead of a class of failures to absorb architecturally.

The three layers, from strongest to weakest

You have three tools to force structure, and you should reach for them in this order — strongest guarantee first.

• Constrained decoding (JSON mode / schema-enforced output). The provider constrains token sampling so the output is guaranteed to be syntactically valid JSON, and with a supplied JSON Schema, guaranteed to match the shape. This is the strongest guarantee because it operates at the sampling layer, not the prompt layer — the model literally cannot emit a token that would break the grammar.
• Tool / function calling. You describe a function with a typed parameter schema; the model emits a structured call to it. This is the same constrained-decoding machinery wearing a different hat, and it's the cleanest fit when the structured object is an action — create_invoice, extract_contact, schedule_meeting. If your data extraction is really "the model deciding to do a thing", model it as a tool and let the SDK enforce the arguments.
• Prompt-and-pray plus validation. You ask for JSON in the prompt and validate what comes back. This is the weakest layer and, unfortunately, the one you fall back to whenever a provider, model, or gateway doesn't support the stronger modes. Older models, some open-weights deployments behind a proxy, and certain streaming paths still land you here.

Here's what people miss: even the strong layers don't free you from validation. JSON mode guarantees syntactic validity — that you can jsonDecode it. It does not guarantee the model filled in the field you needed, respected your enum, or didn't drop a null where you require a string. Schema-enforced modes are much better, but a schema can't express every business rule: this date must be after that date; this array must be non-empty when type == "premium"; this email must belong to a domain you support. So the architecture is always the same three moves: generate as constrained as the provider allows, then validate against your own source of truth, then repair.

Validate against a schema you own

Do not hand-roll if (json['name'] == null) checks scattered across your codebase. Define the contract once and validate against it. In the Dart-heavy world I live in that means a real model class with a strict parser; on a Node or Python backend I'll use JSON Schema directly, or a Zod / Pydantic-style validator that doubles as the schema I send to the provider.

The parser has one job: turn any input into either a valid typed object or a structured error that describes exactly what's wrong — because that error is the input to the repair step. A parser that throws a generic "invalid" is useless here; the specificity of the error determines the quality of the repair.

class SchemaError implements Exception {  final List<String> messages;  SchemaError(this.messages);  @override  String toString() => 'SchemaError: ${messages.join('; ')}';}class ContactRecord {  final String name;  final String email;  final String? company;  ContactRecord({required this.name, required this.email, this.company});  /// Returns the record, or throws SchemaError with a machine-useful message list.  factory ContactRecord.parse(Map<String, dynamic> json) {    final errors = <String>[];    final name = json['name'];    final email = json['email'];    if (name is! String || name.trim().isEmpty) {      errors.add('"name" must be a non-empty string');    }    if (email is! String || !email.contains('@')) {      errors.add('"email" must be a valid email address containing "@"');    }    if (errors.isNotEmpty) {      throw SchemaError(errors); // carries the list to the repair loop    }    return ContactRecord(      name: name as String,      email: email as String,      company: json['company'] as String?,    );  }}

The load-bearing detail is that SchemaError carries a specific, actionable list — not just "invalid JSON". Vague errors produce vague repairs. "email" must contain "@" gets fixed on the next turn; "validation failed" gets you the same broken output again.

A useful discipline: the validator you run in your code and the schema you hand the provider should be derived from the same definition. When they drift, you get output that passes the provider's schema check but fails yours — the worst kind of silent mismatch. With Zod or Pydantic you literally generate the JSON Schema from the validator, so they can't disagree.

The retry-repair loop

When validation fails, you don't throw the whole request away. You hand the model back its own broken output plus the exact validation errors and ask it to fix them. Models are remarkably good at this — repairing a nearly-correct object is a far easier task than generating one from scratch, so the second attempt succeeds the overwhelming majority of the time.

A few things separate a production-grade loop from a foot-gun:

• Bound the retries. Two, maybe three attempts. If the model can't produce valid output in three tries, the problem isn't transient — you need to fail loudly to a fallback, not spin and burn tokens.
• **Feed back the errors, not just "try again".** The model needs to read "email" must contain "@", not learn that it failed. The error list is the entire value of the loop.
• Strip fences and prose before parsing. Even in JSON mode, defensively extract the first {...} block. This one cheap step eliminates a huge class of weak-mode failures.
• Escalate, don't just repeat. On the final attempt I'll bump to a stronger model or drop the temperature to 0. A repair is a good moment to spend more compute — you already know the cheap path failed, so paying more only on the retry keeps your average cost low while raising the ceiling on reliability.
• Set a deadline, not just a retry count. Each repair round is another network round-trip. If the feature has a latency budget, enforce it — a repair that lands after the user gave up isn't a success.

/// Defensive: pull the first balanced JSON object out of a raw string,/// so fences and preamble ("Here's your JSON:") don't blow up jsonDecode.Map<String, dynamic> extractJson(String raw) {  final start = raw.indexOf('{');  final end = raw.lastIndexOf('}');  if (start == -1 || end == -1 || end < start) {    throw const FormatException('no JSON object found');  }  return jsonDecode(raw.substring(start, end + 1)) as Map<String, dynamic>;}Future<ContactRecord> extractContact(String source) async {  String rawText = await callModel(    prompt: buildPrompt(source),    jsonMode: true, // strongest mode the provider offers  );  for (var attempt = 0; attempt < 3; attempt++) {    try {      return ContactRecord.parse(extractJson(rawText));    } on SchemaError catch (e) {      log.warning('repair attempt $attempt: ${e.messages}\nraw: $rawText');      if (attempt == 2) rethrow; // give up -> caller handles fallback      rawText = await callModel(        prompt: repairPrompt(previous: rawText, errors: e.messages),        jsonMode: true,        // On the last shot, spend more: lower temp / stronger model.        temperature: attempt == 1 ? 0.0 : 0.2,      );    } on FormatException catch (e) {      if (attempt == 2) rethrow;      rawText = await callModel(        prompt: repairPrompt(previous: rawText, errors: [e.message]),        jsonMode: true,        temperature: 0.0,      );    }  }  throw StateError('unreachable');}

The repairPrompt is boring but load-bearing. Something like: "The previous response failed validation. The errors were: [list]. Return only the corrected JSON object with no explanation, no markdown, and no code fences." Boring, deterministic, effective. Resist the urge to make it clever — the whole point is to reduce the model's freedom to the single act of patching the fields you named.

Note that I catch FormatException (couldn't even parse it) separately from SchemaError (parsed but wrong). Both feed the repair loop, but distinguishing them in your logs tells you whether the model is failing at syntax (a weak-mode problem) or semantics (a schema-clarity problem) — and the fixes are different.

Practical notes from production

A few opinions I've formed after shipping this pattern more than once:

• Prefer flat schemas. Deeply nested objects and unions are where models drift and where your validation gets brittle. If you can flatten { address: { city, zip } } into city, zip, do it — then reassemble on your side. Every level of nesting is another place the model can put a field one layer too deep.
• Enums over free text. If a field has five valid values, make it an enum in the schema. Constrained decoding will enforce it at the token level, and even in weak mode a closed set collapses the space the model can get wrong.
• Log the raw output on every repair. These logs are gold. They tell you which field the model keeps fumbling, and the fix is usually a one-line clarification in the field's schema description, not more retry logic. Ambiguous field names ("date" — of what? in what format?) are the number-one cause of repeat repairs.
• Descriptions are prompt real estate. The description on each schema field is read by the model. A field called due_date with description "ISO 8601 date, e.g. 2026-07-09" fails far less than a bare due_date. Spend your clarity budget there before you spend it on retries.
• Make streaming a deliberate choice. Streaming and strict structured output are in tension — you can't validate half an object. For structured extraction I buffer the full response and validate once. Save streaming for the chat surface where partial tokens are the product.
• Treat repair rate as a metric. If your loop fires on 15% of requests, your prompt or schema is the bug, not the model. A healthy pipeline repairs rarely; the loop is a safety net, not the primary mechanism. Alert on the repair rate the same way you'd alert on an error rate — a spike means a prompt regression or a model change upstream.
• Idempotency and cost. Because a single logical call can fan out to three model calls, make sure retries are safe to repeat and that you're accounting for the worst-case token spend, not the happy-path one, in your budgeting.

Key takeaways

• Structured output is a layered contract, not a switch you flip. Constrain the model as hard as the provider allows (constrained decoding > tool calling > prompt-and-pray), then always validate on your side.
• Even JSON mode needs validation. Syntactic validity is not semantic correctness — schemas can't encode your business rules, so own the final check.
• Validate against a schema you own and make the error specific and actionable, because that error is the fuel for the repair step.
• The repair loop feeds errors back, bounds retries to two or three, strips fences defensively, and escalates on the last attempt (lower temperature, stronger model).
• Keep schemas flat, prefer enums, write good field descriptions, and log raw output on every repair — most "model problems" are really schema-clarity problems.
• Watch the repair rate as a first-class metric. A loop that fires constantly is telling you the prompt or schema is broken, not that the safety net is working.

Wrap-up

Reliable structured output is not one thing you turn on — it's a layered contract. Constrain the model as hard as the provider lets you, validate against a schema you own rather than the model's promises, and wrap it in a bounded repair loop that feeds specific errors back to the model. Get those three layers right and your parser genuinely never sees malformed JSON — the loop absorbs it upstream. The whole thing is maybe forty lines of code, and it's the difference between a demo that impresses and a feature that survives Monday morning traffic.