LLM evals before features: build eval harnesses, golden datasets, assertions, rubrics, and LLM as judge scoring, then gate CI so prompt changes can't silently regress.
I broke a production feature with a one-line prompt change and didn't find out until support forwarded me the screenshots two days later. The feature was a support-reply drafter for one of our products: a user pastes an angry email, we generate a calm, on-brand response. I edited the system prompt to make replies shorter, ran a handful of examples in my head, read them, nodded, deployed on a Friday. Classic.
By Monday the shorter replies were dropping the one thing that mattered — the actual answer. My "improvement" had traded correctness for brevity on maybe a third of real tickets, and I had no way of knowing, because I'd tested it the way you test a mood: I looked at a few outputs and decided they felt fine. That was the moment I stopped treating LLM features like magic and started treating them like software. Software you can't measure is software you can't ship, and I'd been shipping blind for months.
This post is the system I built in response — LLM evals before features — and why I now consider an eval harness the first thing you build in an AI product, not the last. If you take one idea away, take this: eyeballing model output is not testing, and a repeatable eval is the only instrument that tells you whether a prompt change made the product better or worse.
Here's the uncomfortable part. If that had been a regular code change — a refactor of a pricing function, say — nobody on my team would have let it merge without a test. We have CI. We have coverage gates. We argue about whether a util deserves its own spec. And yet the single most unpredictable component in the whole product, a black box that returns different text every time you call it, was going out on vibes.
Non-deterministic output makes this worse, not better. With deterministic code, eyeballing one run tells you something real. With an LLM, the output you happened to look at is a sample of size one from a distribution you can't see. You changed the prompt, you sampled once, it looked good — that tells you almost nothing about the other thousand shapes of input hitting production. Temperature, retries, provider-side changes, and the sheer variety of real user input all mean the one reply you inspected is not representative. Eyeballing LLM output isn't light testing. It's not testing.
So the first thing I built for our next AI feature wasn't the feature. It was the eval harness. Before the prompt, before the UI, before the streaming. That inversion — evals before features — is the sharpest opinion I hold about building with large language models, and it's the whole argument of this post. If you're shipping an AI feature without a way to measure it, you don't have a feature. You have a demo that happens to be in production.
People hear "test" and picture assertEquals. That breaks immediately here, because there is no single correct summary of a document, no one right support reply, no canonical answer to a fuzzy question. So we redefine the unit.
An eval is a repeatable measurement of output quality against a fixed set of inputs. Three moving parts:
The mental shift is the whole game: you stop asking "is this output correct?" and start asking "did quality go up or down versus the last version?" You're not proving the model is right. You're detecting movement. That's a much cheaper and much more useful question, and unlike "is it correct," it's actually answerable. A regression eval doesn't need ground truth — it needs a stable baseline and a scorer that moves in the same direction as quality.
The runner is the least interesting part, and that's the point — keep it dumb. Load cases, call the model, score, aggregate, exit non-zero if the aggregate drops below a threshold.
async function runEval(cases: Case[], prompt: string, model: string) { const results = []; for (const c of cases) { const output = await callModel(model, prompt, c.input); const score = await scoreCase(c, output); // assertions + rubric + judge results.push({ id: c.id, ...score }); } const aggregate = results.reduce((s, r) => s + r.total, 0) / results.length; return { aggregate, results };}Everything hard lives in scoreCase. Let's go there.
Scorers live on a spectrum from cheap-and-strict to expensive-and-fuzzy. In practice I use all three, layered, cheapest first.
Assertions are deterministic checks. They're free, fast, and they never hallucinate. Most quality problems are boring and catchable this way — did we return valid JSON, is the reply under 1200 characters, did it avoid banned phrases, does it cite a real order ID that appeared in the input.
function assertions(input: Case, output: string): Score { const failures: string[] = []; if (output.length > 1200) failures.push("too_long"); if (!/[.!?]$/.test(output.trim())) failures.push("no_terminal_punct"); if (input.orderId && !output.includes(input.orderId)) { failures.push("dropped_order_id"); } if (/as an ai language model/i.test(output)) failures.push("meta_leak"); return { pass: failures.length === 0, failures };}Every one of these came from a real failure. I don't write assertions I imagine; I write assertions for regressions I've already been burned by. That dropped_order_id check is my Friday deploy, encoded so it can never happen twice. This is the single highest-leverage habit in the whole system: every incident becomes a permanent, free, deterministic guardrail.
Rubrics handle the fuzzy stuff assertions can't. Instead of one yes/no, you break "is this a good reply?" into scored dimensions: Does it answer the question (0–2)? Is the tone appropriate (0–2)? Is it factually grounded in the input (0–2)? A rubric forces you to define quality before you measure it, which is most of the value — half the arguments about whether the AI got "better" are really arguments about an undefined word. Writing the rubric ends them.
LLM-as-judge is where you hand the rubric to another model and ask it to score. It's powerful, it scales in a way human labeling never will, and it's where most teams get fooled. Judges lie in specific, learnable ways:
The fix isn't to abandon judges — it's to distrust them until proven. I calibrate every judge against a batch I've scored by hand. If the judge and I disagree more than roughly 15% of the time, the judge prompt is broken, not my labels, and I iterate on the judge prompt like any other prompt. And I never let a judge score a dimension an assertion can score deterministically. You don't ask a language model whether the JSON parsed. You parse it. Reserve the judge for genuine subjectivity — tone, helpfulness, groundedness — where no regex can reach.
Push everything as far up (toward cheap and deterministic) as it will go.
The dataset is where evals live or die, and it's the part people fake. The temptation is to sit down and invent 20 clever test cases. Those cases test your imagination, not your product — and your imagination is not where your bugs come from.
My rule: the golden set is a museum of real failures. Every case earns its place because something actually went wrong. The pipeline:
We started with 8 cases. We're at around 140 now, and every single one is a scar. The set grows on the schedule of your pain, which is exactly the right schedule. A case invented in a vacuum protects against a bug that never existed; a case pulled from a real escalation protects against the exact thing that already cost you a customer's trust.
Keep the set diverse on the axes that actually matter:
Twenty ugly real cases beat two hundred synthetic pretty ones. If you must bootstrap before you have production data, generate synthetic cases to get moving — but treat them as scaffolding and replace them with real scars as fast as they arrive.
An eval you run manually is an eval you'll skip on a Friday — and Friday is exactly when the dangerous changes ship. The whole point is to make silent regression impossible, and that means CI.
The tricky bit is that eval runs cost money and wall-clock time — you're making real API calls. So I split the work by stakes. Assertions plus a small smoke set of ~15 cases run on every pull request. The full golden set runs on merge to main and nightly.
# .github/workflows/evals.ymlname: evalson: pull_request: paths: ["prompts/**", "src/ai/**"]jobs: smoke: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npm run eval -- --set=smoke --threshold=0.9 env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}The --threshold is the load-bearing part. The eval doesn't just print pretty numbers; it exits non-zero if the aggregate score drops below a bar, so a bad prompt change fails the check like a broken unit test and blocks the merge. The paths filter means we only spend tokens when something AI-adjacent actually changed — no point re-running a judge because someone edited the README.
One rule that saved us real grief: treat the prompt as code. Prompts live in the repo, in version control, reviewed in the same PR as the code that calls them. A prompt change is a code change and goes through the same gate and the same eval. The number of "who edited the prompt in the dashboard?" mysteries dropped to zero the day we did that — and every prompt change now has a diff, an author, and an eval result attached to it.
Two things move under you. You change prompts. And the model changes — a provider ships a new version, deprecates an old one, or silently retunes behavior behind the same model name. Your eval score is the only instrument that catches either kind of drift.
Every eval run writes a row: timestamp, git SHA, model id, score per dimension, cost, and latency. Dump it in a cheap table — for us it's a Firestore collection, because keeping infra at zero cost is a religion — and you can plot quality over time.
run model answer tone grounded cost2026-05-02 claude-x.y 1.81 1.90 1.72 $0.142026-05-19 claude-x.y 1.79 1.88 1.71 $0.14 prompt tweak2026-06-11 claude-x.z (new) 1.62 1.93 1.55 $0.09 model bump
That third row is the whole reason this system exists. The new model was cheaper and slightly warmer in tone — and measurably worse at answering and staying grounded. Without the eval I'd have upgraded for the cost savings and quietly degraded the product for weeks. With it, the tradeoff was a number on a table I could argue about, not a surprise buried in three weeks of support tickets. When you can see drift, a model migration becomes a decision instead of a gamble — you weigh the cost win against the grounding loss with real data, and you own the call.
Evals aren't free. Every run is tokens and wall-clock, and there's a real temptation to evaluate everything on everything constantly. Don't. Match the rigor to the stakes.
| Tier | When it runs | Set size | Scorers |
| --- | --- | --- | --- |
| Smoke | Every PR | ~15 | Assertions + cheap rubric |
| Full | Merge to main, nightly | Full golden set | Assertions + rubric + judge |
| Deep | Model migration, quarterly | Full + hand-scored batch | Everything + human calibration |
A few things that keep the bill sane and the signal high:
(prompt hash, model id, case id) and don't pay to recompute what you already know.The goal is not maximum coverage. It's the most confidence per dollar, with the expensive scorers reserved for the decisions that actually deserve them.
The failure mode I've watched happen elsewhere: one engineer builds a beautiful harness, becomes "the eval person," goes on holiday, and the whole thing rots because nobody else touches it. Evals only work if they're a team habit, not a hero project.
What made it stick for us:
The mindset shift is the real deliverable. Once a team internalizes that LLM output is measurable, "I think it's better" stops being an acceptable sentence in a review. Someone will ask, kindly, "what does the eval say?" That question is the entire culture.