How I choose an LLM in production: build a private eval, route cheap models to the easy 80%, hide every provider behind one interface, and re run the drill quarterly.
"What's the best model right now?" I get asked this maybe twice a week, and I've stopped giving a straight answer. Not to be difficult — because the question is malformed, and answering it honestly takes an hour and a whiteboard.
The real question hiding underneath is: best for what, at what latency, at what cost per thousand calls, with what blast radius if the provider deprecates the snapshot next Tuesday? I don't pick a model. I run a small portfolio of them, judged against a private eval that mirrors my actual workload, and I re-run that judgment every quarter because the ground moves that fast. What follows is exactly how I do model selection at Shpper — the eval harness, the four-axis decision matrix, the routing layer, the provider abstraction, and the quarterly drill — mistakes and wasted weekends included.
If you're an engineer or founder staring at a large language model API bill that's climbing faster than revenue, this is the LLM selection playbook I wish someone had handed me two years ago.
A leaderboard tells you how a model scores on someone else's tasks. Your product is not their tasks. I once watched a model that topped every public benchmark fall apart on our specific job — structured extraction from messy Arabic-and-English retail invoices — while a cheaper, supposedly "worse" model handled it fine, purely because it respected the JSON schema instead of getting creative with it.
The leaderboard reflex has three failure modes I keep running into:
So I stopped asking "what's best" and started asking "what's the right allocation across my traffic, given my evals and my budget, this quarter." That reframe — from picking a model to managing a portfolio of models — is the whole post. Everything else is plumbing.
The single highest-leverage thing I've done in this space was build a private eval suite. Not a fancy one. Around 60 cases in a JSONL file, hand-picked from real production traffic and real failures. It cost me one Friday afternoon and it has since saved me from at least three model migrations I would have regretted.
The rule I hold to: every case is a real task my product actually performs, with a real input and a gradeable expected output. No trivia. No "explain quantum computing to a five-year-old." If a case doesn't map to something a user pays us for, it doesn't belong in the set. A private eval is only useful to the exact degree that it mirrors your production distribution — the moment it drifts toward generic capability testing, you've just rebuilt a worse public benchmark.
Here's the shape of a single eval case:
{ "id": "invoice-extract-042", "task": "extract_line_items", "input": "Fatoora #A-2231, 3 x Widget @ 12.50 AED, VAT 5%...", "expect": { "currency": "AED", "vat_rate": 0.05, "line_count": 3 }, "grader": "json_fields"}The grading matters more than most people admit. The majority of my cases use a deterministic grader — exact field match, valid JSON, a regex, a number within tolerance — because deterministic checks are cheap, fast, and don't drift. They cost nothing to run and give the same verdict every time, which is exactly what you want when you're comparing two models on identical inputs.
I reserve an LLM-as-judge grader only for genuinely open-ended cases like tone or summary quality, and even then I keep a written rubric and hand-audit a sample of the judge's calls. A judge you never check is just a second model you blindly trust — and now you have two models to validate instead of one. If you can express the pass condition as code, express it as code.
The runner itself is deliberately boring:
Future<EvalResult> runCase(EvalCase c, ModelClient model) async { final started = DateTime.now(); final output = await model.complete(c.prompt); final latencyMs = DateTime.now().difference(started).inMilliseconds; return EvalResult( caseId: c.id, passed: c.grader.grade(output, c.expect), latencyMs: latencyMs, inputTokens: output.usage.inputTokens, outputTokens: output.usage.outputTokens, costUsd: model.priceFor(output.usage), );}Notice what I capture per case: pass/fail, latency, and cost. Those three columns are the entire decision. When a new model drops, I point the runner at it, wait a few minutes, and get a table instead of a feeling. A feeling is how you end up loyal to a provider for reasons you can't defend in a budget meeting.
One more habit worth stealing: I version the eval set in git alongside the code. When a real bug slips to production, the fix isn't just a code change — it's a new eval case, committed in the same PR, so the same failure can never ship twice. This is regression testing for probabilistic systems. The suite grows out of my scars, and after a year it's the most honest documentation of what my product actually needs from a model — far more honest than any spec I could write up front.
Once the eval runs against every candidate, I score each model on four axes. Not one blended number — four, because they trade against each other and the correct weighting depends entirely on the task.
| Axis | What I measure | Why it bites |
|------|----------------|--------------|
| Capability | Pass rate on the eval set for this task | The obvious one, and the only one leaderboards cover |
| Latency | p50 and p95 per call, never the average | p95 is what your user actually feels on a bad day |
| Cost | USD per 1k real calls, blended in/out tokens | This is your gross margin, quietly leaking |
| Lock-in | How hard to swap out — API shape, prompt format, unique features | The bill you pay later, when you're stuck |
The weighting flips per task type, and this is the part people skip. For a user-facing autocomplete, latency and cost dominate and I'll cheerfully lose a few capability points for them. For a nightly batch job that generates reports, latency barely registers — I'll wait 30 seconds for a smarter, cheaper-per-token model without blinking. For anything touching money or legal text, capability gets a hard floor and the other three axes only break ties. Same four axes, three completely different rankings.
A quick word on the latency column, because it's the one teams most often measure wrong. The average latency lies to you: a model that's fast 90% of the time and catastrophically slow the other 10% can post a lovely average while making your product feel broken. p95 and p99 are where the real user experience lives. If you only track one latency number, track p95, and track it per route — a slow tail on a background job is fine, a slow tail on autocomplete is a churn machine.
The axis people forget is lock-in, and here's my strong opinion on it: capability that only exists because of a provider-specific feature is not capability you own — it's capability you rent, and the landlord sets the price. A proprietary tool-calling format, a caching mechanism that only lives on one vendor's servers, a fine-tune you can't export — each of these is a leash, and I price the leash into the decision.
Sometimes the leash is worth it; a genuinely differentiated feature can carry a product. But I want it sitting on the table where I can see it, weighed against the migration cost it implies, not discovered eight months later during an outage when the switching cost has quietly compounded into a rewrite.
Here's the shift that fixed my bill. Most traffic is not hard. On a recent build I pulled a week of our LLM calls into a spreadsheet and roughly 80% of them were trivial — classify this message, extract these three fields, is this text spam. The remaining 20% were the genuinely hard reasoning tasks. And I was sending all of it to the biggest, most expensive model, because that was the path of least resistance and it "just worked."
The fix is a router: a cheap, fast model handles the common case and only escalates to the expensive one when it earns the right to. This is the single biggest LLM cost optimization sitting unclaimed in most stacks.
Future<Answer> route(Task task) async { // Cheap, fast model first — handles the easy 80%. final draft = await cheapModel.run(task); if (draft.confidence >= 0.85 && draft.schemaValid) { return draft; // Good enough. No premium call fired. } // Escalate the hard 20% to the expensive model. return expensiveModel.run(task);}Two things make this hold up in production:
On that project the router cut our model spend by a little over half with no measurable drop in output quality on the eval set. The 20% that genuinely needed the big model still got it. The 80% simply stopped overpaying for horsepower it never used. If you do one thing from this whole post, do this one.
I ship lean and I keep infrastructure cheap, so I'm allergic to anything that can hold my product hostage. Providers can and do: rate-limit you at the worst possible moment, deprecate a model snapshot with a few months' notice, quietly reprice, or have a regional wobble while your users are wide awake in Dubai and the provider's on-call engineer is asleep in another timezone.
The thing that saved me was a rule I now treat as non-negotiable: no provider SDK is allowed to leak into my business logic. One internal interface, every provider hidden behind it.
abstract class ModelClient { Future<Completion> complete(Prompt prompt, {ModelParams params}); Usage get lastUsage; String get modelId;}// Concrete adapters. Business code never imports these directly.class ProviderAAdapter implements ModelClient { /* ... */ }class ProviderBAdapter implements ModelClient { /* ... */ }My application code only ever knows about ModelClient. It never imports a vendor package. That single boundary — a plain adapter pattern applied to LLM providers — buys me three things I use constantly:
I learned this the expensive way. Early on I had a provider's SDK types threaded through a dozen files because it was faster in the moment. When I later wanted to trial an alternative, it turned into two days of surgery and a nervous deploy. Now the same trial is an afternoon. That abstraction costs you a little indirection up front and pays rent every single quarter after.
One caveat, because I've seen people take this too far: don't over-abstract into a mushy lowest-common-denominator interface that throws away each provider's best features. Mine exposes the common path cleanly but still lets an adapter opt into provider-specific tricks — prompt caching, structured outputs, native tool calling — behind a capability flag. Portable by default, powerful where it actually matters. The goal is a thin seam, not a framework.
A new model launches roughly every week now. I chase almost none of them on release day. Instead I run a fixed, boring drill, and boring is the entire point — it converts a hype cycle into a checklist I can execute without my ego getting involved.
ModelClient implementation and a price-table entry. Usually under an hour.The whole thing takes an afternoon, and most quarters the verdict is "no change, revisit next time." That's not a failure of the drill — that's the drill working. It exists so I can say no quickly and yes with evidence, instead of migrating on a press release and a gut feeling.
If you're an operator staring at an LLM line item that's climbing faster than revenue, here's the short version I'd give you over coffee:
The goal was never to always run the newest, smartest model. It's to run the right model for each slice of your traffic, know precisely why, and be able to change your mind in an afternoon when the facts change under you.