Shipping AI Features in Flutter Without Setting Your API Bill (or Your Keys) on Fire

Ship LLM features in Flutter without leaking API keys or blowing your bill: a thin backend proxy, token streaming, secure tool use, and prompt caching that cuts cost.

The first "AI feature" I ever shipped in a Flutter app was a single button. You tapped it, waited about ten seconds staring at a spinner, and then the whole LLM response dumped onto the screen at once. It technically worked. It also felt broken, leaked an API key inside the app bundle, and cost roughly triple what I'd budgeted because I was re-sending the same fat 900-token system prompt on every single tap. Three failures, one button.

I've since shipped LLM features into production apps that real users touch daily, and I now watch teams walk into the exact same three traps I did — keys on the device, blocking UI, and a bill that grows faster than usage. So this is the architecture I actually reach for now, the moving parts, why each one exists, and where the money and the milliseconds genuinely go. Not theory. The version that survived contact with a live app and an invoice I had to explain to a co-founder.

Whether you're wiring up OpenAI, Anthropic, Gemini, or any other model provider, the same mental model and the same code patterns keep you out of trouble.

The one-slide mental model for AI in a Flutter app

Before any code, hold this shape in your head, because everything below is just a detail of it:

Flutter client → your thin backend → the model provider.

The client renders. Your backend holds the secret, runs the tools, and enforces the budget. The provider does the thinking. Three boxes. If you find yourself putting model logic in the app or business logic in the proxy, you've drifted, and the pain shows up later as either a leaked key, an unmaintainable client, or a bill nobody can account for.

It's worth being explicit about what lives where, because the boundaries are the whole point:

| Layer | Owns | Never touches |

| --- | --- | --- |

| Flutter client | UI, streaming render, local state | The API key, tool execution, business rules |

| Thin backend | Auth, key, rate limits, tool-use loop | Product logic that belongs in a real service |

| Model provider | Generation, reasoning, tool selection | Your database, your secrets, your users |

Now the details.

Rule zero: the API key never touches the device

If you call the model provider directly from Dart, your key ships inside the app binary. This isn't a theoretical risk you mitigate later — it's a certainty you're choosing now. Anyone can pull an APK apart with apktool, grep the strings, and walk out with your key in about the time it takes to make coffee. iOS is barely harder: proxy the traffic through something like mitmproxy on a jailbroken device or a simulator, and the Authorization header is sitting right there in plaintext.

This is true no matter how you inject the key. Compile-time constants via --dart-define, values read from a bundled .env, secrets stuffed into Info.plist or strings.xml — they all end up in the shipped binary, and the binary is on someone else's device. Flutter compiles Dart to native code, which feels like it should hide things, but string constants survive to the binary just fine.

From there attackers don't need to "hack" anything. They just use your key. They'll run their own workloads on your dime, and the first you'll hear of it is a billing alert — if you set one up — or the invoice itself. I've seen a leaked key rack up a four-figure bill over a long weekend before anyone noticed. Obfuscation doesn't save you either; obfuscation raises the cost of extraction from five minutes to fifteen. That's not a security boundary, it's a speed bump.

So the non-negotiable first decision: put a thin backend between the app and the model provider. The app talks to your endpoint. Your endpoint holds the provider key as a server-side secret and forwards the request.

"Thin" is load-bearing in that sentence. This backend is not where your business logic lives. It's a key vault, plus a proxy, plus the one place you enforce auth, rate limits, and a token budget. On Firebase — my default, because I keep infrastructure close to zero cost — this is a small HTTPS function that verifies the caller's auth token and streams the model response straight back:

// Thin proxy — validate the user, inject the key, stream the model back.exports.chat = onRequest(async (req, res) => {  const user = await verifyAuth(req);  // reject anonymous / abusive callers  await enforceRateLimit(user.uid);    // per-user budget guard  const upstream = await fetch(MODEL_ENDPOINT, {    method: "POST",    headers: {      authorization: `Bearer ${process.env.MODEL_API_KEY}`,      "content-type": "application/json",    },    body: JSON.stringify({ ...req.body, stream: true }),  });  if (!upstream.ok) {    res.status(502).json({ error: "upstream_failed" });    return;  }  res.setHeader("content-type", "text/event-stream");  res.setHeader("cache-control", "no-cache");  upstream.body.pipe(res);  // forward the SSE stream verbatim});

The key lives in process.env on the server, gated behind your auth check. The app never sees it, never stores it, never transmits it. And notice what the proxy does not do: it doesn't build prompts, it doesn't own conversation state, it doesn't make product decisions. The moment you're tempted to add "just a little logic" here, stop and ask whether it belongs on the client or in a proper backend service. This function should stay boring enough that you never think about it again.

Don't forward the request body blindly

One thing people skip: don't forward req.body verbatim forever. Whitelist the fields you accept. If a client can pass arbitrary parameters straight to the provider, a curious user can flip on an expensive mode, swap in a pricier model, or crank max_tokens to the ceiling and hand you the bill. The proxy is thin, not gullible. A tiny allow-list closes the hole:

// Only these fields reach the provider. Everything else is dropped.function sanitize(body) {  return {    messages: body.messages,    model: ALLOWED_MODELS.has(body.model) ? body.model : DEFAULT_MODEL,    max_tokens: Math.min(body.max_tokens ?? 512, MAX_TOKENS_CEILING),  };}

Now a hostile client can't turn your getWeather helper into a free general-purpose LLM sitting behind your paid key — a real abuse pattern the moment your endpoint is discoverable.

Stream tokens — don't make users watch a spinner

An LLM generating a few hundred tokens can take several seconds to finish. If you wait for the whole thing, the user stares at a loading indicator, decides it hung, and taps again — which, delightfully, doubles your cost for the same answer. Stream the response and render tokens as they land. Perceived latency collapses because feedback starts in a few hundred milliseconds instead of after the full generation.

Providers expose this as server-sent events (SSE). Forward that stream through your proxy untouched — that's the upstream.body.pipe(res) line above — then consume it in Dart as a Stream<String> and append chunks to your UI state:

Stream<String> streamReply(String prompt) async* {  final req = http.Request('POST', Uri.parse('$backend/chat'))    ..headers['authorization'] = 'Bearer $firebaseIdToken'    ..headers['content-type'] = 'application/json'    ..body = jsonEncode({'prompt': prompt});  final resp = await http.Client().send(req);  if (resp.statusCode != 200) {    throw ChatException('backend returned ${resp.statusCode}');  }  await for (final line in resp.stream      .transform(utf8.decoder)      .transform(const LineSplitter())) {    if (!line.startsWith('data:')) continue;    final data = line.substring(5).trim();    if (data == '[DONE]') return;    final delta = jsonDecode(data)['delta'] as String?;    if (delta != null) yield delta;  }}

In the widget layer, feed that into a StreamBuilder, or — my preference for anything real — accumulate into a state notifier (Riverpod, Bloc, or a plain ChangeNotifier) so the buffer survives rebuilds and you can hold onto the partial text if the user backgrounds the app mid-generation. A StreamBuilder alone will lose your accumulated text on any rebuild unless you buffer outside it, and that's a bug you'll only notice on a real device with a real keyboard popping up.

A few UX details matter more than they look:

• Render partial text immediately and keep a blinking cursor at the tail so the output reads as alive. The difference between "instant" and "sluggish" is entirely about when the first token appears, not the last.
• Auto-scroll to the bottom as tokens arrive, but stop the instant the user scrolls up. Nothing is more infuriating than fighting the scroll position while trying to read something halfway up.
• Give an explicit stop button. Users will start a generation, realize it's the wrong question, and want out. Cancelling the HTTP request stops the tokens — and stops you paying for output nobody will read. Wire the stop button to Client.close() or an http.Request you can abort, and make sure your proxy notices the dropped connection and cancels upstream too. A stop button that only stops the display while the backend keeps generating is theatre, and it still costs you.

When to fall back to non-streaming

Streaming isn't free complexity-wise. For short, structured outputs the app will parse anyway — a classification, an extracted field, a yes/no route — skip streaming entirely. The user never sees those tokens, so there's no perceived-latency win, and a clean single JSON response is far easier to validate than a reassembled stream. Stream what humans read; buffer what machines parse. That one rule decides the transport for you every time.

Tool-use and function calling: let the model call your code

The moment your feature needs real data — the user's actual orders, a live price, a record in your database — prompt text alone won't cut it. The model doesn't know your data, and if you paste it all into the prompt you're both paying for those tokens every call and leaking data you didn't need to. This is where tool-use (function calling) earns its keep. You describe functions the model may call, and instead of answering from thin air it responds with a structured request to invoke one.

The critical property: tools execute on your backend, never on the device. The flow is a small loop that lives entirely server-side:

• App sends the user message to your proxy.
• Proxy forwards it to the model along with a list of available tool schemas.
• Model replies: "call getOrderStatus with {orderId: 123}".
• Your backend runs that function against your own database, appends the result to the conversation, and calls the model again.
• Model produces the final natural-language answer, which you stream to the app.

Here's the shape of that loop on the backend, trimmed to the essentials:

async function runWithTools(messages) {  for (let step = 0; step < MAX_TOOL_STEPS; step++) {    const reply = await callModel({ messages, tools: TOOL_SCHEMAS });    if (reply.stop_reason !== "tool_use") return reply;  // done    const call = reply.toolCall;    const args = validateArgs(call.name, call.input);     // never trust raw args    const result = await TOOLS[call.name](args);          // hits YOUR db, server-side    messages.push(reply, { role: "tool", name: call.name, content: result });  }  throw new Error("tool loop exceeded step budget");}

Three things I learned the expensive way.

First, bound the loop. A model can, occasionally, get into a call-a-tool-again cycle, and without a MAX_TOOL_STEPS ceiling each iteration is another paid round trip. I've watched a runaway loop quietly make eleven model calls to answer one question. Cap it, and treat hitting the cap as an error you log, not a silent retry.

Second, keep tool schemas tight — clear names, minimal parameters, a one-line description each. Vague schemas make the model guess, and guessing is exactly where hallucinated arguments come from. getData with a free-form query string is an invitation for nonsense; getOrderStatus(orderId: string) is not. The narrower the surface, the less the model can misuse it.

Third, and most important, validate and authorize every argument before it touches your database. Treat tool calls as untrusted input, because functionally they are — the model is a text generator that has been socially engineered by whatever the user typed. If your tool takes an orderId, confirm that order actually belongs to the authenticated user before returning it:

async function getOrderStatus({ orderId }, ctx) {  const order = await db.orders.get(orderId);  if (!order || order.ownerUid !== ctx.uid) {    throw new ForbiddenError("order not visible to caller");  }  return { status: order.status, eta: order.eta };}

Skip that check and you've built a very polite way for user A to read user B's orders by asking nicely. Tool-use doesn't get to skip authorization; it needs it more, because the request path now runs through a component that will happily generate whatever orderId the conversation nudges it toward.

Prompt design that survives production

Prompts written on a good day fall apart under the variety of real user input. A few habits that consistently hold up:

• Separate the system prompt from user input. The system prompt sets role, tone, and hard constraints; user text goes in its own turn. Concatenating them into one blob is how you get prompt injection, where a user types "ignore your instructions" and, because it's the same message, sometimes the model listens.
• Ask for structured output when you'll parse it. If the app needs fields, request JSON and validate it against a schema. Never regex values out of prose — that pipeline breaks the first time the model adds a friendly sentence before the data.
• Pin the response shape with an example, not a paragraph of instructions. Models imitate format far more reliably than they follow prose rules. One concrete example of the exact JSON you want beats three sentences describing it.
• Version your prompts and log which version produced which output. When quality shifts after you tweaked a prompt three weeks ago, you'll want to diff. A prompt is code; treat it like code — put it in source control, not in a database field someone edits live.
• Keep the system prompt stable. Beyond consistency, a stable prefix is what makes prompt caching possible — and that's the single biggest cost lever you have. More on that next.

LLM cost control is an architecture concern, not an afterthought

You pay per token, in and out, and it compounds faster than intuition suggests. A chat feature feels cheap per message until you multiply by a stable prefix, a growing conversation history, and a few thousand daily users. The levers I actually pull, roughly in order of impact:

• Cache the stable prefix. Most providers let you mark the unchanging part of your prompt — system instructions, tool schemas, few-shot examples — as cacheable, so you're not billed full input rate to re-process it on every call. That fat 900-token system prompt from the top of this post? Marking it cacheable cut its per-call input cost dramatically on the requests that hit the cache. This is almost always the biggest single saving, and it's why "keep the prefix stable" is a cost rule, not a style preference. To actually hit the cache, the cacheable content must be identical and at the front — reorder or rewrite it and you re-pay full price.
• Right-size the model per task. Classification, extraction, routing, and short summaries do not need your most capable model. Reserve the expensive tier for genuinely hard reasoning and route everything else to a cheaper, faster one. A router that picks the model based on task type paid for itself in a week for me. The instinct to use the best model everywhere is how bills balloon.
• Cap max_tokens. An unbounded response is an unbounded bill and an unbounded wait. Set a ceiling that fits the UI — if a card shows two lines, the model doesn't get to write six paragraphs.
• Trim conversation history. You don't need to resend the entire chat forever. Keep a sliding window of recent turns, or summarize older ones into a compact note. Otherwise every message in a long conversation costs more than the last, purely from re-sending history — the growth is quadratic over a session, which is the part people don't feel coming.
• Rate-limit per user on the backend. One buggy client or one abusive account should never run up an open-ended tab. This is where the auth check from the proxy earns its place — you can't rate-limit per user if you don't know who the user is.
• Log token usage per request. You cannot control what you don't measure. Store input and output token counts, the model used, and whether the cache hit, keyed by feature. Then you can answer "which feature costs what," which is the question your co-founder will eventually ask, and you'll want a dashboard instead of a shrug.

Design the sad path: failure and graceful degradation

Providers have bad days. Rate limits, timeouts, the occasional 500. Decide up front what your app does when the model is unavailable — because "infinite spinner" is not a plan. Sometimes it's a cached response, sometimes a cheaper fallback model, sometimes an honest "the assistant is busy, try again." Build the sad path deliberately: a bounded retry with backoff for transient errors, a hard fail with a readable message for the rest, and never a silently doubled request. The AI feature that gracefully says "not right now" beats the one that hangs, every time, and users forgive the former.

Key takeaways

• Get the key off the device. A thin authenticated backend proxy is non-negotiable — the key lives in server env, gated behind an auth check, never in the app binary.
• Keep the proxy thin but not gullible. Whitelist the fields it forwards so a client can't swap models or crank max_tokens on your dime.
• Stream what humans read; buffer what machines parse. Streamed tokens crush perceived latency; single JSON responses are easier to validate.
• Buffer streaming state outside StreamBuilder and give users a stop button that actually cancels upstream.
• Run tool-use server-side, bound the loop, and authorize every argument as if a stranger typed it — because effectively one did.
• Keep the system prompt stable so it's cacheable. Prompt caching is the single biggest cost lever you have.
• Right-size the model, cap tokens, trim history, rate-limit per user, and log token usage from day one.
• Design the sad path. A graceful "not right now" beats an infinite spinner.

Takeaway

The shape that holds up in production is boringly consistent: a Flutter client that renders streamed tokens with a genuinely responsive UI, talking to a thin authenticated backend that holds the keys, runs the tools server-side, and enforces the budget. Everything else is a variation on those three boxes.

Get the key off the device — that one is non-negotiable, not a nice-to-have. Stream what humans read and buffer what machines parse. Keep your prompt prefix stable so it's cacheable, because that's where the money hides. Bound your tool loops. Validate every tool argument as if a stranger typed it, because effectively one did. And measure token usage from day one so cost is a number you watch, not a surprise you receive. Do that, and adding AI to your Flutter app stops being a liability you nervously monitor and becomes a feature you can actually afford to ship — and keep shipping.