Streaming LLM UX in Flutter: cut perceived latency with time to first token, optimistic rendering, buffered repaints, real cancellation and incremental Markdown.
A large language model that takes eight seconds to produce a full answer can feel either painfully slow or perfectly snappy — and the difference has almost nothing to do with the model. It's in how you render the stream. I've shipped a few LLM-backed features in production now, and the single biggest lesson is that perceived latency is a UI problem, not an inference problem. The model isn't getting faster; you're just spending its time better.
This is a deep dive on the interaction layer that sits between the token stream and the user's eyes. Not prompt engineering, not model selection — the small, unglamorous details that decide whether your streaming LLM UX feels alive or laggy. I frame everything around Flutter and Dart because that's where I ship, but the principles carry to any client — React, SwiftUI, a terminal REPL, whatever holds the socket. If it consumes tokens over Server-Sent Events (SSE) or a chunked HTTP response, the same rules apply.
The number your users actually feel is time-to-first-token (TTFT), not total generation time. Once text starts moving, people stop perceiving "waiting" and start perceiving "reading." A response that streams for six seconds feels dramatically faster than a spinner that resolves in four, because the six-second stream gives the brain something to do the entire time. It's the same reason a progress bar that moves beats a faster spinner that doesn't: motion reads as progress, and progress reads as speed.
Practically, this reshapes priorities:
Before you optimize, measure. In practice the first token is delayed by a predictable list of suspects, roughly in the order you should attack them:
await before opening the stream — every synchronous hop on the request path pushes the first token later. Move non-essential work off the hot path.The point isn't to eliminate all of these — it's to know which one you're paying for. Log a timestamp when the request leaves and another when the first delta lands; that single interval is the most important number in the whole feature.
The user's message should render the instant they submit, before any network round-trip confirms anything. Same for the assistant bubble — create an empty one immediately so the layout doesn't jump when tokens arrive. This is the cheapest perceived-speed win available and most people skip it.
In a Flutter chat surface, I model each message as an immutable object with a status, and the streaming assistant message is just a normal message whose content grows:
enum MessageStatus { streaming, complete, error, cancelled }class ChatMessage { final String id; final Role role; final String content; final MessageStatus status; const ChatMessage({ required this.id, required this.role, this.content = '', this.status = MessageStatus.complete, }); ChatMessage append(String delta) => copyWith(content: content + delta);}The key is that the streaming message is not a special widget or an overlay — it's an ordinary list item that happens to be mutating. That keeps your scroll position, your bubble styling, and your copy/share affordances all working with zero special-casing.
There's a subtlety here that bites people once they wire in a backend: generate the message ID on the client, before you send. If you wait for the server to assign an ID, you can't reconcile the streamed content with the optimistic bubble, and you'll either duplicate the message or lose your place. Mint a UUID locally, render optimistically against it, and let the server echo it back. The same client-side ID also makes retries, edits, and error recovery trivial, because the identity of the message never depends on a network response.
Here's the counter-intuitive one. A fast model can emit tokens faster than a phone can usefully repaint, and if you call setState (or notify listeners) on every single delta, you'll thrash the UI thread, drop frames, and — ironically — make a fast stream feel janky. The fix is to decouple the network cadence from the render cadence.
I accumulate deltas into a buffer and flush to the UI on a fixed interval, roughly aligned with the display refresh rather than the token arrival rate:
class StreamBuffer { final void Function(String text) onFlush; final Duration interval; final StringBuffer _pending = StringBuffer(); Timer? _timer; StreamBuffer({required this.onFlush, this.interval = const Duration(milliseconds: 50)}); void add(String delta) { _pending.write(delta); _timer ??= Timer(interval, _flush); } void _flush() { _timer = null; if (_pending.isEmpty) return; onFlush(_pending.toString()); _pending.clear(); } void dispose() { _timer?.cancel(); if (_pending.isNotEmpty) onFlush(_pending.toString()); // never lose the tail }}A ~50ms flush window is invisible to a human reader but slashes the number of layout passes. The one bug to watch for: always flush the residual buffer on completion or dispose, or the last few tokens vanish. I've been bitten by exactly that — the answer looks complete in logs and truncated on screen.
It's tempting to reach for a plain debounce here, but debounce is the wrong tool: if tokens keep arriving faster than the debounce window, the flush never fires and the text freezes until the stream stalls. What you want is a trailing throttle — the first delta arms a timer, and the buffer flushes at a steady cadence for as long as tokens keep coming. That's exactly what the _timer ??= guard above gives you: it arms once, fires on the interval, and re-arms on the next delta after a flush. The result is a smooth, predictable repaint rhythm regardless of whether the model is dribbling one token a second or dumping fifty.
If you want to get fancy, align the interval with the display's refresh rate — flushing once per frame is the theoretical ceiling of useful repaints, since the screen can't show more than that anyway. In Flutter you can drive the flush from a Ticker or SchedulerBinding frame callback instead of a Timer to lock onto the vsync cadence. In practice a plain 50ms timer is close enough that I rarely bother, but it's a real lever if you're chasing frame-perfect smoothness on high-refresh displays.
Every streaming UI needs a stop button, and it needs to do three things the instant it's tapped, in this order:
cancelled — keep the partial text, because partial answers are often still useful.The mistake I see most is a stop button that only flips a UI flag while tokens keep streaming in the background — you've stopped showing the response but you're still paying for it. Wire cancellation all the way down to the request. With Dart streams it's clean because cancelling the StreamSubscription propagates:
StreamSubscription<String>? _sub;void start(Stream<String> tokens) { _sub = tokens.listen( _buffer.add, onDone: () => _finish(MessageStatus.complete), onError: (_) => _finish(MessageStatus.error), );}Future<void> stop() async { await _sub?.cancel(); // tears down the network stream _buffer.dispose(); // flushes the tail _finish(MessageStatus.cancelled);}If you're on a plain HTTP client, make sure the cancel actually closes the response body — some clients keep draining the socket unless you explicitly abort. Otherwise "stop" is a lie.
The same teardown path has to run for cases the user didn't explicitly trigger. If they navigate away from the chat screen mid-stream, your dispose/deactivate lifecycle hook must cancel the subscription — otherwise the stream keeps running against a widget that no longer exists, and you'll either leak the connection or crash on a flush into a disposed state. Route the disposal through the exact same stop() logic; don't write a second, subtly-different cleanup path.
Timeouts deserve the same treatment. A model that goes quiet — no error, no tokens, just silence — is a real failure mode, especially on flaky mobile networks. Wrap the stream in an idle timeout so that if no delta arrives within some window, you finish with MessageStatus.error and surface a retry affordance rather than leaving a blinking cursor forever. Dart's Stream.timeout gives you this almost for free, and it turns a hung UI into a recoverable one.
A pile of small touches, individually trivial, that collectively make the thing feel considered:
**. Your renderer must degrade gracefully rather than flashing broken formatting on every frame. Streaming plain text and only "upgrading" to rich formatting once a block closes is a reasonable compromise.This one is worth its own paragraph because it's where most streaming chat UIs visibly break. The naive approach — run your full Markdown parser on the accumulated buffer every flush — produces flicker: a code fence that's open on one frame closes on the next, so the reader sees raw `` characters, then a styled block, then a heading that briefly renders as #` text. It looks broken even though the final output is correct.
A few strategies that hold up:
The plain-text-then-upgrade compromise is genuinely fine for most apps. Users care far more that the text keeps flowing smoothly than that a code block is syntax-highlighted the instant it opens. Optimize for the flow.
The model's speed is mostly out of your hands; the feeling of speed is entirely in yours. Get the first token on screen as fast as possible, render optimistically so the UI never waits on the network, throttle your repaints so a fast stream stays smooth, and make cancellation cut all the way down to the socket. None of this is hard — it's just a dozen small decisions that most implementations get wrong by default. Nail them and an eight-second answer feels instant. Skip them and even a fast model feels like it's dragging its feet.