Streaming UX Patterns for LLM Apps

By Michael Samuel Naeem · March 31, 2026 · 6 min read

An LLM that takes eight seconds to answer feels broken if the screen sits blank, and perfectly fine if words start flowing at second one. That gap is entirely a UX problem, and streaming UX patterns are how you solve it: rendering the model's output incrementally as it's generated, surfacing intermediate states like tool calls, and handling interruption and failure without the interface freezing. The model's latency is mostly fixed; how it feels is yours to design.

I've built enough chat and assistant UIs to believe the streaming layer is where a lot of "this AI product feels premium" actually comes from. It's not the model — it's whether the interface respects the user's attention while the model thinks. Here's what goes into doing it well.

Time-to-first-token is the metric that matters

Users don't experience your average latency; they experience the wait before anything happens. A response that takes 9 seconds total but shows its first token at 400ms feels dramatically faster than one that appears fully formed at 6 seconds. So optimize for time-to-first-token (TTFT) relentlessly, and stream everything after it.

This reframes performance work. Prompt bloat, a slow retrieval step, or a cold model all push TTFT out and cost you the perception of speed even if total generation is quick. Measure TTFT as a first-class metric alongside total latency, and treat a regression in it as seriously as an error rate spike.

SSE is the right default transport

For streaming tokens from server to browser, server-sent events (SSE) is the pragmatic default. It rides ordinary HTTP, gives you a simple event stream, and reconnects automatically. The server flushes chunks as the model produces them; the client appends them to the DOM.

const res = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({ messages }),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  for (const line of decoder.decode(value).split("\n")) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6);
    if (payload === "[DONE]") return;
    const { delta } = JSON.parse(payload);
    appendToBubble(delta);              // paint immediately
  }
}

Reach for WebSockets only when you genuinely need full duplex — live voice, collaborative editing, a session where the client streams audio up while the model streams tokens down. That bidirectional case is exactly the shape of voice agents with STT and TTS pipelines, where SSE alone isn't enough. For text chat, SSE keeps the stack simpler and the reconnection story free.

Show the machine thinking

Modern LLM apps don't just emit text — they call tools, retrieve documents, and sometimes reason in visible steps. Streaming only the final prose wastes the chance to make the wait informative. The states worth surfacing:

• Thinking / retrieving. A lightweight indicator while the model plans or fetches context, ideally naming what it's doing ("Searching your documents…") rather than a generic spinner.
• Tool calls in flight. When the model calls a tool, show it: "Looking up order #4821." This both fills dead time and builds trust by making the system legible.
• Streaming tokens. The main event — text appearing as generated.
• Done, with affordances. Copy, regenerate, and feedback controls appearing cleanly at completion.

The trap I keep seeing is treating everything before the final text as a black-box spinner. Users tolerate waiting far better when they can see what they're waiting for. If your app streams structured output — JSON, a form, function arguments — you also have to decide how to render partial structure, which ties into how you design structured outputs and function calling; a half-streamed JSON object should degrade to a sensible loading state, not flash malformed content.

Handle cancellation, errors, and reconnection as first-class

Happy-path streaming is easy. The reasons streaming UIs feel flaky are all on the unhappy paths.

Cancellation. Let users stop a runaway generation. Wire a stop button to an AbortController and actually cancel the upstream model request server-side — not just hide the tokens — so you stop paying for output nobody wants.

Mid-stream errors. The model can fail after emitting half a response — a timeout, a content filter, a dropped upstream connection. Decide the behavior deliberately: keep the partial text and append a clear "response interrupted, retry?" affordance, rather than blanking what the user was already reading. Never let a mid-stream error erase visible content.

Reconnection. Networks drop, especially on mobile. SSE reconnects, but you need server-side handling to resume or cleanly restart rather than replaying a duplicate stream. If you can, make the generation idempotent by request id so a reconnect doesn't double-charge or double-render.

Small details that separate good from janky

The polish lives in a handful of interaction choices:

The auto-scroll one bites everybody: naive "always scroll to bottom" fights the user the moment they try to read something above, which is infuriating. Detect upward scroll intent and yield control, offering a "jump to latest" button instead.

Streaming Markdown safely is the other classic footgun. A code block streams in as an unterminated ``` for a second or two; a naive renderer will flash a broken, unclosed block. Buffer until fences balance, or use a parser built to tolerate partial input.

Streaming is a UX contract, not a feature flag

The mistake is treating streaming as "we turned on stream: true." Done well, it's a contract with the user: something is always happening, you can see what, you can stop it, and nothing you've read will vanish. That contract is what makes a slow model feel like a fast product.

My advice: instrument TTFT, make intermediate states legible, and spend most of your engineering effort on cancellation, errors, and reconnection — because that's where perceived reliability is won or lost. The token-by-token typing effect is the easy 20%. The other 80% is what users feel without being able to name it.

Resources

• MDN — Using server-sent events
• WHATWG — Streams standard
• Vercel AI SDK — streaming documentation
• OpenAI — streaming API responses
• MDN — AbortController