Request/response gives you one moment to render. A stream gives you eight hundred and any of them can fail. Most of the AI interfaces we've reviewed get the first second and the last second wrong.
Every UI pattern we have was built for request/response. You fire a request, you show a spinner, the response arrives, you render it. One moment of uncertainty, one moment of resolution and the shape of the result is known before anything is painted.
Token streaming breaks all three assumptions. There is no single moment of resolution - there are hundreds. The shape of the result is unknown while you're already rendering it. And the failure can arrive at token four hundred, long after you've committed pixels to the screen.
We've built or reviewed a dozen streaming interfaces over the past two years. The mistakes cluster at the two ends: the first second, before anything has arrived and the last second, when something goes wrong after most of it has. The middle - text appearing progressively - is the part everyone gets right and the part that matters least.
Time to first token is dead air. Depending on the model, the prompt size and whether a retrieval step runs first, it's anywhere from 300ms to six seconds. A spinner is the wrong answer for the long end of that range, because a spinner communicates "this is taking a moment" and says nothing about what's coming.
Skeleton loaders are the instinctive fix and they're worse. A skeleton is a promise about shape - three lines here, a card there - and with a model response you don't know the shape. When the skeleton shows four lines and the answer is one word, the correction is more jarring than the wait was.
What worked for us was communicating stage rather than progress. If a retrieval step is running, say so. If a tool call is out, name it. The user's tolerance for four seconds of "searching the policy documents" is enormous compared to their tolerance for four seconds of an unlabelled spinner and the honest version costs nothing to build because that state already exists in your orchestration layer. You're just not surfacing it.
Text arriving progressively means a container growing continuously and a growing container means everything below it moves. If there's anything interactive down there - a button, a link, a follow-up input - you have built a UI where the click target moves under the cursor. We shipped that once and watched a user miss the same button three times.
Three rules came out of it. Anything below a streaming region gets pinned or gets pushed off-screen deliberately, never left floating in the path of the growth. Scroll follows the stream only until the user scrolls up - the moment they do, autoscroll is off until they return to the bottom, no exceptions and no smart re-engagement heuristics. And the container reserves a minimum height on first token so the initial jolt from zero happens once rather than as a sequence of small shifts.
Partial markdown is its own version of this. A stream will hand you an unclosed code fence, half a table, a link with no closing bracket. Parse that naively every frame and the layout thrashes between interpretations - three lines become a code block become three lines again. We buffer inside an unterminated construct and render it as plain text until it closes. Slightly late is invisible; flickering between two layouts is not.
One framework-specific note, since we build in SolidJS: the naive setText(t => t + chunk) re-renders the whole block on every token. Keeping the stream as a list of chunks and rendering the list lets fine-grained reactivity append rather than replace. On a long response the difference is the whole frame budget.
Users stop things. They read the first two sentences, see it's going the wrong way and want out.
Two things go wrong here. The first is that the abort never reaches the model - the client stops listening but the server keeps generating and you keep paying for tokens nobody will read. Propagating cancellation all the way through the orchestration layer is not free and it's routinely skipped.
The second is a product decision disguised as a technical one: what happens to the text already on screen? Wiping it treats it as garbage, but the user read it - that's why they cancelled. We keep it, mark it visibly incomplete and let it stay in the transcript. Anything else makes users hesitate before cancelling, which is the opposite of what a cancel button is for.
The failure mode nobody designs for: four hundred tokens in, a tool call fails or the connection drops. The standard error pattern - clear the view, show an error state - is now actively hostile, because it destroys content the user has been reading for six seconds.
Mid-stream errors have to be additive. The partial response stays, an inline marker shows where it stopped and the retry appends rather than replaces. This also means your error handling can't live in a wrapper component that owns the whole region - it has to live inside the stream's own state, which is usually a refactor rather than an addition and it's why teams skip it.
Every streaming surface we build now runs the same state machine. Not a boolean isLoading, which is where all of this starts going wrong.
idle ──► connecting ──► waiting ──► streaming ──► settling ──► done
│ │ │ │
▼ ▼ ▼ ▼
error error error* error* / cancelled*
(* partial content retained)
The states that earn their place are waiting (connected, no tokens yet - this is where stage labels live) and settling (stream closed, final parse and layout lock done before interactive elements come back). Collapsing either into a generic loading flag is what produces the flicker at the end that most AI interfaces still have.
The progressive-text part of streaming UI is easy and largely solved by the framework. The hard parts are the edges: honest stage communication in the dead air before the first token, layout that doesn't move under the user's cursor, cancellation that actually cancels and keeps what was read and errors that add to the response instead of destroying it. Model behind it is irrelevant to all four. These are interface problems and they're the difference between an AI feature that feels engineered and one that feels like a demo.