Streaming AI Responses in the Browser

Part of Frontend Consumption & Client Patterns.

Token-by-token output is the reason most engineers meet Server-Sent Events today. A model generates text incrementally, the server forwards each fragment as it arrives, and the interface fills in as if someone were typing. The transport choice is easy — output flows one way, so SSE fits exactly — but the UI problems are specific and none of them are about the protocol: tokens arrive faster than a browser should re-render, markdown is syntactically invalid halfway through, users cancel mid-generation and expect the server to stop paying for it, and a dropped connection during a two-minute generation must not silently lose the answer. This guide covers the client patterns that solve those problems, and the server-side contract each one depends on.

How It Works

An AI streaming endpoint is an ordinary SSE endpoint with a conventional event vocabulary. The convention that has settled across implementations uses named events to separate content from control, which lets the client route fragments to the renderer while handling lifecycle separately.

The event vocabulary of a generation The four named events of a token stream: repeated delta events carrying fragments, a usage event with token counts, and a terminal done or error event. The event vocabulary of a generation named events across one generation delta ×N text fragments usage token counts done finish_reason error instead of done
Separating content from lifecycle is what lets the client route fragments straight to the renderer while still knowing, definitively, when the answer is complete.
event: delta
data: {"text":"Server-"}

event: delta
data: {"text":"Sent "}

event: delta
data: {"text":"Events"}

event: usage
data: {"prompt_tokens":412,"completion_tokens":58}

event: done
data: {"finish_reason":"stop"}

Four properties of this shape matter for the client.

Fragments are not words. A delta may be a partial word, a single character, or a chunk of several words, and it may split a multi-byte character or an emoji across two events if the producer is careless. The client must concatenate rather than interpret, and the server should never split within a UTF-8 sequence.

Order is everything and there is no key. Unlike an entity feed, deltas have no identity to deduplicate by — fragment three is only meaningful after fragments one and two. This makes resume genuinely harder than for a normal event stream, and it is why the id field should carry a fragment index.

The terminal event is explicit. A generation that ends normally sends done. A stream that simply closes without it means something broke, and the difference decides whether the UI shows a finished answer or an error.

Cancellation is a client-initiated event with a server-side cost. Closing the EventSource stops the browser reading, but unless the server notices the disconnect it keeps generating — and keeps being billed for tokens nobody will see.

// Minimal correct consumer: accumulate, render, finish.
const es = new EventSource(`/api/generate?prompt=${encodeURIComponent(prompt)}`);
let answer = '';

es.addEventListener('delta', (e) => {
  answer += JSON.parse(e.data).text;    // concatenate; never parse a fragment alone
  render(answer);
});

es.addEventListener('done', () => {
  es.close();                            // no retry — the generation is complete
  finalise(answer);
});

es.addEventListener('error', () => {
  if (es.readyState === EventSource.CLOSED) showFailed();
});

Client-Side Consumption

Rendering without janking the page

A fast model emits tokens far more often than a screen refreshes. Re-rendering per delta at 80 tokens per second means 80 renders per second of an element whose content grows monotonically — and in a framework that reconciles a tree, that is where the frame budget goes. Coalesce on the animation frame instead: accumulate into a plain variable and render at most once per frame.

// Coalesce deltas onto the animation frame: at most one render per painted frame.
function createTokenSink(render) {
  let buffer = '';
  let pending = false;

  const flush = () => {
    pending = false;
    render(buffer);
  };

  return {
    push(text) {
      buffer += text;
      if (!pending) {
        pending = true;
        requestAnimationFrame(flush);
      }
    },
    value: () => buffer,
    reset() { buffer = ''; },
  };
}

const sink = createTokenSink((text) => setAnswer(text));
es.addEventListener('delta', (e) => sink.push(JSON.parse(e.data).text));

This caps renders at the display refresh rate regardless of token rate, and it costs nothing when tokens are slow — the first delta of an idle period still renders on the next frame. The same technique applied to a store is described in State-Management Integration for SSE.

Rendering partial markdown safely

Model output is usually markdown, and markdown mid-generation is frequently invalid: an unclosed code fence, a half-written link, a table with one row. Feeding that to a renderer produces flickering — a paragraph becomes a code block becomes a paragraph again — and, if the renderer emits HTML, an injection surface.

Two rules make it stable. Close unterminated constructs before rendering, in a copy of the text rather than in the buffer itself; and sanitise the output rather than trusting the model.

// Render a *repaired copy* so the buffer stays exactly what the model sent.
function repairForRender(md) {
  let out = md;

  // 1. Unclosed fenced code block: close it so the rest is not swallowed.
  const fences = (out.match(/^```/gm) || []).length;
  if (fences % 2 === 1) out += '\n```';

  // 2. Trailing partial link or image: hide it until the closing paren arrives.
  out = out.replace(/!?\[[^\]]*\]\([^)]*$/, '');

  // 3. Unbalanced inline code / emphasis at the very end.
  if ((out.match(/`/g) || []).length % 2 === 1) out = out.replace(/`[^`]*$/, '');
  if ((out.match(/\*\*/g) || []).length % 2 === 1) out = out.replace(/\*\*[^*]*$/, '');

  return out;
}

function renderMarkdown(raw) {
  const html = markdownToHtml(repairForRender(raw));
  container.innerHTML = sanitize(html);     // always sanitise model output
}

Repairing a copy matters: if you mutate the buffer, the closing fence you added becomes part of the text the next delta appends to, and the final answer is corrupt.

Cancellation that actually stops the work

Closing an EventSource is a purely local act — it stops the browser reading and, over HTTP/2, sends RST_STREAM. Whether the server notices depends on whether its handler watches for disconnect. Rely on that alone and generations continue invisibly; the token bill is the first place anyone notices.

The robust pattern pairs a local close with an explicit cancel call, because a POST cannot be lost the way a connection reset can be missed.

// Cancel: close locally AND tell the server, keyed by generation id.
function startGeneration(prompt, sink) {
  const generationId = crypto.randomUUID();
  const es = new EventSource(
    `/api/generate?gid=${generationId}&prompt=${encodeURIComponent(prompt)}`,
  );

  es.addEventListener('delta', (e) => sink.push(JSON.parse(e.data).text));
  es.addEventListener('done', () => es.close());

  return {
    cancel() {
      es.close();                                   // stop reading immediately
      // keepalive lets the request survive a page unload.
      fetch(`/api/generate/${generationId}/cancel`, { method: 'POST', keepalive: true });
    },
  };
}

On the server, the cancel endpoint sets a flag the generation loop checks between tokens; the disconnect handler does the same. Either path stops the work, and the explicit one survives cases where the disconnect is not observed promptly.

Resuming a dropped generation

Because deltas are position-dependent, resume needs a fragment index rather than an entity id. Put the index in the id field: EventSource then returns it as Last-Event-ID automatically on reconnect, and the server can skip everything the client already has.

// Client: nothing to do — the browser sends Last-Event-ID on automatic reconnects.
es.addEventListener('delta', (e) => {
  // e.lastEventId is the fragment index, e.g. "gen_8f2:37"
  sink.push(JSON.parse(e.data).text);
});
// Server: resume from the fragment index the client reports.
app.get('/api/generate', async (req, res) => {
  const gid = req.query.gid;
  const cursor = req.headers['last-event-id'];               // "gen_8f2:37" or undefined
  const from = cursor ? Number(cursor.split(':')[1]) + 1 : 0;

  const run = await generations.get(gid);                     // buffered fragments so far
  writeHeaders(res);

  // Replay what the client missed while disconnected…
  for (let i = from; i < run.fragments.length; i++) {
    res.write(`id: ${gid}:${i}\nevent: delta\ndata: ${JSON.stringify({ text: run.fragments[i] })}\n\n`);
  }
  // …then attach to the live generation.
  run.onFragment((text, i) => {
    res.write(`id: ${gid}:${i}\nevent: delta\ndata: ${JSON.stringify({ text })}\n\n`);
  });
});

This requires the server to buffer the fragments of an in-flight generation, which is a small cost for a large usability gain: a user whose wifi blips mid-answer sees the answer complete rather than restart. The general mechanism is the same one described in Event ID & Retry Mechanism Design.

Server-Side Implementation

The server side is a thin adapter: consume the model provider’s stream, re-emit it in your own event vocabulary, and make disconnects stop the work. Re-emitting rather than proxying matters — it decouples your client from a provider’s wire format and lets you add your own ids, usage events and errors.

// Node: adapt a provider stream into our own SSE vocabulary, with cancellation.
app.get('/api/generate', async (req, res) => {
  const gid = req.query.gid ?? crypto.randomUUID();
  const controller = new AbortController();
  let index = 0;

  res.writeHead(200, {
    'Content-Type': 'text/event-stream; charset=utf-8',
    'Cache-Control': 'no-cache, no-transform',
    'X-Accel-Buffering': 'no',
  });
  res.flushHeaders();

  // Two independent cancellation paths — either one stops the model.
  req.on('close', () => controller.abort('client_disconnected'));
  cancellations.on(gid, () => controller.abort('user_cancelled'));

  try {
    for await (const chunk of model.stream({ prompt: req.query.prompt, signal: controller.signal })) {
      const frame =
        `id: ${gid}:${index++}\n` +
        `event: delta\n` +
        `data: ${JSON.stringify({ text: chunk.text })}\n\n`;
      res.write(frame);
      await generations.append(gid, chunk.text);      // buffer for resume
    }
    res.write(`event: usage\ndata: ${JSON.stringify(await model.usage())}\n\n`);
    res.write('event: done\ndata: {"finish_reason":"stop"}\n\n');
  } catch (err) {
    if (controller.signal.aborted) return;            // cancellation is not an error
    res.write(`event: error\ndata: ${JSON.stringify({ message: 'generation_failed' })}\n\n`);
  } finally {
    res.end();
  }
});

Note that the abort signal is threaded into the model call itself. Without it, aborting only stops writing to a closed socket — the generation continues to completion on the provider’s side and is billed in full.

Edge Cases & Network Interference

  • Buffering proxies destroy the effect entirely. A stream that arrives in 4 KB batches renders as three paragraph-sized jumps instead of typing. This is the most visible possible symptom of the buffering problem covered in Proxy & CDN Configuration for SSE — users describe it as “the AI is slow” rather than as a bug.
  • Multi-byte characters split across deltas. A provider that chunks by bytes can split an emoji or an accented character across two events, producing a replacement character in the middle of the answer. Concatenate raw strings and decode once; never JSON.parse a fragment as if it were standalone text.
  • done never arrives. Model timeouts, provider errors and platform ceilings all end the stream without the terminal event. Treat a close without done as an incomplete generation and offer a retry, rather than silently presenting a truncated answer as final.
  • The user navigates away mid-generation. EventSource is torn down by the navigation, but the cancel request may not be sent in time — use fetch(..., { keepalive: true }) so the browser is permitted to complete it during unload.
  • Very long generations hit platform ceilings. On serverless runtimes a multi-minute generation can outlive the invocation; see Edge & Serverless SSE Deployment for the planned-handover pattern.
  • Auto-scroll fighting the user. Following the output to the bottom is right until the user scrolls up to read. Track whether the container is pinned to the bottom and stop auto-scrolling the moment it is not, resuming when the user returns.
  • Markdown renderers that mutate state. Some renderers cache parse trees across calls; feeding them a growing string can produce stale output. Render from the repaired copy each time and verify the renderer is pure.
What buffering does to a token stream Timeline contrasting smooth token arrival with a buffered path that releases four seconds of tokens in a single burst. What buffering does to a token stream time → First token unbuffered path 0.2 s Steady flow one delta at a time 0.2–4 s Nothing buffered path 0–4 s All at once 4 KB released 4.1 s
Users describe this as "the AI is slow". The generation was smooth; the delivery was not, and no model change will fix it.

Performance & Scale Considerations

The client cost is dominated by re-rendering, not by parsing. Parsing a delta is a JSON.parse of a tiny object — tens of microseconds — while re-rendering a growing markdown document is milliseconds and grows with answer length. That asymmetry is why frame coalescing matters and why re-parsing the entire markdown buffer on every frame becomes the bottleneck on long answers.

Per-token cost on the client Bar chart of the per-token cost of parsing a delta, appending to a buffer, re-parsing the markdown document and reconciling a framework tree. Per-token cost on the client per token, on a 4 KB answer string concat 1× JSON.parse(delta) 10× framework re-render 2 000× markdown re-parse 2 000× relative cost per token
Parsing is free; rendering is not. That asymmetry is the entire argument for coalescing on the animation frame and splitting settled blocks from the live tail.
// Cost per token at 80 tokens/s, order of magnitude:
//   JSON.parse(delta)                 ~0.01 ms   ← negligible
//   string concatenation              ~0.001 ms  ← negligible
//   markdown re-parse of 4 KB answer  ~2 ms      ← 160 ms/s if done per token
//   framework re-render of the tree   ~1–5 ms    ← the reason to coalesce

Two mitigations extend the ceiling. Coalescing on the frame caps re-parses at 60 per second regardless of token rate. Splitting the answer into blocks — rendering completed paragraphs once and only re-parsing the trailing block — turns an O(n) cost per frame into an O(1) one, which is what keeps very long answers smooth.

Server-side, the dominant cost is not connections but the model itself, which reframes the usual capacity question. Ten thousand concurrent generations is ten thousand streams — trivial for the connection layer described in Connection Pooling for SSE Servers — and an enormous provider bill. That is why cancellation is a cost control, not just a nicety.

Validation & Debugging 

# 1. Do fragments actually arrive incrementally, or in one burst?
curl -N -s 'https://api.example.com/api/generate?prompt=hello' \
  | while read -r line; do printf '%s %s\n' "$(date +%s.%N)" "$line"; done | head -20
# Timestamps should climb smoothly; identical timestamps mean something buffered.

# 2. Does the terminal event arrive?
curl -N -s 'https://api.example.com/api/generate?prompt=hello' | grep -c 'event: done'

# 3. Does cancellation actually stop generation? Watch provider token usage
#    while disconnecting early.
timeout 2 curl -N -s 'https://api.example.com/api/generate?gid=test-1&prompt=write+an+essay' > /dev/null
curl -s https://api.example.com/api/generate/test-1/status
# Expect state=cancelled, not state=running.

In the browser, the Network panel’s EventStream view shows deltas with arrival times — the fastest way to distinguish “the model is slow” from “something is buffering”. For the render side, a performance profile during generation shows immediately whether time is going to markdown parsing or to framework reconciliation, which decides whether block-splitting or coalescing is the fix.

Worth tracking in production: time to first delta (the metric users perceive as responsiveness), deltas per second, generations ending without done, and cancellations as a share of generations. A rising share of streams with no done is usually a platform ceiling rather than a model problem.

Production Checklist

Frequently Asked Questions

Why does the text arrive in chunks instead of smoothly, even though the model streams token by token?

Almost always an intermediary buffering the response — a proxy, a CDN, or a compression stage — collecting several kilobytes before forwarding. The generation is smooth and the delivery is not. Disable buffering and compression on the stream path and confirm with a timestamped curl: if the arrival timestamps cluster, the bytes were held somewhere between your server and the browser.

How do I stop the model from running when the user cancels?

Two paths, both needed. Close the EventSource so the browser stops reading, and post an explicit cancel keyed by generation id with keepalive: true so it survives a page unload. On the server, wire both the request close handler and the cancel endpoint to the same abort signal, and pass that signal into the model call — otherwise you stop writing to a dead socket while the provider keeps generating and billing.

What is the right way to render markdown that is still being written?

Render a repaired copy, never the buffer itself. Close an odd number of code fences, strip a trailing partial link, and balance unclosed inline markers — then parse and sanitise that copy. Mutating the real buffer corrupts the final answer, because the next delta appends after the characters you added.

Can a user resume a generation after their connection drops?

Yes, if the server buffers the fragments of an in-flight generation and the id field carries a fragment index. EventSource sends the last id back automatically on reconnect, so the handler replays fragments after that index and then attaches to the live generation. Without an index, resume is impossible — deltas have no identity to deduplicate by.

Should each delta be a separate named event or plain messages?

Use named events. Separating delta from usage, error and done lets the client route content to the renderer while handling lifecycle separately, and it means adding a new control event later does not disturb existing clients. A single untyped message channel forces every consumer to branch on the payload shape instead.

Deep Dives

Cancelling an In-Flight AI Stream Make a stop button actually stop the work: close the EventSource, post an explicit cancel that survives unload, thread the abort signal into the model call, and prove tokens stopped. Rendering Partial Markdown from a Token Stream Render markdown that is still being written without flicker or injection: repair unterminated constructs in a copy, split completed blocks from the live tail, and sanitise every frame. Streaming LLM Tokens to the Browser with SSE Wire a model's token stream to an EventSource client end to end: the event vocabulary, the server adapter, incremental rendering and the failure modes that make output arrive in chunks.