Streaming LLM Tokens to the Browser with SSE

Part of Streaming AI Responses in the Browser, under Frontend Consumption & Client Patterns.

The end-to-end path from a model’s token stream to text appearing in a browser is short, and each of its four hops has one way to get it wrong that produces the same visible symptom: output that arrives in lumps instead of flowing.

Symptom & Developer Intent

  • The model streams, the server logs show tokens arriving continuously, and the browser shows nothing for several seconds and then a paragraph at once.
  • The first token takes far longer to appear than the provider’s own latency suggests.
  • Tokens appear smoothly in local development and in lumps in production.
  • The answer renders correctly but the page stutters visibly while it does.
  • The stream sometimes ends without any indication that the answer is complete.

The intent is a token stream that reaches the browser incrementally and renders without janking the page.

Root Cause Analysis

Four hops sit between the model and the screen, and each can hold bytes.

Four hops, four ways to lose the typing effect The path from the model provider through the server, the network and the render loop, with the mistake that batches output at each hop. Four hops, four ways to lose the typing effect Provider stream async iterator collecting into a string first Your server write per delta no flushHeaders / no flush Network path proxy · CDN · gzip buffering or compression Render loop setState per token correct text, stuttering page what goes wrong here
All four produce the same visible symptom, which is why the fix has to be found by elimination rather than by reading the code that looks most suspicious.

The provider stream to your server. Most SDKs expose an async iterator that yields as chunks arrive. Consuming it into a string and then writing turns streaming into batching at the very first hop — surprisingly common in code that was written for a non-streaming API and adapted later.

Your server to the wire. The response must be flushed per event. Node buffers writes until the buffer fills unless headers are flushed and the runtime is allowed to send small chunks; the same explicit-flush requirement described in Buffer Management & Chunked Transfer Encoding applies here with more visible consequences.

The network path. Any proxy, CDN or compression stage that accumulates bytes converts smooth output into lumps — this is the hop that explains “works locally, batches in production”.

The render loop. Once bytes arrive, re-rendering a growing markdown document on every token consumes the frame budget. The text is correct and the page stutters.

The event vocabulary matters as well. A stream that sends only unnamed messages forces the client to infer completion from the connection closing, which is indistinguishable from a failure.

Step-by-Step Resolution

The server as a thin adapter Adapter pipeline: iterate the provider stream, re-emit in your own event vocabulary with a fragment index, and flush per delta. The server as a thin adapter the fragment index is what makes a mid-generation reconnect recoverable Provider chunk async iterator Re-emit event: delta id: gid:n fragment index Flush bytes on the wire never collect your vocabulary enables resume
Re-emitting rather than proxying decouples your clients from a provider wire format and gives you somewhere to put ids, usage and your own errors.

Step 1 — Define the event vocabulary 

event: delta      data: {"text":"Server-"}     ← content fragments
event: usage      data: {"prompt_tokens":412}  ← metadata, once
event: done       data: {"finish_reason":"stop"}
event: error      data: {"message":"…"}

Separating content from lifecycle lets the client route fragments straight to the renderer and handle completion explicitly.

Step 2 — Adapt the provider stream without buffering it 

// server.js — consume the provider's iterator and re-emit immediately.
app.get('/api/generate', async (req, res) => {
  const gid = req.query.gid ?? crypto.randomUUID();
  const controller = new AbortController();
  let index = 0;

  res.writeHead(200, {
    'Content-Type': 'text/event-stream; charset=utf-8',
    'Cache-Control': 'no-cache, no-transform',
    'X-Accel-Buffering': 'no',
  });
  res.flushHeaders();                       // client sees onopen immediately

  req.on('close', () => controller.abort('client_disconnected'));

  try {
    // Iterate — do NOT collect into a string first.
    for await (const chunk of model.stream({ prompt: req.query.prompt, signal: controller.signal })) {
      res.write(
        `id: ${gid}:${index++}\n` +
        `event: delta\n` +
        `data: ${JSON.stringify({ text: chunk.text })}\n\n`,
      );
    }
    res.write(`event: usage\ndata: ${JSON.stringify(await model.usage())}\n\n`);
    res.write('event: done\ndata: {"finish_reason":"stop"}\n\n');
  } catch (err) {
    if (!controller.signal.aborted) {
      res.write(`event: error\ndata: ${JSON.stringify({ message: 'generation_failed' })}\n\n`);
    }
  } finally {
    res.end();
  }
});

The id carries a fragment index, which is what makes resume possible — deltas have no other identity to resume from.

Step 3 — Consume and accumulate on the client 

// client.js
function generate(prompt, onText, onDone) {
  const es = new EventSource(`/api/generate?prompt=${encodeURIComponent(prompt)}`);
  let answer = '';

  es.addEventListener('delta', (e) => {
    answer += JSON.parse(e.data).text;      // concatenate; never interpret a fragment
    onText(answer);
  });

  es.addEventListener('done', () => {
    es.close();                              // explicit close: no reconnect, no re-generation
    onDone(answer, { complete: true });
  });

  es.addEventListener('error', () => {
    if (es.readyState === EventSource.CLOSED) onDone(answer, { complete: false });
  });

  return es;
}

Closing on done is essential: without it the browser reconnects when the server ends the response and starts a second generation.

Step 4 — Coalesce rendering onto the animation frame 

// One render per painted frame, regardless of token rate.
function frameSink(render) {
  let buffer = '';
  let queued = false;
  return (text) => {
    buffer = text;
    if (queued) return;
    queued = true;
    requestAnimationFrame(() => { queued = false; render(buffer); });
  };
}

const paint = frameSink((text) => { output.textContent = text; });
generate(prompt, paint, finalise);

Step 5 — Remove buffering from the network path

The application is now correct; production behaviour depends on the hops in between.

location /api/generate {
    proxy_pass http://app;
    proxy_buffering off;
    gzip off;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_read_timeout 600s;      # long generations must not be cut
}

Validation & Monitoring

The decisive test is arrival timing, not content.

Reading arrival timestamps Timeline comparing evenly spaced token arrivals on a clean path against clustered timestamps that indicate buffering. Reading arrival timestamps time → delta 1 clean path 0.21 s delta 2 evenly spaced 0.24 s deltas 1–48 all one timestamp 3.90 s done buffered path 3.90 s
Timestamping each arriving line is the whole diagnosis: evenly spaced means clean, clustered means something upstream released them together.
# Timestamp every line as it arrives — the gaps ARE the diagnosis.
curl -N -s 'https://api.example.com/api/generate?prompt=count+to+twenty' \
  | while IFS= read -r line; do printf '%s  %s\n' "$(date +%H:%M:%S.%N)" "$line"; done | head -25

Smoothly climbing timestamps mean the path is clean. Identical timestamps for many lines mean something upstream released them together — test each hop in turn as described in Proxy & CDN Configuration for SSE.

# Does the terminal event arrive?
curl -N -s "$URL" | grep -c 'event: done'      # expect 1

Worth tracking in production: time to first delta (perceived responsiveness), deltas per second, and the share of generations that end without done. That last number is the one that catches provider timeouts and platform ceilings, and it is invisible in ordinary error rates because the HTTP response was a perfectly good 200.

// Client beacons that make the experience measurable.
let firstAt = null;
es.addEventListener('delta', () => {
  if (firstAt === null) {
    firstAt = performance.now();
    beacon('ai_time_to_first_delta_ms', Math.round(firstAt - startedAt));
  }
});
es.addEventListener('done', () => beacon('ai_generation_complete', 1));

Production Checklist

Frequently Asked Questions

Why do tokens arrive in lumps in production but smoothly locally?

Because production has hops that development does not: a reverse proxy, a CDN, or a compression stage, each of which accumulates bytes before forwarding. Timestamp each arriving line with curl at every hop; the hop where the timestamps start clustering is the one holding the tokens.

Should I use EventSource or fetch with a stream reader for this?

EventSource when a GET with query parameters or cookie auth is enough — you get parsing, reconnection and resume for free. Switch to fetch with a reader when the prompt is too large for a URL or you need an Authorization header, and accept that you then own reconnection and Last-Event-ID handling yourself.

Why must the client close the connection on the done event?

Because the server ends the response after the generation, and EventSource treats a closed response as a transient failure and reconnects — starting a second generation and billing you for it. An explicit close() on done sets readyState to CLOSED and cancels the retry.

Is one delta per token the right granularity?

It is a reasonable default, and the frame-coalescing sink means the render cost does not scale with it. If your provider emits very small fragments and network overhead becomes measurable, batch fragments on a short timer server-side — but keep batches under about 50 ms or the typing effect starts to look stepped.