Cancelling an In-Flight AI Stream

Part of Streaming AI Responses in the Browser, under Frontend Consumption & Client Patterns.

Closing an EventSource stops the browser reading. Whether it stops the model generating — and the bill accruing — depends entirely on what the server does next, and the default answer is “nothing”.

Symptom & Developer Intent

  • The stop button clears the interface instantly, but provider token usage keeps climbing.
  • Monthly costs exceed the sum of what users actually read.
  • Server logs show generations completing minutes after their clients disappeared.
  • Cancelling and immediately restarting produces two concurrent generations.
  • Navigating away mid-generation never cancels anything.

The intent is a cancel that terminates the generation at the provider, promptly, on every path a user can take.

Root Cause Analysis

Cancellation has to cross three boundaries, and each is a separate failure.

Three boundaries a cancel has to cross Stack of the browser, the server handler and the provider call, with what stops at each boundary when cancellation is incomplete. Three boundaries a cancel has to cross Browser es.close() stops reading only Server handler req close / cancel POST may notice late, or not at all Provider call model.stream(signal) keeps generating without the signal Billing tokens generated stops only when the call does what actually stops
Stopping at any one of these leaves the layer below it running — and the provider is the layer that is billing you.

Browser to server. Calling close() tears down the client side. Over HTTP/2 that sends RST_STREAM; over HTTP/1.1 it closes the socket. Either way the server learns only if it is watching — and it may not learn immediately, because a server that is not writing at that moment does not notice a peer that has gone away.

Server handler to the generation. Even a server that fires its close handler must have something to cancel. Without an abort signal threaded into the model call, the handler’s cleanup ends the response while the provider request continues to completion in the background.

Navigation. When the user navigates away or closes the tab, ordinary fetch calls made during unload are frequently dropped by the browser. A cancel posted without keepalive never leaves.

There is also an ordering trap: cancelling and immediately regenerating can produce two live generations if the cancel is asynchronous and the new request arrives first. Keying by a client-generated generation id, and rejecting work for a cancelled id, resolves it.

Step-by-Step Resolution

Two channels, one abort signal Sequence showing the client closing the stream and posting an explicit cancel, both routed into one AbortController that aborts the provider call. Two channels, one abort signal Client Server Provider es.close() (may be noticed late) POST /cancel keepalive: true controller.abort() signal aborts the model call
The POST is the reliable channel: a socket close can go unnoticed for a long time, while an explicit cancel keyed by generation id cannot be missed.

Step 1 — Give every generation a client-generated id 

// The client owns the id so it can cancel a generation it has not yet heard back about.
function startGeneration(prompt) {
  const gid = crypto.randomUUID();
  const es = new EventSource(
    `/api/generate?gid=${gid}&prompt=${encodeURIComponent(prompt)}`,
  );
  return { gid, es };
}

Step 2 — Cancel on both channels 

function cancelGeneration({ gid, es }) {
  es.close();                                     // stop reading immediately

  // keepalive lets this request survive page unload.
  return fetch(`/api/generate/${gid}/cancel`, { method: 'POST', keepalive: true })
    .catch(() => { /* the connection close is the fallback signal */ });
}

The explicit POST is what makes this reliable: a socket close can go unnoticed for a long time, while a POST is an unambiguous instruction.

Step 3 — Cancel on unload as well as on the button 

// pagehide covers tab close, navigation and bfcache entry — beforeunload does not.
window.addEventListener('pagehide', () => {
  if (currentGeneration) {
    navigator.sendBeacon(`/api/generate/${currentGeneration.gid}/cancel`);
  }
});

sendBeacon is designed for exactly this: the browser guarantees delivery attempts after the page is gone.

Step 4 — Thread the abort signal into the model call

This is the step that actually stops the billing.

// server.js
const cancellations = new EventEmitter();
const active = new Map();                        // gid → AbortController

app.get('/api/generate', async (req, res) => {
  const gid = req.query.gid;
  const controller = new AbortController();
  active.set(gid, controller);

  // Two independent cancellation paths, one signal.
  req.on('close', () => controller.abort('client_disconnected'));
  cancellations.once(gid, () => controller.abort('user_cancelled'));

  writeHeaders(res);

  try {
    // The signal MUST reach the provider call, or aborting only stops writing.
    for await (const chunk of model.stream({ prompt: req.query.prompt, signal: controller.signal })) {
      res.write(`event: delta\ndata: ${JSON.stringify({ text: chunk.text })}\n\n`);
    }
    res.write('event: done\ndata: {"finish_reason":"stop"}\n\n');
  } catch (err) {
    if (!controller.signal.aborted) {
      res.write(`event: error\ndata: ${JSON.stringify({ message: 'generation_failed' })}\n\n`);
    }
  } finally {
    active.delete(gid);
    res.end();
  }
});

app.post('/api/generate/:gid/cancel', (req, res) => {
  const gid = req.params.gid;
  cancellations.emit(gid);                       // aborts an in-flight generation
  cancelled.add(gid, { ttlMs: 60_000 });         // and blocks one that has not started
  res.status(202).end();
});

Step 5 — Reject work for an id that was already cancelled

The TTL set above closes the race where the cancel arrives before the generation request.

app.get('/api/generate', async (req, res) => {
  if (cancelled.has(req.query.gid)) {
    return res.status(409).json({ error: 'generation_cancelled' });
  }
  // …
});

Step 6 — Make the UI honest about what stopped 

// Cancelling is not an error; say so, and keep what was already produced.
async function onStopClick() {
  setStatus('stopping');
  await cancelGeneration(currentGeneration);
  setStatus('stopped');
  keepPartialAnswer();          // the user asked to stop, not to discard
}

Validation & Monitoring

The only convincing test is provider-side usage, because everything else can look correct while generation continues.

Tokens generated after the user pressed stop Bar chart of completion tokens generated after cancellation under three implementations: close only, close plus server cleanup, and a threaded abort signal. Tokens generated after the user pressed stop average per cancelled generation of a long answer close() only 800 close + server cleanup 780 abort signal threaded 12 completion tokens billed after cancel
The middle bar is the trap: the server tidied up correctly and the provider carried on to the end of the answer regardless.
# 1. Start, cancel after two seconds, then check the generation's state.
gid=$(uuidgen)
timeout 2 curl -N -s "https://api.example.com/api/generate?gid=$gid&prompt=write+a+long+essay" > /dev/null
curl -sX POST "https://api.example.com/api/generate/$gid/cancel"
sleep 3
curl -s "https://api.example.com/api/generate/$gid/status"
# Expect {"state":"cancelled"} — not "running", not "completed".

# 2. Compare provider token usage for a cancelled generation against a completed one.
curl -s "https://api.example.com/api/generate/$gid/usage"
# completion_tokens should be a fraction of a full answer.

In production, three numbers tell the story: cancellations as a share of generations, mean completion tokens for cancelled generations, and the interval between cancel and last token. That last one is the leak indicator — if it exceeds a second or two, the abort signal is not reaching the provider.

// Server: record how long generation continued after the cancel arrived.
const cancelledAt = Date.now();
controller.signal.addEventListener('abort', () => {
  metrics.observe('ai_cancel_to_stop_ms', Date.now() - cancelledAt);
});

Production Checklist

Frequently Asked Questions

Is closing the EventSource enough to stop the server?

Only if the server is watching for the disconnect and has something to abort. The close is delivered as a socket close or RST_STREAM, which a server that is not currently writing may notice late or not at all. Pair it with an explicit cancel request and thread an abort signal into the model call.

Why does my cancel request never arrive when the user closes the tab?

Because ordinary fetch calls issued during unload are commonly dropped. Use navigator.sendBeacon, or fetch with keepalive: true, both of which the browser is required to attempt after the page is gone — and trigger them from pagehide rather than beforeunload.

What happens if the cancel arrives before the generation request?

Without protection, the generation starts and runs to completion despite having been cancelled. Remember cancelled ids for a short TTL and reject any generation request for one, which closes the race deterministically rather than relying on ordering.

Should a cancelled answer be discarded from the interface?

No. The user asked to stop generating, not to throw away what they were reading. Keep the partial text, mark it as stopped rather than failed, and offer to continue or regenerate — discarding it is a surprising loss of work.