Idempotent Event ID Generation #Permalink to this section

Part of Backend Stream Generation & Connection Management.

SSE connections break. Load balancers time out, mobile radios drop, proxies restart. The browser’s EventSource reconnects automatically and sends a Last-Event-ID header with the last ID the client received. If your server cannot match that ID to a position in the event stream, the client either replays events it already processed or silently misses events that happened during the gap—both are correctness bugs in production.

Idempotent event ID generation is the combination of three guarantees: uniqueness (no two events share an ID), monotonicity (receivers can detect out-of-order delivery), and recoverability (the server can seek to any ID still in its replay window). Getting all three right across distributed deployments, CDNs, and proxy layers is the subject of this guide.

How the id Field Works #Permalink to this section

The WHATWG HTML specification defines the id: field as a UTF-8 string emitted before or after data: in each event block. The browser’s EventSource implementation stores this value internally; when the connection closes and the browser reconnects, it sends Last-Event-ID: <value> as an HTTP request header (not a query parameter—a common misconception).

Wire format (confirmed from the spec):

id: 7812943001
event: price-update
data: {"ticker":"AAPL","price":189.42}

The empty line after data: dispatches the event. An id: with an empty value (id:\n) resets the stored ID to the empty string; the browser will not send Last-Event-ID on the next reconnect if the last seen ID was empty. This is a valid way to mark segments of a stream as non-resumable.

ID Field Parsing Rules #Permalink to this section

The server reads Last-Event-ID from the reconnect request and uses it to seek the event log. It must not use the id: field value itself for internal ordering without also persisting events; the field is a hint to the client, not a contract with the server.

Choosing an ID Strategy #Permalink to this section

Four strategies dominate production SSE deployments. The right choice depends on whether you run a single process or a distributed fleet.

Database sequences are the safest starting point for single-region deployments: SELECT nextval('sse_event_id_seq') in PostgreSQL gives a guaranteed-monotonic 64-bit integer with no clock dependency. Drawback: every event generation takes a round-trip to the DB.

ULIDs (Universally Unique Lexicographically Sortable Identifiers) are 128-bit values encoded as 26-character Crockford Base32 strings: 48 bits of millisecond timestamp followed by 80 bits of randomness. They sort correctly as strings, require no coordination, and fit cleanly as SSE id: values. Libraries exist for every major language.

Snowflake-style IDs (Twitter Snowflake, Sonyflake, etc.) pack a timestamp, a datacenter/node ID, and a per-node sequence counter into a 64-bit integer. They require each server instance to be assigned a unique node ID at startup—typically via environment variable or a distributed lock on startup—but then generate IDs entirely in process memory at microsecond throughput.

Server-Side Implementation #Permalink to this section

Node.js — ULID-based Event Stream #Permalink to this section

import { monotonicFactory } from 'ulid';
import { createClient } from 'redis';

const ulid = monotonicFactory();          // monotonic variant: same-ms calls increment the low bits
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const STREAM_KEY = 'sse:events';
const REPLAY_WINDOW = 500;                // keep last 500 events per channel

/**
 * Write an event to the Redis Stream and return its ULID.
 * The Redis entry key is the ULID; XADD trims to REPLAY_WINDOW automatically.
 */
export async function publishEvent(channel, type, payload) {
  const id = ulid();                      // e.g. "01HZ6M3P2E0000000000000000"
  await redis.xAdd(
    `${STREAM_KEY}:${channel}`,
    '*',                                  // Redis auto-assigns stream entry ID
    { eid: id, type, data: JSON.stringify(payload) },
    { TRIM: { strategy: 'MAXLEN', threshold: REPLAY_WINDOW, strategyModifier: '~' } }
  );
  return id;
}

/**
 * SSE endpoint — Express handler.
 * Reads Last-Event-ID, replays missed events, then tails live events.
 */
export async function sseHandler(req, res) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('X-Accel-Buffering', 'no');   // disable nginx buffering
  res.setHeader('Connection', 'keep-alive');
  res.flushHeaders();

  const channel = req.params.channel;
  const streamKey = `${STREAM_KEY}:${channel}`;
  const lastId = req.headers['last-event-id'] || '0-0'; // Redis stream entry start

  // --- Replay missed events ---
  // Map the ULID last-event-id back to a Redis stream range query.
  // We store the ULID in the payload field 'eid', so we scan by timestamp component.
  const missed = await redis.xRange(streamKey, lastId, '+');
  for (const { message } of missed) {
    if (message.eid === lastId) continue;    // skip the event the client already has
    res.write(`id: ${message.eid}\nevent: ${message.type}\ndata: ${message.data}\n\n`);
  }

  // --- Tail new events ---
  let cursor = '$';                           // only new entries after this point
  let active = true;

  req.on('close', () => { active = false; });

  while (active) {
    const results = await redis.xRead(
      [{ key: streamKey, id: cursor }],
      { COUNT: 50, BLOCK: 5000 }             // block up to 5 s, batch up to 50
    );
    if (!results) continue;                  // timeout — loop again to check `active`
    for (const { messages } of results) {
      for (const { id: redisId, message } of messages) {
        if (!active) break;
        res.write(`id: ${message.eid}\nevent: ${message.type}\ndata: ${message.data}\n\n`);
        cursor = redisId;
      }
    }
  }
}

The monotonicFactory from the ulid package guarantees that two calls within the same millisecond return strictly increasing values by incrementing the random suffix. This avoids the clock-rollback hazard that affects naive Date.now() implementations.

Python — Snowflake ID with FastAPI #Permalink to this section

The following uses a Snowflake-style generator compatible with Python FastAPI SSE deployments.

import asyncio
import time
import os
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

# --- Snowflake ID generator ---
# Bit layout: 41 bits timestamp (ms) | 10 bits node | 12 bits sequence
EPOCH = 1_700_000_000_000              # custom epoch in ms (reduces ID size)
NODE_ID = int(os.environ.get("NODE_ID", "1")) & 0x3FF   # 10 bits, 0–1023
_sequence = 0
_last_ms = -1

def snowflake_id() -> int:
    global _sequence, _last_ms
    ms = int(time.time() * 1000) - EPOCH
    if ms == _last_ms:
        _sequence = (_sequence + 1) & 0xFFF   # 12-bit counter; wraps at 4096/ms
        if _sequence == 0:
            # Sequence exhausted in this ms — busy-wait for next ms
            while ms <= _last_ms:
                ms = int(time.time() * 1000) - EPOCH
    else:
        _sequence = 0
    _last_ms = ms
    return (ms << 22) | (NODE_ID << 12) | _sequence

# --- SSE endpoint ---
app = FastAPI()

async def event_generator(request: Request, last_id: int):
    """Yield SSE-formatted lines. last_id = 0 means fresh connection."""
    async for event in fetch_events_since(last_id):     # your data layer
        if await request.is_disconnected():
            break
        eid = snowflake_id()
        yield f"id: {eid}\nevent: {event['type']}\ndata: {event['payload']}\n\n"

@app.get("/events")
async def sse_endpoint(request: Request):
    raw_last_id = request.headers.get("last-event-id", "0")
    try:
        last_id = int(raw_last_id)
    except ValueError:
        last_id = 0                        # malformed header → start fresh

    return StreamingResponse(
        event_generator(request, last_id),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",
        },
    )

Note the per-process _sequence counter and _last_ms guard: a single Python process is single-threaded (GIL), so this is safe without a lock. For multi-process deployments behind a load balancer, NODE_ID must differ per instance—otherwise two workers on the same millisecond with the same node ID will produce identical IDs.

Client-Side Deduplication #Permalink to this section

Even with a well-designed server, edge conditions (load balancer failover, mid-replay restart) can deliver the same id: value twice. A client-side dedup cache provides a last line of defence.

// useSSE.js — production-grade EventSource hook with dedup
export function useSSE(url) {
  const processedIds = new Set();
  const MAX_DEDUP_CACHE = 500;          // cap memory: ~500 string entries ≈ 30 KB

  const source = new EventSource(url, { withCredentials: true });

  source.addEventListener('message', (e) => {
    const id = e.lastEventId;

    // Idempotent guard
    if (id && processedIds.has(id)) {
      console.debug('[SSE] duplicate skipped', id);
      return;
    }
    if (id) {
      processedIds.add(id);
      // Evict oldest entries to bound memory
      if (processedIds.size > MAX_DEDUP_CACHE) {
        processedIds.delete(processedIds.values().next().value);
      }
    }

    handleEvent(JSON.parse(e.data));
  });

  // Handle server-side sync-required event (gap too large)
  source.addEventListener('sync-required', (e) => {
    processedIds.clear();
    resetLocalState(JSON.parse(e.data));   // full snapshot
  });

  return source;
}

The lastEventId property on the MessageEvent reflects the id: field from the most recent event block that contained an id: line—not necessarily the current event’s ID if the server omitted the field. Always emit id: on every event block, not just periodically, to keep this value accurate.

For a complete React hook wrapping this pattern, see Building a useEventSource React Hook.

Edge Cases and Network Interference #Permalink to this section

Proxy and CDN Stripping #Permalink to this section

Several reverse proxies and CDN edge nodes strip or buffer SSE id: fields. Confirmed failure modes:

The X-Accel-Buffering: no header disables nginx proxy buffering for a specific response. Set it on every SSE response; do not rely on nginx configuration alone since the response header takes precedence.

For a deeper treatment of buffer settings that interact with ID delivery, see Buffer Management & Chunked Transfer Encoding.

Clock Skew in Distributed Deployments #Permalink to this section

Wall-clock timestamps are unsafe as the sole ID component across multiple server nodes. NTP corrections can move the clock backwards by tens of milliseconds; two nodes with identical node IDs generating IDs at the same corrected millisecond will collide.

Mitigations in order of strength:

1. Monotonic factory variant (ULID monotonicFactory, Snowflake sequence counter): detects same-ms condition and increments within-ms.
2. Hybrid Logical Clocks (HLC): combine physical time with a logical counter that advances on every send/receive event. Suitable for multi-region deployments where causal ordering matters.
3. Centralised sequence service: a single Redis INCR or a Postgres sequence is the simplest guarantee. Latency cost is one round-trip per event; acceptable at < 10 k events/s.

Last-Event-ID Gap Handling #Permalink to this section

If the requested ID has aged out of the replay window (TTL expired, MAXLEN trim), do not return 4xx. Return 200 OK and emit a sync-required event first:

// Express gap-handler middleware
function handleReplayGap(req, res, streamKey, redis) {
  return async (lastId) => {
    const oldest = await redis.xRange(streamKey, '-', '+', { COUNT: 1 });
    if (!oldest.length) return;                        // stream is empty

    const oldestId = oldest[0].message.eid;
    if (lastId < oldestId) {                           // ULID string comparison works lexicographically
      const snapshot = await buildStateSnapshot();
      const freshId = ulid();
      res.write(
        `id: ${freshId}\nevent: sync-required\ndata: ${JSON.stringify(snapshot)}\n\n`
      );
      return freshId;                                  // caller resumes tailing from here
    }
    return lastId;                                     // within window, proceed normally
  };
}

Null Byte in IDs #Permalink to this section

The WHATWG spec requires that if an id: field value contains U+0000 (null byte), the entire field is ignored. Never use binary IDs directly; always encode as hex, Base32, or Base64 before emitting.

Performance and Scale Considerations #Permalink to this section

Memory: Replay Window Sizing #Permalink to this section

A replay window of N events per channel costs roughly N × avg_event_size bytes in Redis. For 500 events averaging 400 bytes each, that is 200 KB per channel. With 10 000 channels, peak Redis memory for replay windows alone reaches 2 GB. Tune MAXLEN per channel based on reconnect frequency: high-churn channels (mobile clients) need deeper windows.

The Connection Pooling for SSE Servers guide covers how many Redis connections the replay query pool should hold relative to concurrent SSE connections.

CPU: ID Generation Throughput #Permalink to this section

At 1 000 concurrent connections each receiving 5 events/s, the server generates 5 000 IDs/s. Any in-process strategy comfortably handles this. Database or Redis-backed generation is only a bottleneck above ~50 k events/s per node.

Backpressure and ID Ordering #Permalink to this section

When backpressure causes the server to drop or coalesce events, the ID sequence will have gaps. This is intentional and correct. Clients must not assume contiguous IDs; they should treat the id: value as an opaque cursor, not an event count. Document this contract in your API specification.

Multi-Node Fan-Out #Permalink to this section

In a multi-node deployment where Redis Pub/Sub fan-out routes events to the correct SSE server, event IDs must be assigned before publishing to Redis—not by each subscriber node independently. This ensures all clients, regardless of which server they reconnect to, receive the same ID for the same event.

Publisher → assigns ID → XADD to Redis Stream → Subscriber nodes tail stream → forward to clients

Never assign the ID inside the SSE handler after receiving from Redis; the same logical event would get different IDs on different servers.

Validation and Debugging #Permalink to this section

Verify with curl #Permalink to this section

# Initial connection — check id: fields appear on every event
curl -N -H "Accept: text/event-stream" https://api.example.com/events/orders

# Simulate reconnect with a specific Last-Event-ID
curl -N \
  -H "Accept: text/event-stream" \
  -H "Last-Event-ID: 01HZ6M3P2E0000000000000ABC" \
  https://api.example.com/events/orders

Expected: the second request should receive only events with IDs strictly greater than 01HZ6M3P2E0000000000000ABC, and the first event should not be the sync-required event (meaning the ID was found in the replay window).

Confirm Proxy Does Not Strip IDs #Permalink to this section

# Capture the upstream request headers at the proxy to verify Last-Event-ID passes through
tcpdump -A -i eth0 port 443 | grep -i 'last-event-id'

# Or with httpie for human-friendly output
http --stream GET https://api.example.com/events/orders \
  "Last-Event-ID:01HZ6M3P2E0000000000000ABC"

DevTools Inspection #Permalink to this section

1. Open Chrome DevTools → Network tab → filter by EventStream or text/event-stream.
2. Click the SSE request → EventStream sub-tab.
3. Each row shows the event type, data, lastEventId, and time. Verify lastEventId updates on every event, not just the first.
4. Disconnect (disable network in DevTools) and re-enable. Watch the new request’s Headers tab for Last-Event-ID.

Debug Endpoint #Permalink to this section

Expose a lightweight introspection endpoint on your SSE server (restrict to internal networks):

// GET /debug/sse-state  — internal only
app.get('/debug/sse-state', async (req, res) => {
  const info = await redis.xInfo('STREAM', 'sse:events:orders');
  res.json({
    length: info.length,
    firstEntry: info['first-entry'],
    lastEntry: info['last-entry'],
    activeConnections: connectionRegistry.size,
    oldestIdInWindow: info['first-entry']?.[0],
  });
});

Structured Logging #Permalink to this section

Log every replay request with the requested ID, whether it was found in the window, and how many events were replayed. This lets you detect rising gap rates before they surface as user-visible data loss:

{
  "event": "sse_replay",
  "channel": "orders",
  "requested_id": "01HZ6M3P2E0000000000000ABC",
  "found_in_window": true,
  "replayed_count": 7,
  "connection_id": "conn_8f2a",
  "client_ip": "203.0.113.42"
}

Alert if found_in_window: false rate exceeds 2% of reconnect attempts in a 5-minute window—this indicates replay window exhaustion and imminent data loss for mobile clients on poor connections.

Production Checklist #Permalink to this section

• Every SSE event block includes an id: field; no events are emitted without one
• IDs are assigned before publishing to the message bus (Redis Stream / Pub-Sub)
• Node IDs or sequence-generator instances are unique per server process at startup
• X-Accel-Buffering: no and Cache-Control: no-cache are set on every SSE response
• Proxy layer tested with tcpdump to confirm Last-Event-ID header passes through
• Replay window sized to cover 99th-percentile reconnect interval × event rate + 20% buffer
• Gap handling emits sync-required event with full state snapshot (not 4xx)
• Debug endpoint /debug/sse-state restricted to internal networks
• Alert configured on found_in_window: false reconnect rate > 2%
• Client dedup cache bounded to ≤ 500 entries to prevent unbounded memory growth

Frequently Asked Questions #Permalink to this section

Related #Permalink to this section

• Backend Stream Generation & Connection Management
• Generating Monotonic Event IDs for SSE
• Redis Pub/Sub Fan-Out for SSE
• Buffer Management & Chunked Transfer Encoding
• Rate Limiting & Backpressure Handling
• Event ID & Retry Mechanism Design