Observability & Metrics for SSE

Part of Backend Stream Generation & Connection Management.

Standard HTTP observability assumes requests that begin, do something, and end. An SSE request begins and then stays — for minutes or hours — so every default is wrong: request rate under-counts by orders of magnitude, request duration is a histogram of session lengths rather than latency, and an error rate of zero tells you nothing because a stream that delivers no events still returns 200. Teams routinely discover a broken feed from user reports while every dashboard stays green. This guide covers what to measure instead, how to wire it up in Node.js, Go and Python, how to trace an event from producer to browser across a connection that outlives the request that opened it, and which of those signals are worth an alert.

How It Works

Three signal families cover an SSE deployment, and each answers a different question.

Three signal families, three questions One stream produces connection signals, delivery signals and freshness signals, each answering a different question about health. Three signal families, three questions One SSE stream open for minutes to hours Connection open · connects · duration Delivery sent · dropped · queue depth Freshness publish→write · first event
No single number covers a stream: it can be connected and silent, or delivering promptly to a population that is churning. Each family answers a question the others cannot.

Connection signals describe the population of open streams: how many exist, how long they live, how often they are established, and why they end. A gauge of currently open streams is the single most useful number on the dashboard — it should be flat-ish and track your user population, and any step change in it is a deploy, an outage, or a client bug.

Delivery signals describe what flows through those streams: events published, events written per stream, bytes written, queue depth, and events dropped by policy. This is the family that catches the “connection is fine but nothing arrives” failure, which the connection signals alone cannot see.

Freshness signals are derived and are the ones that map to user experience: time from event creation to socket write, time to first event after connect, and the age of the newest event a client has received. A stream can be open, healthy and utterly stale; only a freshness signal reveals it.

The instrumentation itself is unusual in one respect: because the request never ends, you cannot record duration on completion and be done. Metrics must be updated at three moments — on connect, on each write, and on disconnect — and the disconnect path must run even when the process is being shut down, or the gauge drifts upward forever.

// The three moments, in the order they must be wired.
onConnect:    streamsOpen.inc();  connectsTotal.inc({ topic });
onWrite:      eventsSent.inc({ topic });  bytesSent.inc(bytes);
onDisconnect: streamsOpen.dec();  streamDuration.observe(seconds, { reason });

Server-Side Implementation

Node.js with prom-client

The registry below defines the minimum viable metric set. Note that sse_streams_open is a gauge, not a counter, and that the disconnect handler decrements it exactly once — guarding against double-decrement is worth the two lines, because a gauge that goes negative is worse than no gauge at all.

// metrics.js
const client = require('prom-client');

client.collectDefaultMetrics();

const streamsOpen = new client.Gauge({
  name: 'sse_streams_open',
  help: 'Currently open SSE streams',
  labelNames: ['topic'],
});

const connectsTotal = new client.Counter({
  name: 'sse_connects_total',
  help: 'SSE connections established',
  labelNames: ['topic', 'resumed'],      // resumed: did the client send Last-Event-ID?
});

const streamDuration = new client.Histogram({
  name: 'sse_stream_duration_seconds',
  help: 'How long streams stayed open',
  labelNames: ['topic', 'reason'],       // reason: client_closed | server_shutdown | write_error
  buckets: [1, 5, 15, 60, 300, 900, 3600, 14400],
});

const eventsSent = new client.Counter({
  name: 'sse_events_sent_total',
  help: 'Events written to client sockets',
  labelNames: ['topic', 'event_type'],
});

const eventsDropped = new client.Counter({
  name: 'sse_events_dropped_total',
  help: 'Events discarded by a backpressure or rate-limit policy',
  labelNames: ['topic', 'policy'],       // policy: queue_full | rate_limited | coalesced
});

const publishToWrite = new client.Histogram({
  name: 'sse_publish_to_write_seconds',
  help: 'Latency from event creation to socket write',
  buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5],
});

const queueDepth = new client.Gauge({
  name: 'sse_queue_depth',
  help: 'Outbound queue depth summed across streams',
  labelNames: ['topic'],
});

module.exports = {
  client, streamsOpen, connectsTotal, streamDuration,
  eventsSent, eventsDropped, publishToWrite, queueDepth,
};

Wiring it into the handler is mechanical, but the finally-style guarantee on disconnect is the part that must not be skipped:

// stream-route.js
const m = require('./metrics');

app.get('/events', (req, res) => {
  const topic = req.query.topic ?? 'default';
  const resumed = String(Boolean(req.headers['last-event-id'] || req.query.lastEventId));
  const startedAt = process.hrtime.bigint();
  let closed = false;                                  // guard: decrement exactly once

  res.writeHead(200, {
    'Content-Type': 'text/event-stream; charset=utf-8',
    'Cache-Control': 'no-cache, no-transform',
    'X-Accel-Buffering': 'no',
  });
  res.flushHeaders();

  m.streamsOpen.inc({ topic });
  m.connectsTotal.inc({ topic, resumed });

  const send = (event) => {
    const frame = `id: ${event.id}\nevent: ${event.type}\ndata: ${event.json}\n\n`;
    res.write(frame);
    m.eventsSent.inc({ topic, event_type: event.type });
    // createdAt is stamped by the producer, so this measures the whole internal path.
    m.publishToWrite.observe((Date.now() - event.createdAt) / 1000);
  };

  const finish = (reason) => {
    if (closed) return;
    closed = true;
    unsubscribe(topic, send);
    m.streamsOpen.dec({ topic });
    m.streamDuration.observe(
      { topic, reason },
      Number(process.hrtime.bigint() - startedAt) / 1e9,
    );
  };

  subscribe(topic, send);
  req.on('close', () => finish('client_closed'));       // the normal path
  res.on('error', () => finish('write_error'));         // EPIPE and friends
  shutdown.onDrain(() => { finish('server_shutdown'); res.end(); });
});

app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', m.client.register.contentType);
  res.end(await m.client.register.metrics());
});

Go

Go’s prometheus/client_golang follows the same shape, with defer doing the work that the guard does in Node:

var (
    streamsOpen = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "sse_streams_open",
        Help: "Currently open SSE streams",
    }, []string{"topic"})

    streamDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "sse_stream_duration_seconds",
        Buckets: []float64{1, 5, 15, 60, 300, 900, 3600, 14400},
    }, []string{"topic", "reason"})

    eventsSent = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "sse_events_sent_total",
    }, []string{"topic", "event_type"})
)

func events(w http.ResponseWriter, r *http.Request) {
    topic := r.URL.Query().Get("topic")
    started := time.Now()
    reason := "client_closed"

    streamsOpen.WithLabelValues(topic).Inc()
    defer func() {
        streamsOpen.WithLabelValues(topic).Dec()
        streamDuration.WithLabelValues(topic, reason).Observe(time.Since(started).Seconds())
    }()

    // … stream setup …

    for {
        select {
        case <-r.Context().Done():
            return
        case <-shutdownC:
            reason = "server_shutdown"
            return
        case ev := <-sub:
            if _, err := fmt.Fprintf(w, "id: %s\ndata: %s\n\n", ev.ID, ev.JSON); err != nil {
                reason = "write_error"
                return
            }
            flusher.Flush()
            eventsSent.WithLabelValues(topic, ev.Type).Inc()
        }
    }
}

Tracing across the stream boundary

Distributed tracing assumes a span tree rooted in a request. An SSE stream breaks that assumption: the request that opened the stream is one span, but each delivered event belongs to whatever produced it, minutes later. Modelling it as one enormous span is useless — a trace of an hour-long stream tells you nothing about the event you care about.

The pattern that works is two-level. Keep a short span for the connection setup (auth, subscription, replay), end it once the stream is live, and then create one span per delivered event as a child of the producer’s trace context, with a link to the connection span. The producer’s context travels with the event through the message bus.

// Producer side: put the trace context on the event as it is published.
const { trace, context, propagation } = require('@opentelemetry/api');

function publish(topic, payload) {
  const carrier = {};
  propagation.inject(context.active(), carrier);        // W3C traceparent into the envelope
  bus.publish(topic, JSON.stringify({
    id: nextId(),
    createdAt: Date.now(),
    trace: carrier,
    payload,
  }));
}

// Stream side: continue the producer's trace for the delivery of THIS event.
const tracer = trace.getTracer('sse');

function deliver(res, event, connectionSpanContext) {
  const parent = propagation.extract(context.active(), event.trace);
  tracer.startActiveSpan('sse.deliver', {
    links: [{ context: connectionSpanContext }],        // link, not parent
    attributes: {
      'sse.topic': event.topic,
      'sse.event_id': event.id,
      'sse.event_type': event.type,
      'sse.bytes': Buffer.byteLength(event.json),
    },
  }, parent, (span) => {
    res.write(`id: ${event.id}\ndata: ${event.json}\n\n`);
    span.end();
  });
}

The result is a trace per business event that runs from the API call that caused it, through the bus, to the exact socket write — with a link back to the connection that carried it. That is the trace you want when a customer says an update never arrived.

Client-Side Consumption

Half of the failure modes are only visible from the browser. A proxy that buffers, a corporate network that blocks the path, a client bug that opens six connections: none of these produce a server-side signal that distinguishes them from normal operation. Report a small number of client metrics and the picture closes.

// Client-side stream telemetry, sampled.
function instrument(es, url, sample = 0.1) {
  if (Math.random() > sample) return es;
  const opened = performance.now();
  let firstEvent = null;
  let reconnects = 0;

  es.addEventListener('open', () => {
    if (reconnects) beacon('sse_reconnect', { url, count: reconnects });
  });

  es.addEventListener('message', () => {
    if (firstEvent === null) {
      firstEvent = performance.now() - opened;
      beacon('sse_time_to_first_event_ms', { url, value: Math.round(firstEvent) });
      const entry = performance.getEntriesByType('resource').find((r) => r.name.includes(url));
      if (entry) beacon('sse_protocol', { url, value: entry.nextHopProtocol });
    }
  });

  es.addEventListener('error', () => {
    reconnects += 1;
    beacon('sse_error', { url, readyState: es.readyState });
  });

  return es;
}

Time to first event is the highest-value client metric by a wide margin: a p95 in the tens of seconds is a buffering intermediary, and no server-side signal will show it.

Edge Cases & Network Interference

  • The gauge that never comes down. If the disconnect path is missed on any exit route — a thrown error, a shutdown, a write failure — sse_streams_open drifts upward and eventually reports more streams than the process has descriptors. Compare it against the real descriptor count periodically and alert on divergence.
  • Cardinality explosions. Labelling metrics with user or connection identifiers creates one time series per user. Keep labels to topic, event type and a small enumerated reason; put identifiers in logs, never in metric labels.
  • Per-process gauges behind a load balancer. Each replica reports its own open-stream count. Sum them in the query, and remember that a replica which dies without scraping loses its final value — the sum dips even though nothing was lost.
  • Histogram buckets tuned for requests. Default duration buckets top out around ten seconds, so every stream lands in +Inf and the histogram is useless. Use buckets that span seconds to hours.
  • Sampling that discards the interesting traces. Head-based sampling at 1% will rarely capture the event that failed to deliver. Sample delivery spans at a low rate but keep every span that records a drop or a write error.
  • Metrics endpoint on the streaming server. Scraping /metrics on a process holding fifty thousand streams is cheap, but the default Node registry collects garbage-collection stats that briefly pause the loop. Keep the scrape interval at 15 seconds or more.
How an open-stream gauge drifts upward Progression of a gauge that increments on connect but misses a decrement on an error exit path, drifting until it reports more streams than the process has descriptors. How an open-stream gauge drifts upward Connect gauge +1 Error exit no decrement inc() Drift gauge > real FDs handler skipped Reconciled compared to /proc/fd periodic check a guarded, idempotent cleanup on every path prevents the cycle
Every exit path must decrement exactly once. The reconciliation loop exists because a gauge that only ever goes up looks plausible for a long time.

Performance & Scale Considerations

Per-event instrumentation is on the hot path. A counter increment is tens of nanoseconds and irrelevant; a histogram observation with several labels is hundreds of nanoseconds and still fine; a log line per event is emphatically not, and at 20 000 streams receiving one event per second it will dominate the process’s CPU.

Instrumentation cost per delivered event Bar chart comparing the per-event cost of a counter increment, a labelled histogram observation, a sampled span and a JSON log line. Instrumentation cost per delivered event relative cost per event on a typical Node process counter.inc() 1× histogram + labels 4× sampled span 5× JSON log line 200× relative cost per delivered event
Metrics per event, logs per connection, traces per sampled event — the log line is two orders of magnitude more expensive than the metric it would duplicate.

The rule that scales: metrics per event, logs per connection, traces per sampled event. That keeps steady-state cost proportional to event rate for the cheap signal and proportional to connection churn for the expensive one.

// Cost per event, measured on a typical Node process (order of magnitude):
//   counter.inc()                  ~0.0001 ms
//   histogram.observe() + labels   ~0.0004 ms
//   JSON log line                  ~0.02   ms   ← 50–200× the metric cost
//   span start/end (sampled 1%)    ~0.0005 ms amortised

Queue depth deserves a gauge rather than a per-event metric: sample it on a timer across the registry instead of updating it on every enqueue and dequeue.

Validation & Debugging

Prove the gauge is honest before trusting any dashboard built on it:

# 1. Baseline the gauge and the real descriptor count.
curl -s localhost:9090/metrics | grep '^sse_streams_open'
ls /proc/$(pgrep -f sse-server)/fd | wc -l

# 2. Open ten streams, confirm the gauge moved by exactly ten.
for i in $(seq 1 10); do curl -N -s localhost:8080/events > /dev/null & done
sleep 2; curl -s localhost:9090/metrics | grep '^sse_streams_open'

# 3. Kill them and confirm it returns to baseline within a few seconds.
kill %1 %2 %3 %4 %5 %6 %7 %8 %9 %10
sleep 5; curl -s localhost:9090/metrics | grep '^sse_streams_open'

The queries worth having on a dashboard, in PromQL:

# Open streams across all replicas
sum(sse_streams_open)

# Reconnect churn — a rising ratio means the path is unstable
sum(rate(sse_connects_total[5m])) / sum(sse_streams_open)

# Median stream lifetime; a collapse to a round number is a timeout somewhere
histogram_quantile(0.5, sum(rate(sse_stream_duration_seconds_bucket[15m])) by (le))

# Delivery freshness — publish to socket write
histogram_quantile(0.95, sum(rate(sse_publish_to_write_seconds_bucket[5m])) by (le))

# Drop rate by policy
sum(rate(sse_events_dropped_total[5m])) by (policy)

Three alerts justify a page. Open streams falling by more than half in five minutes means a mass disconnection. Publish-to-write p95 above one second means the fan-out path is congested. Median stream duration collapsing to under a minute means something in the path is cutting connections — and matching that number against documented timeouts in the proxy and CDN configuration identifies the hop.

Production Checklist

Frequently Asked Questions

Why does request duration tell me nothing useful for SSE?

Because for a stream it measures how long the user kept the page open, not how long anything took. A p99 of four hours is a healthy stream, and a p50 of sixty seconds is a timeout being hit — the opposite of how you read the same metric for ordinary requests. Track stream duration deliberately, with hour-scale buckets, and read it as a connection-stability signal rather than a latency one.

How do I trace a single event from the API call that created it to the browser?

Inject the W3C trace context into the event envelope when it is published, carry it through the message bus, and start the delivery span with that context as its parent while linking to the connection span. You get one trace per business event spanning producer, bus and socket write — rather than one unusable hour-long span per connection.

Should I log every event I deliver?

No. At twenty thousand streams and one event per second per stream that is twenty thousand log lines per second, which will cost more CPU than the streaming itself. Log connection lifecycle events — open, resume, close with reason — and use metrics for per-event volume. Sample delivery traces instead when you need per-event detail.

My open-stream gauge keeps climbing even though users disconnect. What is wrong?

An exit path that skips the decrement. The usual culprits are an exception thrown before the cleanup handler is registered, a write error that ends the response without firing the close handler, and shutdown paths that exit the process without draining. Register the cleanup immediately after the gauge increment, guard it against running twice, and reconcile the gauge against the process descriptor count on a timer.

Which single metric should I alert on if I can only have one?

Client-reported time to first event. It is the only signal that catches the whole class of failures where the connection is established and healthy from the server's point of view but no data reaches the user — buffering proxies, compression, and blocked network paths. Every server-side metric looks perfect during exactly that failure.

Deep Dives

Alerting on SSE Connection-Count Drops Write alerts that catch a mass disconnection without paging for every deploy: relative thresholds, deploy suppression, and the companion signals that make the page actionable. Instrumenting SSE Connections with Prometheus Expose the four metrics that make a long-lived event stream observable — open streams, connect rate, stream duration and delivery latency — without cardinality blowups or a drifting gauge. Tracing SSE Events with OpenTelemetry Trace a single event from the API call that produced it, through the message bus, to the socket write — without creating one hour-long span per connection.