Instrumenting SSE Connections with Prometheus

Part of Observability & Metrics for SSE, under Backend Stream Generation & Connection Management.

The default HTTP metrics every framework ships are actively misleading for streams: request rate under-counts by three orders of magnitude and request duration measures how long users kept a tab open. This page replaces them with four metrics that mean something.

Symptom & Developer Intent

  • The Grafana dashboard is green while users report a dead feed.
  • Request rate for the stream endpoint shows a handful of requests per hour and looks like a dead endpoint.
  • Request-duration histograms are entirely in the +Inf bucket, so p99 is meaningless.
  • Nobody can answer “how many streams are open right now?” without SSHing to a box and counting descriptors.
  • After an incident there is no way to tell whether connections dropped or events stopped.

The intent is a metric set that answers those questions directly, and a gauge that can be trusted.

Root Cause Analysis

Prometheus HTTP middleware is built around a request that terminates: it observes duration and increments a counter on completion. For a stream, completion happens minutes to hours later, so every derived signal is wrong.

What default HTTP metrics report for a stream Table contrasting what standard request metrics measure for an ordinary request and for a long-lived stream. What default HTTP metrics report for a stream For a request For a stream request rate load connects only request duration latency session length error rate failures always ~0 in-flight requests concurrency open streams
The same metric name means something entirely different once the request stops ending — which is why a green dashboard can sit above a dead feed.

The deeper problem is that a stream has two independent health properties and standard metrics conflate them. A connection can be established — socket open, no errors — while delivering nothing, because the producer stopped or a proxy is buffering. Conversely, events can be flowing while connection churn is pathological. One number cannot express both, which is why the metric set below separates connection signals from delivery signals.

The gauge itself introduces the classic failure: inc() on connect and dec() on disconnect is only correct if the decrement runs on every exit path — normal close, write error, shutdown, and any exception thrown in between. Miss one and the gauge drifts upward forever, eventually reporting more open streams than the process has descriptors.

Step-by-Step Resolution

The three moments instrumentation must cover Pipeline of the instrumentation points: increment on connect, observe per write, and decrement exactly once on every exit path. The three moments instrumentation must cover close, write error and shutdown all route through the same guarded cleanup Connect gauge +1 · counter Each write sent · latency Exit gauge −1 · duration once per event
The guard on the third moment is what makes the gauge trustworthy — without it the number only ever grows.

Step 1 — Define the metric set

Four metrics cover the ground. Keep labels to a low-cardinality set — topic and an enumerated reason — and never include user or connection identifiers.

// metrics.js
const client = require('prom-client');
client.collectDefaultMetrics();

const streamsOpen = new client.Gauge({
  name: 'sse_streams_open',
  help: 'Currently open SSE streams',
  labelNames: ['topic'],
});

const connectsTotal = new client.Counter({
  name: 'sse_connects_total',
  help: 'SSE connections established',
  labelNames: ['topic', 'resumed'],
});

const streamDuration = new client.Histogram({
  name: 'sse_stream_duration_seconds',
  help: 'How long streams stayed open',
  labelNames: ['topic', 'reason'],
  // Seconds to hours — request-shaped buckets would put every stream in +Inf.
  buckets: [1, 5, 15, 60, 300, 900, 3600, 14400],
});

const deliveryLatency = new client.Histogram({
  name: 'sse_publish_to_write_seconds',
  help: 'Latency from event creation to socket write',
  buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5],
});

module.exports = { client, streamsOpen, connectsTotal, streamDuration, deliveryLatency };

Step 2 — Instrument the handler with a single-fire cleanup

The guard is the part that makes the gauge trustworthy.

const m = require('./metrics');

app.get('/events', (req, res) => {
  const topic = req.query.topic ?? 'default';
  const resumed = String(Boolean(req.headers['last-event-id']));
  const started = process.hrtime.bigint();
  let done = false;                       // ensures exactly one decrement

  res.writeHead(200, {
    'Content-Type': 'text/event-stream; charset=utf-8',
    'Cache-Control': 'no-cache, no-transform',
  });
  res.flushHeaders();

  m.streamsOpen.inc({ topic });
  m.connectsTotal.inc({ topic, resumed });

  const finish = (reason) => {
    if (done) return;
    done = true;
    unsubscribe(topic, send);
    m.streamsOpen.dec({ topic });
    m.streamDuration.observe({ topic, reason },
      Number(process.hrtime.bigint() - started) / 1e9);
  };

  const send = (event) => {
    res.write(`id: ${event.id}\ndata: ${event.json}\n\n`);
    // createdAt is stamped by the producer, so this covers the whole internal path.
    m.deliveryLatency.observe((Date.now() - event.createdAt) / 1000);
  };

  subscribe(topic, send);
  req.on('close', () => finish('client_closed'));
  res.on('error', () => finish('write_error'));
  onShutdown(() => { finish('server_shutdown'); res.end(); });
});

Step 3 — Expose the endpoint 

app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', m.client.register.contentType);
  res.end(await m.client.register.metrics());
});

Keep it on an internal port or behind network policy — the metric set reveals topic names and traffic shape.

Step 4 — Scrape and reconcile 

# prometheus.yml — 15s is frequent enough for connection-count movement.
scrape_configs:
  - job_name: 'sse'
    scrape_interval: 15s
    static_configs:
      - targets: ['sse-1:9090', 'sse-2:9090', 'sse-3:9090']

Add a reconciliation loop so a drifting gauge is caught by the system rather than by a person:

// Periodically compare the gauge against the real descriptor count.
const fs = require('node:fs/promises');

setInterval(async () => {
  const fds = (await fs.readdir(`/proc/${process.pid}/fd`)).length;
  const gauge = await m.streamsOpen.get();
  const counted = gauge.values.reduce((a, v) => a + v.value, 0);
  if (counted > fds) {
    logger.error({ event: 'gauge_drift', counted, fds });   // impossible: gauge is wrong
  }
}, 60_000);

Step 5 — Build the four queries that matter 

# Open streams across every replica
sum(sse_streams_open)

# Reconnect churn: connects per open stream. Rising means an unstable path.
sum(rate(sse_connects_total[5m])) / sum(sse_streams_open)

# Median stream lifetime. A round number is a timeout, not user behaviour.
histogram_quantile(0.5, sum(rate(sse_stream_duration_seconds_bucket[15m])) by (le))

# Delivery freshness, p95, producer to socket write.
histogram_quantile(0.95, sum(rate(sse_publish_to_write_seconds_bucket[5m])) by (le))

Validation & Monitoring

Prove the gauge before building anything on top of it.

Proving the gauge before trusting it Verification sequence opening ten streams, confirming the gauge moved by ten, killing them and confirming it returns to baseline. Proving the gauge before trusting it Operator Process /metrics baseline → 0 open 10 streams read → 10 kill all clients read → 0 within seconds
A gauge nobody has proved is a number nobody should alert on. This takes a minute and catches the leak that would otherwise surface as a mystery months later.
# Baseline.
curl -s localhost:9090/metrics | grep '^sse_streams_open'

# Open ten streams; the gauge must move by exactly ten.
for i in $(seq 1 10); do curl -N -s localhost:8080/events > /dev/null & done
sleep 2 && curl -s localhost:9090/metrics | grep '^sse_streams_open'

# Kill them; it must return to baseline within seconds.
kill $(jobs -p); sleep 5
curl -s localhost:9090/metrics | grep '^sse_streams_open'

Then force the failure paths deliberately: kill a client mid-write, and send SIGTERM while streams are open. Both must produce a decrement and a sse_stream_duration_seconds observation with the right reason label. A path that produces neither is the leak that will drift your gauge in production.

Three alerts justify a page:

groups:
  - name: sse
    rules:
      - alert: SSEMassDisconnect
        expr: sum(sse_streams_open) < 0.5 * sum(sse_streams_open offset 5m)
        for: 2m
      - alert: SSEDeliveryLagging
        expr: histogram_quantile(0.95, sum(rate(sse_publish_to_write_seconds_bucket[5m])) by (le)) > 1
        for: 5m
      - alert: SSEStreamsCutShort
        expr: histogram_quantile(0.5, sum(rate(sse_stream_duration_seconds_bucket[30m])) by (le)) < 90
        for: 10m

The third is the one that catches an infrastructure timeout: a median stream life under ninety seconds means something in the path is cutting connections, and matching the number against documented defaults in Proxy & CDN Configuration for SSE identifies the hop.

Production Checklist

Frequently Asked Questions

Why not just use the framework's built-in HTTP metrics?

Because they measure the wrong thing for a response that never ends. Request rate counts one event per stream rather than per delivered message, and request duration becomes a histogram of how long users kept tabs open — with every observation in the top bucket. The connection gauge and the delivery-latency histogram are the equivalents that carry information.

Should the topic be a metric label?

Yes, if the set of topics is small and bounded — a handful of feed types is fine. If topics are per-user or per-document, the cardinality explodes into one time series per entity and will take down your Prometheus. In that case label by topic class and put the specific identifier in logs.

My gauge is negative. How?

The decrement ran more than once for one stream — typically because both the request close handler and an error handler fired. Guard the cleanup with a boolean that makes it idempotent; that single flag is what keeps the gauge honest across every exit path.

How do I aggregate the gauge across replicas correctly?

Use sum(sse_streams_open) rather than avg, and be aware that a replica which dies between scrapes takes its last value with it, so the sum dips by that replica's share until the series goes stale. Correlate any sudden step with replica count before treating it as a mass disconnection.