Tracing SSE Events with OpenTelemetry

Part of Observability & Metrics for SSE, under Backend Stream Generation & Connection Management.

Auto-instrumentation turns an SSE endpoint into one span that stays open for the life of the connection. It is technically a trace and practically useless: it tells you a stream existed, not whether the event a customer is asking about was ever written to their socket.

Symptom & Developer Intent

  • Traces for the stream endpoint show a single span lasting minutes or hours with no children.
  • The trace waterfall for a business action stops at “published to bus” — delivery is invisible.
  • A customer reports a missing update and there is no way to tell whether it was produced, fanned out, or written.
  • Span exporters warn about long-running spans or drop them for exceeding a duration limit.
  • Sampling produces either no useful traces or an unaffordable volume.

The intent is one trace per delivered event, rooted in the producer’s trace, ending at the socket write.

Root Cause Analysis

Distributed tracing models a request as a span tree with a bounded lifetime. SSE breaks two of those assumptions at once.

Three ways to model the trace, and what each gives you Comparison of one long connection span, delivery spans parented to the connection, and delivery spans parented to the producer with a link. Three ways to model the trace, and what each gives you Exports when Answers One connection span on close, hours later nothing useful Child of connection per event wrong causality Child of producer + link per event end-to-end delivery
Only the third model answers the question people actually ask: was this event delivered to this customer, and where did the time go.

First, the connection outlives everything that happens on it. A span that covers the whole connection cannot be exported until the connection closes, so it is invisible while it matters and enormous when it finally arrives.

Second, the causal parent of a delivered event is not the request that opened the stream. It is whatever produced the event — an API call, a job, a database change — which happened later and in a different process. Modelling delivery as a child of the connection span records the wrong causality: it says “this stream produced this event”, when the truth is “this API call produced this event, which happened to be delivered on this stream”.

The correct model uses both relationships. The producer’s context is the parent, because that is the causal chain. The connection span is a link, because the stream is context rather than cause. OpenTelemetry supports exactly this distinction, and using it is what makes the resulting traces readable.

Step-by-Step Resolution

How the trace context reaches the socket write Path of a W3C trace context injected at publish time, carried through the bus envelope, and extracted to parent the delivery span. How the trace context reaches the socket write one publish produces one delivery span per subscribing connection Producer inject traceparent Bus envelope trace travels as data Extract parent context sse.deliver + link to connection W3C header JSON field per subscriber
The bus is the boundary that loses context unless you carry it as data — Redis pub/sub has no header concept, so the envelope field is the only option.

Step 1 — Keep the connection span short

Cover setup — authentication, subscription, replay — and end the span once the stream is live. Do not hold it open for the connection’s lifetime.

const { trace, context, propagation, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('sse');

app.get('/events', (req, res) => {
  const span = tracer.startSpan('sse.connect', {
    attributes: {
      'sse.topic': req.query.topic,
      'sse.resumed': Boolean(req.headers['last-event-id']),
      'http.version': req.httpVersion,
    },
  });

  // Keep the span context so delivery spans can LINK to it later.
  const connectionCtx = span.spanContext();

  try {
    authorise(req);
    writeHeaders(res);
    replayMissed(res, req.headers['last-event-id']);
    subscribe(req.query.topic, (event) => deliver(res, event, connectionCtx));
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
  } finally {
    span.end();          // ends in milliseconds, not hours
  }
});

Step 2 — Inject the producer’s context into the event envelope

The trace context must survive the message bus, so it travels as data alongside the payload.

// Producer: whatever creates the event — an API handler, a job, a change feed.
function publish(topic, payload) {
  const carrier = {};
  propagation.inject(context.active(), carrier);      // W3C traceparent + tracestate

  bus.publish(topic, JSON.stringify({
    id: nextId(),
    createdAt: Date.now(),
    trace: carrier,                                   // travels with the event
    payload,
  }));
}

Step 3 — Start the delivery span as a child of the producer, linked to the connection 

function deliver(res, event, connectionCtx) {
  const parentCtx = propagation.extract(context.active(), event.trace);

  tracer.startActiveSpan(
    'sse.deliver',
    {
      links: [{ context: connectionCtx }],            // the stream is context, not cause
      attributes: {
        'sse.topic': event.topic,
        'sse.event_id': event.id,
        'sse.event_type': event.type,
        'sse.payload_bytes': Buffer.byteLength(event.json),
        'sse.queue_ms': Date.now() - event.createdAt,
      },
    },
    parentCtx,                                        // producer trace continues here
    (span) => {
      try {
        res.write(`id: ${event.id}\nevent: ${event.type}\ndata: ${event.json}\n\n`);
      } catch (err) {
        span.recordException(err);
        span.setStatus({ code: SpanStatusCode.ERROR });
      } finally {
        span.end();
      }
    },
  );
}

Step 4 — Sample so the interesting traces survive

Head-based sampling at 1% will almost never capture the delivery that failed. Sample low by default and force-keep anything that went wrong.

// Force a sampling decision for the spans that matter regardless of the base rate.
function deliverWithBias(res, event, connectionCtx, sink) {
  const forced = sink.queueDepth > sink.capacity * 0.8 || sink.lastWriteFailed;
  tracer.startActiveSpan('sse.deliver', {
    attributes: { 'sampling.priority': forced ? 1 : 0 },
    links: [{ context: connectionCtx }],
  }, (span) => { /* … */ });
}

A tail-based sampler in the collector is the cleaner version of the same idea: keep every trace containing an error or a delivery slower than a threshold, sample the rest at a low rate.

Step 5 — Disable auto-instrumentation for the stream route

Otherwise the framework’s HTTP instrumentation recreates exactly the long span you removed.

const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');

new HttpInstrumentation({
  ignoreIncomingRequestHook: (req) => (req.url ?? '').startsWith('/events'),
});

Validation & Monitoring

The result to verify is a readable waterfall for a business action, ending at socket writes.

Span volume per hour at three sampling strategies Bar chart of delivery spans exported per hour under full sampling, one percent head sampling and tail sampling biased to errors. Span volume per hour at three sampling strategies 10 000 streams × 5 events/s 100% sampling 180 000 000 1% head sampling 1 800 000 1% + all errors (tail) 1 820 000 spans exported per hour
Tail sampling keeps the traces worth having: a small base rate plus every delivery that errored or ran slow.
POST /api/orders                     42 ms
├─ db.insert order                   11 ms
├─ bus.publish orders                 2 ms
│  ├─ sse.deliver (event 8821)        1.2 ms   ↳ link: sse.connect (conn 4f2a)
│  ├─ sse.deliver (event 8821)        0.9 ms   ↳ link: sse.connect (conn 91bd)
│  └─ sse.deliver (event 8821)        1.4 ms   ↳ link: sse.connect (conn c30e)

Three deliveries for one publish is the fan-out, visible as siblings. If a subscriber that should have received the event has no delivery span, the gap is between the bus and that node — which is precisely the question that was unanswerable before.

# Confirm the context is actually on the wire in the bus envelope.
redis-cli SUBSCRIBE orders | head -20 | grep -o '"traceparent":"[^"]*"'
# "traceparent":"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"

Worth tracking as derived metrics: delivery span duration p95, the ratio of delivery spans to publish spans (which should equal the subscriber count), and the count of delivery spans ending in error. A ratio below the subscriber count means events are being lost between the bus and some nodes.

Production Checklist

Frequently Asked Questions

Why link to the connection span instead of making it the parent?

Because the connection did not cause the event. The API call or job that produced it did, and that is the chain someone debugging a missing update needs to follow. A link records "this delivery happened on that stream" without falsely claiming the stream caused the event — and it keeps the producer's trace intact end to end.

Will one span per delivered event overwhelm my tracing backend?

Yes, at any real volume — which is why the sampling strategy is part of the design rather than an afterthought. Sample delivery spans at a low base rate and force-keep the ones that error or exceed a latency threshold. Metrics carry the volume; traces carry the exceptions.

How does the trace context survive Redis pub/sub or Kafka?

As data. Inject it into the message envelope as a traceparent field before publishing, and extract it on the consuming side. Some Kafka instrumentations do this through headers automatically; Redis pub/sub has no header concept, so the envelope field is the only option.

Should the event payload be a span attribute?

No. Payloads frequently contain personal data, they inflate span size, and most backends truncate long attributes anyway. Record the event id, type and byte size — enough to correlate with your logs and to reason about size, without putting customer data in a tracing system.