Alerting on SSE Connection-Count Drops

Part of Observability & Metrics for SSE, under Backend Stream Generation & Connection Management.

An open-stream gauge is the best single indicator of streaming health and the easiest to alert on badly. Absolute thresholds fire every night, deploys page the on-call every release, and the alert that finally matters gets muted along with the noise.

Symptom & Developer Intent

  • The connection-count alert fires on every deploy and has been silenced as a result.
  • A real mass disconnection went unnoticed for twenty minutes because the alert was muted.
  • The alert fires overnight when the population naturally halves, and again each morning.
  • A scaling event makes the summed gauge dip, producing a page for a replica that was intentionally removed.
  • When it does fire, the page says “connections dropped” and nothing about where to look.

The intent is an alert that fires for real losses, stays quiet for normal variation and deploys, and arrives with enough context to act on.

Root Cause Analysis

Connection counts move for four different reasons, and only one deserves a page.

Four reasons the count moves, one that deserves a page Table of the causes of a connection-count drop with the accompanying connect-rate behaviour and whether it warrants an alert. Four reasons the count moves, one that deserves a page Connect rate Page? Daily rhythm falls gradually no Deploy spikes then settles no — suppress Scale-in unchanged no Mass disconnect spikes hard yes Clients cut off flat — nobody returns yes, urgent
The connect rate is the discriminator: a drop with a reconnect spike is a move, and a drop without one is an outage.

Daily rhythm. The population follows the working day. Any absolute threshold is either too high at 3 a.m. or too low at 10 a.m.

Deploys. A rolling restart moves every stream from old replicas to new ones. The gauge dips for the length of the drain and recovers as clients reconnect, which is exactly the shape of a real incident.

Scaling. Removing a replica removes its gauge series. The sum drops by that replica’s share while nothing is actually wrong — and the series goes stale rather than reporting zero, so the drop can look larger than it is.

Real mass disconnection. A proxy restart, a load balancer timeout change, a bad deploy or a network partition drops streams that do not come back cleanly. Here the count falls and the connect rate spikes as everyone reconnects at once — or, in the worse case, it falls and the connect rate stays flat, meaning clients cannot reconnect at all.

That last distinction is what makes a good alert: the connect rate is the signal that separates “streams moved” from “streams were lost”.

Step-by-Step Resolution

The page fired — where to look first Triage tree from a connection-drop alert: a connect-rate spike points at a restart, a flat connect rate points in front of the application, and flat events point at the producer. The page fired — where to look first What is the connect rate doing? Streams moved restart or LB event spiking Clients cannot reach us LB · TLS · DNS flat Producer or bus not connections at all normal, events zero
Three branches, three different teams. Putting this tree in the alert annotation is worth more than another alert rule.

Step 1 — Alert on relative change, not an absolute floor

Compare against the recent past so the daily rhythm cancels out.

# Half the streams disappeared compared with five minutes ago.
sum(sse_streams_open) < 0.5 * sum(sse_streams_open offset 5m)

Step 2 — Require the drop to persist

A single scrape landing mid-drain is not an incident.

- alert: SSEMassDisconnect
  expr: sum(sse_streams_open) < 0.5 * sum(sse_streams_open offset 5m)
  for: 3m
  labels:
    severity: page
  annotations:
    summary: "SSE streams down sharply versus five minutes ago"

Step 3 — Suppress during deploys

Expose a deploy indicator as a metric and gate the alert on it, rather than relying on a human to silence it.

# Only alert when no deploy is in progress for this service.
sum(sse_streams_open) < 0.5 * sum(sse_streams_open offset 5m)
  unless on() (deploy_in_progress{service="sse"} == 1)

Where the deployment tool cannot emit that, changes() on the process start time is a serviceable proxy:

# Suppress if any replica restarted in the last five minutes.
(sum(sse_streams_open) < 0.5 * sum(sse_streams_open offset 5m))
  unless on() (changes(process_start_time_seconds{job="sse"}[5m]) > 0)

Step 4 — Add the companion alert for the worse failure

A drop with no reconnect attempts means clients cannot get back — the more serious case, and invisible to the first alert once the count stabilises at its new low level.

- alert: SSEClientsCannotReconnect
  # Streams are down AND almost nobody is trying to reconnect.
  expr: |
    sum(sse_streams_open) < 0.5 * sum(sse_streams_open offset 15m)
    and sum(rate(sse_connects_total[5m])) < 0.2 * sum(rate(sse_connects_total[15m] offset 15m))
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "SSE streams lost and reconnects are not arriving"
    runbook: "Check the load balancer, TLS certificates and DNS before the application."

Step 5 — Alert on the silent failure the count cannot see

A stable connection count with no events flowing is the failure users notice first and the gauge never reveals.

- alert: SSEStreamsOpenButSilent
  expr: |
    sum(sse_streams_open) > 100
    and sum(rate(sse_events_sent_total[10m])) == 0
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "Streams are open but no events have been written for ten minutes"

Step 6 — Put the diagnosis in the annotation

The page should tell the responder which direction to look.

annotations:
  dashboard: "https://grafana.internal/d/sse/overview"
  runbook: |
    1. Connect rate spiking? Streams moved — check for a restart or load balancer event.
    2. Connect rate flat? Clients cannot reach us — check load balancer, TLS, DNS.
    3. Count stable but events zero? Producer or bus problem, not connections.

Validation & Monitoring

Test each alert by causing the condition it claims to detect, in a staging environment.

A deploy and an incident look identical for two minutes Timeline of a rolling deploy showing the connection-count dip and recovery that mirrors a real incident until the recovery completes. A deploy and an incident look identical for two minutes time → Drain begins count falls t+0 Half gone looks like an incident t+40 s Reconnects land count recovers t+90 s Steady nothing was wrong t+3 min
This is why suppression is a deploy signal rather than a longer wait: waiting long enough to tell them apart also delays every real page by the same amount.
# 1. Mass disconnect: restart the proxy while streams are open.
sudo systemctl restart nginx
# Expect SSEMassDisconnect to fire and resolve as clients reconnect.

# 2. Cannot reconnect: block the port so clients drop and cannot return.
sudo iptables -A INPUT -p tcp --dport 443 -j DROP
# Expect SSEClientsCannotReconnect after the `for` window; then flush the rule.

# 3. Silent streams: stop the producer while leaving connections up.
sudo systemctl stop event-producer
# Expect SSEStreamsOpenButSilent; connection alerts should stay quiet.

Track alert quality the way you track any other signal: count firings, and how many led to an action. A rule that fired eleven times last month and produced no action is training people to ignore the next one — tighten the threshold or the for window rather than leaving it.

Keep a companion dashboard panel that shows open streams and connect rate on the same axis. The relationship between those two lines is the diagnosis, and having it one click from the page is worth more than any additional alert.

Production Checklist

Frequently Asked Questions

Why not alert when open streams fall below a fixed number?

Because the population follows the working day, so any fixed number is wrong for most of the day. A relative comparison against five or fifteen minutes ago cancels the rhythm out and asks the question you actually care about: did a lot of streams disappear suddenly?

How do I stop deploys paging the on-call every release?

Gate the alert on a deploy indicator your deployment tool emits, or fall back to a restart-count guard as a proxy for "a replica restarted recently". Both are better than a longer for window, which delays detection of real incidents by the same amount.

What does a drop with no matching spike in reconnects mean?

That clients cannot reach you at all. A normal mass disconnection produces an immediate surge of reconnect attempts; if that surge is missing, the failure is in front of the application — load balancer, TLS certificate, DNS or network — and the application logs will show nothing, because the requests never arrive.

Should I alert per replica or on the sum?

Alert on the sum for user impact, and use a separate, lower-severity rule for a single replica whose count falls to zero while others are healthy. The sum answers "are users affected"; the per-replica rule answers "is one node broken", and mixing the two in one alert makes both harder to act on.