Log Tailing & CI Output Streaming

Part of Real-Time Application Patterns.

Every CI system, deployment tool and hosting dashboard has a page where text scrolls as a process runs: build logs, test output, container logs, a migration’s progress. Server-Sent Events fits this problem almost perfectly — the data is text, it flows one way, and the browser’s automatic reconnect with Last-Event-ID maps directly onto “continue from where I was”. It also stresses every weak point of a naive streaming implementation. Log output is bursty (a compiler can print ten thousand lines in a second, then nothing for a minute), unbounded (a long-running service logs forever), and mostly uninteresting (users usually want the last screen, or the lines matching an error). This guide covers the design that handles all three: byte-offset cursors for exact resume, bounded replay so a reconnect never re-sends a gigabyte, line batching to survive bursts, server-side filtering, and a client that renders a million lines without freezing. It is written for engineers building CI dashboards, internal platform tooling and log viewers.

How It Works

The core design decision is the cursor. For logs, the natural cursor is a byte offset into the log file or object: it is exact, cheap to seek to, and meaningful even across restarts of the streaming service. Each event carries a batch of complete lines, and its id is the offset immediately after the last byte in the batch.

Byte offsets as the replay cursor Flow from a log source appended by a process, through a tailer that reads from an offset and cuts at line boundaries, to SSE events whose id is the next offset, and a browser that returns it on reconnect. Byte offsets as the replay cursor Process appends to log write Log store file, object, stream seek Tailer read from offset batch lines SSE event id = next offset frames Browser Last-Event-ID
The id is where the next read starts. A reconnect resumes at exactly the next byte, with no duplicated or missing lines.
retry: 1000

event: lines
id: 18234
data: {"from":16012,"lines":["[12:03:01] Compiling core v0.4.2","[12:03:02] Compiling api v1.9.0"]}

event: lines
id: 19877
data: {"from":18234,"lines":["[12:03:04] warning: unused variable `ctx`","  --> src/handler.rs:88:9"]}

event: status
data: {"state":"running","step":"build","elapsedS":62}

event: end
id: 20411
data: {"state":"succeeded","exitCode":0}

Lines are sent inside a JSON array rather than as raw data: lines. Raw lines are tempting — SSE already splits data on newlines — but log output can contain anything, including lines that start with data: or id:, lone carriage returns from progress spinners, and invalid UTF-8 from binary output. JSON escaping sidesteps all of it, and the from field lets the client detect a gap if a batch is ever lost. The multiline data field rules explain what goes wrong with raw lines in more detail.

Offsets must be cut at line boundaries. A batch that ends mid-line would either split the line across two events or require the client to stitch fragments. The tailer reads up to a size limit, then trims back to the last newline and reports the offset of that newline plus one.

Server-Side Implementation

The tailer is the heart of it: read from an offset, follow the source as it grows, batch lines by size and time, and stop when the process ends.

// tail.js — follow a growing file from an offset, yielding line batches.
import { open } from 'node:fs/promises';
import { watch } from 'node:fs';

export async function* tailFile(path, fromOffset, { maxBatchBytes = 64 * 1024, isDone }) {
  const fh = await open(path, 'r');
  let offset = fromOffset;
  const buf = Buffer.alloc(maxBatchBytes);
  let wake = null;
  const watcher = watch(path, () => wake?.());              // growth notifications
  try {
    for (;;) {
      const { bytesRead } = await fh.read(buf, 0, maxBatchBytes, offset);
      if (bytesRead > 0) {
        const lastNl = buf.lastIndexOf(0x0a, bytesRead - 1);
        if (lastNl >= 0) {
          const chunk = buf.subarray(0, lastNl + 1).toString('utf8');
          const lines = chunk.split('\n'); lines.pop();       // drop the empty tail after the final \n
          yield { from: offset, next: offset + lastNl + 1, lines };
          offset += lastNl + 1;
          continue;                                         // more may already be waiting
        }
        if (bytesRead === maxBatchBytes) {                  // a single enormous line: emit it truncated
          yield { from: offset, next: offset + bytesRead, lines: [buf.toString('utf8', 0, 2048) + ' …[line truncated]'] };
          offset += bytesRead;
          continue;
        }
      }
      if (await isDone()) return;
      await new Promise((r) => { wake = r; setTimeout(r, 500); });  // wait for growth or poll
    }
  } finally {
    watcher.close();
    await fh.close();
  }
}

The handler bounds the replay and turns batches into events:

const MAX_REPLAY_BYTES = 2 * 1024 * 1024;       // never replay more than 2 MB on connect

app.get('/api/builds/:id/log', requireBuildAccess, async (req, res) => {
  const build = await builds.get(req.params.id);
  const size = await logSize(build.logPath);
  let from = Number(req.get('Last-Event-ID') ?? req.query.from ?? NaN);

  if (Number.isNaN(from)) from = Math.max(0, size - MAX_REPLAY_BYTES);   // new viewer: last 2 MB
  if (size - from > MAX_REPLAY_BYTES) {
    res.write(`event: truncated\ndata: ${JSON.stringify({ skippedBytes: size - MAX_REPLAY_BYTES - from })}\n\n`);
    from = size - MAX_REPLAY_BYTES;
  }

  openStream(res, { retryMs: 1000 });
  const controller = new AbortController();
  req.on('close', () => controller.abort());

  for await (const b of tailFile(build.logPath, from, { isDone: () => builds.isFinished(build.id) })) {
    if (controller.signal.aborted) return;
    const ok = res.write(`event: lines\nid: ${b.next}\ndata: ${JSON.stringify({ from: b.from, lines: b.lines })}\n\n`);
    if (!ok) await once(res, 'drain');            // respect backpressure from slow clients
  }
  const final = await builds.get(build.id);
  res.write(`event: end\ndata: ${JSON.stringify({ state: final.state, exitCode: final.exitCode })}\n\n`);
  res.end();
});

Two limits protect the server. MAX_REPLAY_BYTES stops a new viewer of a 4 GB log from triggering a 4 GB replay; they get the last 2 MB and a truncated event, with a “load earlier output” control that fetches older ranges over plain HTTP. And waiting on drain means a viewer on a slow connection slows only their own tail, instead of making the process buffer the whole log in memory for them — see handling slow consumers with SSE backpressure.

When the log lives in a Redis Stream rather than a file — a common choice for build runners that ship output to a central service — the stream entry id is the cursor, and XREAD BLOCK replaces the file watcher. The Python version below shows the same shape: bounded replay with XREVRANGE for new viewers, then blocking reads that batch whatever has arrived:

# log_stream.py — FastAPI tail of a Redis Stream, entry id as the SSE id.
import json
from fastapi import Request
from fastapi.responses import StreamingResponse

MAX_REPLAY = 2000        # entries, for a viewer without a cursor

@app.get("/api/builds/{build_id}/log")
async def build_log(build_id: str, request: Request):
    key = f"buildlog:{build_id}"
    cursor = request.headers.get("last-event-id")

    async def gen():
        nonlocal cursor
        yield "retry: 1000\n\n"
        if cursor is None:                                   # new viewer: last N entries only
            tail = await redis.xrevrange(key, count=MAX_REPLAY)
            tail.reverse()
            if tail:
                cursor = tail[-1][0].decode()
                lines = [f[b"line"].decode("utf-8", "replace") for _, f in tail]
                yield f"event: lines\nid: {cursor}\ndata: {json.dumps({'lines': lines})}\n\n"
            else:
                cursor = "0-0"
        while not await request.is_disconnected():
            resp = await redis.xread({key: cursor}, count=500, block=15000)
            if not resp:
                if await build_finished(build_id):
                    yield "event: end\ndata: {}\n\n"
                    return
                yield ": hb\n\n"
                continue
            entries = resp[0][1]
            cursor = entries[-1][0].decode()
            lines = [f[b"line"].decode("utf-8", "replace") for _, f in entries]
            yield f"event: lines\nid: {cursor}\ndata: {json.dumps({'lines': lines})}\n\n"

    return StreamingResponse(gen(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

Decoding with errors="replace" matters: build output is not guaranteed to be valid UTF-8, and a single stray byte must not crash the stream. Redis Stream ids are ordered and seekable, so they have the same exact-resume property as byte offsets; the MAXLEN you set on the stream becomes the hard replay limit.

When logs are not files — container logs from a runtime API, or lines arriving through a message queue — the cursor becomes the source’s own sequence (a Redis Stream id, a Kafka offset, a line number) but everything else is unchanged. Tailing server logs to the browser with SSE builds the version backed by journalctl and Docker, and filtering a live log stream on the server adds query parameters that cut traffic before it leaves the process.

Client-Side Consumption

A log viewer’s client has two jobs: parse batches into a line store without gaps, and render a potentially huge number of lines smoothly.

// log-view.js — line store with gap detection and a bounded in-memory window.
export function connectLog(url, view, { maxLines = 200_000 } = {}) {
  let expectedFrom = null;
  const lines = [];

  const es = new EventSource(url, { withCredentials: true });
  es.addEventListener('truncated', (e) => view.showTruncated(JSON.parse(e.data).skippedBytes));
  es.addEventListener('lines', (e) => {
    const batch = JSON.parse(e.data);
    if (expectedFrom !== null && batch.from !== expectedFrom) view.markGap(expectedFrom, batch.from);
    expectedFrom = Number(e.lastEventId);
    lines.push(...batch.lines);
    if (lines.length > maxLines) lines.splice(0, lines.length - maxLines);   // keep memory bounded
    view.scheduleRender(lines);                                                // once per frame
  });
  es.addEventListener('end', (e) => { es.close(); view.finish(JSON.parse(e.data)); });
  return () => es.close();
}

e.lastEventId is the id of the event being dispatched, which the server set to the next offset, so it is exactly the from the next batch should have. A mismatch means data was lost somewhere — rare, but worth showing honestly as a gap marker rather than silently.

Rendering must be virtualised: only the lines in the viewport become DOM nodes. A 200,000-line log rendered naively creates 200,000 elements and takes seconds to lay out; a virtual list renders about fifty. And the view must respect the user’s scroll position — auto-scroll to the bottom only while the user is already at the bottom, and stop following the moment they scroll up to read. Streaming CI build output with ANSI colours covers both, plus converting colour escape codes safely.

Time to render 200,000 log lines Bar chart comparing initial render time for a plain DOM list, a DOM list with ANSI colour spans, and a virtualised list. Time to render 200,000 log lines One div per line 4,200 ms One div per line + colour spans 7,900 ms Virtualised list 35 ms milliseconds to first paint after the replay arrives
A virtualised list renders the viewport, not the log, so its cost stays flat whether the build printed a thousand lines or a million.

Edge Cases & Network Interference

Log streams concentrate the classic streaming failures because they are both bursty and long-lived.

  • Burst buffering. A compiler burst fills proxy buffers instantly, so a buffering proxy may appear to work in testing with sparse output and fail under real bursts. Disable buffering on the route regardless of how it behaves with small outputs.
  • Compression. gzip on a text stream compresses beautifully and can hold small writes in the compressor until a block fills. Either disable compression for the route or flush the compressor after each event; see serving SSE from Express with compression enabled.
  • Log rotation. If the file is rotated mid-tail, the offset suddenly exceeds the new file’s size. Detect size < offset, emit a rotated event, and restart from zero of the new file. Because offsets are only meaningful within one file generation, prefix the id with a generation number (3:18234) so a client resuming after rotation is recognised rather than seeking into the wrong file.
Tailer states around a log rotation State diagram of a tailer following a file, detecting that the file shrank below its offset, emitting a rotated event and resuming at offset zero of the new generation. Tailer states around a log rotation FOLLOW gen 3 offset n SHRANK size below offset EMIT rotated event to clients FOLLOW gen 4 offset 0 stat reopen resume
The generation prefix on the id is what lets a reconnecting client be told "your offset belonged to the previous file" instead of silently reading the wrong bytes.
  • Carriage-return progress bars. Tools like npm and cargo redraw a line using \r. Sent verbatim, each redraw becomes a new line. Collapse \r-separated segments to the last one per line on the server.
  • Secrets in output. Build logs leak tokens. Mask known secret values before bytes leave the server; a client-side mask is useless because the raw text already crossed the network.
  • Very long lines. Minified bundles and JSON dumps produce multi-megabyte lines. Truncate on the server, with a marker and a download link for the raw log.

Mitigation checklist:

Performance & Scale Considerations

The cost profile of log streaming is dominated by bursts. Averages are misleading: a build that produces 30 MB of output over ten minutes might emit 20 MB of it in fifteen seconds of test output.

Output rate over a typical CI build Line chart of log output rate over ten minutes of a build, showing quiet dependency install, a large burst during compilation, a quiet period, and a second burst during tests. Output rate over a typical CI build bytes per second events per second × 10 0 400 800 1200 1600 0 2 4 6 8 10 minutes output (KB per second)
The stream has to be sized for the bursts. Batching lines into events of up to 64 KB keeps the event rate flat even when the byte rate spikes.

Batching turns bytes into a bounded number of events: at 64 KB per batch, even 1.5 MB per second is about 25 events per second, well within what a client can dispatch. Cut batches by time as well as size — flush at least every 100 ms — so that a slow trickle of lines is not held back waiting to fill a batch.

Many viewers of the same build are common (a failing main branch attracts a crowd). Each viewer reading the file independently multiplies disk reads, which the OS page cache usually absorbs. For object-storage-backed logs the reads are not free: put a per-build fan-out in front, so one tailer per node reads the source and all local viewers receive its batches, in the same shape as Redis pub/sub fan-out.

Validation & Debugging 

# Follow a build log and confirm events arrive promptly during a burst.
curl -sN -b s.txt https://ci.example.com/api/builds/8812/log \
  | awk '/^id:/ { printf "%s id=%s\n", strftime("%T"), $2; fflush() }'

# Resume from an exact offset and check the first batch starts there.
curl -sN -b s.txt -H 'Last-Event-ID: 18234' https://ci.example.com/api/builds/8812/log \
  | grep -m1 '^data:' | jq .from      # → 18234

# New viewer of a huge log: expect a truncated event first.
curl -sN -b s.txt https://ci.example.com/api/builds/7001/log | head -3

Log each viewer session with the bytes replayed and whether it was truncated:

{"evt":"log_stream_open","build":8812,"from":0,"replay_bytes":2097152,"truncated":true}
{"evt":"log_stream_close","build":8812,"bytes_sent":5188211,"batches":301,"drain_waits":12,"reason":"end"}

A high drain_waits count for office users suggests a proxy is throttling or buffering; for mobile users it is expected.

Production Checklist

Frequently Asked Questions

Why use byte offsets rather than line numbers as ids?

A byte offset can be passed straight to a file read or a ranged object fetch, so resuming costs one seek. Line numbers require counting lines from the start or maintaining an index. Use line numbers only when the source is already line-indexed.

Can I send each log line as its own SSE event?

At low rates, yes. During bursts, one event per line multiplies framing overhead and client dispatch work by thousands. Batching lines into events of up to tens of kilobytes keeps both flat.

How much history should a new viewer receive?

Enough to fill a few screens and show recent context — a megabyte or two, or the last few thousand lines. Older output should be fetched on demand over HTTP, not replayed through the stream.

How do I stop secrets appearing in streamed logs?

Mask them on the server before any byte is written to the stream: replace every known secret value, and common token patterns, with a placeholder as each batch is cut. Masking in the browser is too late, because the raw text has already crossed the network and may sit in proxy logs or caches.

Should the stream end when the build finishes?

Yes. Send a terminal event with the final state, end the response, and have the client close the EventSource. Otherwise the browser reconnects to a finished log forever.

Deep Dives

Filtering a Live Log Stream on the Server Filter live log streams by level, text or regex before they leave the server — safe query parameters, bounded regex cost, context lines and filter changes. Streaming CI Build Output with ANSI Colors Render live CI output with its colours intact — convert ANSI escape codes safely, carry style across batches, collapse carriage-return redraws and follow the tail. Tailing Server Logs to the Browser with SSE Build a browser log tail for journald, Docker and Kubernetes pods over SSE — spawning the follower safely, cursors that resume, and killing it on disconnect.