Debugging Supabase Realtime subscription dropouts end-to-end

The pager went off at 03:14 on a Tuesday and the message was the one you never want to see in a realtime app: "dashboard hasn't updated in six hours, customer thinks their orders are lost." I opened the Supabase logs expecting a stack trace, a 500, a broken RLS policy — something loud. There was nothing. The subscription was healthy according to the SDK, the socket was open according to the browser, and the Postgres CDC stream was ticking along according to the server. Everything agreed the pipe was fine. The pipe was not fine. Somewhere between the database emitting a change and the client rendering it, events were quietly falling on the floor, and neither end of the wire had the courtesy to admit it.

This article rebuilds that debugging session as a working TypeScript project you can clone and run against your own Supabase instance. We wire up @supabase/supabase-js from scratch, add a dropout detector that watches join and leave timestamps, then reproduce three failure modes I have hit in production — a network flap, a JWT expiring mid-session, and an RLS policy change that silently unsubscribes half the channel. From there we bolt on a heartbeat monitor, exponential backoff with jitter, a REST-based catch-up that reconciles rows missed during the outage, and finally the metrics and structured logs you need to catch the next one before the pager does. The rule I want you to leave with: a realtime subscription is not "connected" until you can prove it delivered the last event, and the SDK will not prove that for you.

This is for the engineer whose realtime app works fine in staging and drifts in production, and who is tired of blaming "the network." By the end you will have a repo that detects dropouts within seconds, recovers without losing rows, and emits the signals your on-call rotation actually needs at 3 a.m.

Step 1: Scaffolding a TypeScript Baseline for the Supabase Realtime Client

Realtime subscription dropouts are one of those bugs that only reveal themselves under a specific combination of network conditions, client configuration, and channel lifecycle. Before we can chase them, we need a small, deterministic harness that connects to a Supabase project the same way a production client would. This first step lays that harness down.

We deliberately keep the surface tiny: a config loader that reads two environment variables, a factory that instantiates the client with explicit Realtime options, and a pair of test files that pin the shape of both. Every diagnostic we bolt on in later steps then has one obvious place to plug into.

Setup

Create a Bun project with three source files and two test files:

codebase/
├── package.json
├── tsconfig.json
├── src/
│   ├── config.ts
│   ├── client.ts
│   └── index.ts
└── tests/
    ├── config.test.ts
    └── client.test.ts

The only runtime dependency is @supabase/supabase-js. typescript and @types/bun sit in devDependencies so that tsc --noEmit and bun test both have the types they need:

{
  "name": "supabase-realtime-subscription-dropouts",
  "type": "module",
  "private": true,
  "scripts": {
    "test": "bun test",
    "typecheck": "tsc --noEmit"
  },
  "dependencies": {
    "@supabase/supabase-js": "^2.45.4"
  },
  "devDependencies": {
    "@types/bun": "^1.1.10",
    "typescript": "^5.6.2"
  }
}

tsconfig.json turns on the strict family (strict, noImplicitAny, noUnusedLocals, noUnusedParameters) and targets ES2022 with bundler resolution. That combination lets us keep native .ts imports without a build step.

Implementation

Start with src/config.ts. Reading env vars from an injectable record (instead of touching process.env directly) is the small design choice that pays off later — every test can hand in its own fake environment without polluting the real one:

export interface RealtimeConfig {
  url: string;
  anonKey: string;
}

export function loadConfig(
  env: Record<string, string | undefined> = process.env,
): RealtimeConfig {
  const url = env.SUPABASE_URL ?? "";
  const anonKey = env.SUPABASE_ANON_KEY ?? "";
  if (!url) {
    throw new Error("SUPABASE_URL is required (set it in your environment)");
  }
  if (!anonKey) {
    throw new Error(
      "SUPABASE_ANON_KEY is required (set it in your environment)",
    );
  }
  return { url, anonKey };
}

The loader fails loud rather than silently returning empty strings. A missing credential is the single most common cause of a "phantom" dropout — the subscription never actually opened — so we refuse to instantiate anything until both values are present.

Next, src/client.ts wraps createClient with Realtime-tuned options:

import { createClient, type SupabaseClient } from "@supabase/supabase-js";
import type { RealtimeConfig } from "./config.ts";

export function createRealtimeClient(config: RealtimeConfig): SupabaseClient {
  return createClient(config.url, config.anonKey, {
    auth: {
      persistSession: false,
      autoRefreshToken: false,
    },
    realtime: {
      params: {
        eventsPerSecond: 10,
      },
    },
  });
}

Two things are worth calling out. persistSession and autoRefreshToken are disabled because this harness is a short-lived process — we do not want the auth loop reconnecting behind our back and masking a Realtime failure as an auth event. And eventsPerSecond: 10 sets a low, predictable throttle so later steps can reason about backpressure without racing the default limit.

src/index.ts re-exports both surfaces so tests (and any future CLI entry point) import from a single place:

export { loadConfig, type RealtimeConfig } from "./config.ts";
export { createRealtimeClient } from "./client.ts";

The two test files pin the contracts. tests/config.test.ts covers the four env-var branches: both present, each one missing, and an empty environment. tests/client.test.ts asserts that the returned client exposes channel, removeChannel, and a realtime transport whose endpoint contains the URL we passed in — enough shape-checking to catch a broken upgrade of @supabase/supabase-js in CI without spinning up a real socket.

Verification

Run the test suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 7 pass
 0 fail
 11 expect() calls
Ran 7 tests across 2 files. [201.00ms]

Seven passing tests, no failures — four config branches and three client-shape checks — completed in about 200 ms.

What we built

We now have a Supabase Realtime client that we can instantiate deterministically. The credential path is validated up front, and the Realtime transport is configured with explicit, non-default options that later steps will lean on.

The tests act as a smoke fence around both files. If an upgrade of @supabase/supabase-js silently changes the client's shape (say, renames removeChannel or restructures realtime.endPoint), the fence trips before we ever see a mysterious dropout in a production reproduction.

Strict TypeScript plus Bun's zero-config test runner keeps the feedback loop short. A full run costs a couple hundred milliseconds, which is exactly the budget we want for the diagnostic experiments coming in step 2.

Most importantly, everything is contained. No global singletons, no ambient env reads, and no build step between edit and test — a clean baseline for the observability we are about to add.

Repository

The state of the code after this step: e93829f

Step 2: Wiring a Postgres Changes Channel to a Structured JSON Log

Step 1 gave us a Supabase Realtime client we can instantiate deterministically. That is enough to connect, but it is not enough to observe — and dropouts are, by definition, a failure of observation. Before we can reproduce or diagnose them, every state transition the client goes through has to leave a printable trace.

In this step we bolt two small pieces onto the baseline: a dependency-free structured logger, and a subscribeToTable helper that opens a postgres_changes channel and pipes its lifecycle callbacks and row payloads into that logger. Both are pure enough to unit-test without a real socket, so the observability itself gains a fence around it.

Setup

Two new source files and two new test files land alongside the step 1 layout. No new runtime dependencies — the logger is a plain function and the subscribe helper only touches types we already pull in through @supabase/supabase-js:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts        # new
│   ├── subscribe.ts     # new
│   └── index.ts         # re-exports the new surfaces
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts   # new
    └── subscribe.test.ts # new

package.json is unchanged from step 1. Everything here compiles under the same strict tsconfig.json and runs under the same bun test invocation.

Implementation

Start with src/logger.ts. The logger is intentionally a one-file abstraction: a LogEvent shape, an injectable LogSink, and a factory that stamps every event with an ISO timestamp. The clock is a parameter so tests can pin the timestamp without mocking Date:

export type LogLevel = "info" | "warn" | "error";

export interface LogEvent {
  ts: string;
  level: LogLevel;
  msg: string;
  data?: Record<string, unknown>;
}

export type LogSink = (event: LogEvent) => void;

export const consoleSink: LogSink = (event) => {
  const line = JSON.stringify(event);
  if (event.level === "error") {
    console.error(line);
    return;
  }
  if (event.level === "warn") {
    console.warn(line);
    return;
  }
  console.log(line);
};

export function createLogger(
  sink: LogSink = consoleSink,
  now: () => Date = () => new Date(),
): Logger {
  const emit = (level: LogLevel, msg: string, data?: Record<string, unknown>) => {
    sink({ ts: now().toISOString(), level, msg, data });
  };
  return {
    info: (msg, data) => emit("info", msg, data),
    warn: (msg, data) => emit("warn", msg, data),
    error: (msg, data) => emit("error", msg, data),
  };
}

Two design choices are worth naming. The sink is a function, not a class, so tests just push events into an array and assert against them. And the level routing lives inside consoleSink rather than the factory, which keeps createLogger trivial and lets a downstream consumer swap in a file sink or a network sink without touching the factory at all.

Next, src/subscribe.ts wraps the Realtime client with the smallest surface that a dropout investigation needs. It declares structural ChannelLike and ClientLike interfaces instead of importing the concrete Supabase types, which keeps the helper testable with a fake channel and immune to internal shape changes in @supabase/supabase-js:

export interface ChannelLike {
  on(
    type: "postgres_changes",
    filter: { event: ChangeEvent; schema: string; table: string; filter?: string },
    callback: (payload: PostgresChangePayload) => void,
  ): ChannelLike;
  subscribe(
    callback?: (status: SubscribeStatus, err?: Error) => void,
  ): ChannelLike;
  unsubscribe(): Promise<"ok" | "timed out" | "error">;
}

export interface ClientLike {
  channel(name: string): ChannelLike;
  removeChannel(ch: ChannelLike): Promise<"ok" | "timed out" | "error">;
}

The body of subscribeToTable is deliberately linear. It derives a channel name from the schema, table, and event, emits a channel.opening log before touching the network, registers the postgres_changes listener, and hands the status callback to a tiny helper:

export function subscribeToTable(
  client: ClientLike,
  options: SubscriptionOptions,
  logger: Logger,
): Subscription {
  const schema = options.schema ?? "public";
  const event = options.event ?? "*";
  const table = options.table;
  const channelName = options.channelName ?? defaultChannelName(schema, table, event);

  const filter: { event: ChangeEvent; schema: string; table: string; filter?: string } = {
    event,
    schema,
    table,
  };
  if (options.filter) {
    filter.filter = options.filter;
  }

  logger.info("channel.opening", { channel: channelName, schema, table, event });

  const channel = client.channel(channelName);
  channel
    .on("postgres_changes", filter, (payload) => {
      logger.info("pg.change", {
        channel: channelName,
        eventType: payload.eventType,
        schema: payload.schema,
        table: payload.table,
        commit_timestamp: payload.commit_timestamp,
        new: payload.new ?? null,
        old: payload.old ?? null,
      });
    })
    .subscribe((status, err) => {
      logStatus(logger, channelName, status, err);
    });

  return {
    channel,
    close: async () => {
      logger.info("channel.closing", { channel: channelName });
      await client.removeChannel(channel);
    },
  };
}

Two invariants matter here. First, we log the intent to open before calling client.channel(...) — if the client throws synchronously, the log still records that we tried, which is exactly the trace a dropout investigation wants. Second, close() is an explicit method on the returned Subscription rather than a global teardown, so any consumer that opens N channels tears down N channels without ambiguity.

The status callback is split out because it is the piece we will iterate on most in later steps. Right now the rule is simple: SUBSCRIBED is info, CHANNEL_ERROR is error and captures err.message, and the two "we lost the channel" states — TIMED_OUT and CLOSED — are warn:

function logStatus(
  logger: Logger,
  channelName: string,
  status: SubscribeStatus,
  err?: Error,
): void {
  const base = { channel: channelName, status };
  if (status === "SUBSCRIBED") {
    logger.info("channel.status", base);
    return;
  }
  if (status === "CHANNEL_ERROR") {
    logger.error("channel.status", { ...base, error: err?.message ?? null });
    return;
  }
  logger.warn("channel.status", { ...base, error: err?.message ?? null });
}

The two-level if/return shape keeps the function under the codebase's max-nesting rule. When we start distinguishing "silent" timeouts from real errors in a later step, this helper is the only place that changes.

Finally, src/index.ts re-exports the new surfaces so consumers keep a single import path:

export { loadConfig, type RealtimeConfig } from "./config.ts";
export { createRealtimeClient } from "./client.ts";
export {
  createLogger,
  consoleSink,
  type Logger,
  type LogEvent,
  type LogLevel,
  type LogSink,
} from "./logger.ts";
export {
  subscribeToTable,
  type ChangeEvent,
  type ChannelLike,
  type ClientLike,
  type PostgresChangePayload,
  type SubscribeStatus,
  type Subscription,
  type SubscriptionOptions,
} from "./subscribe.ts";

The tests do the heavy lifting for confidence. tests/logger.test.ts pins the ISO-timestamp contract, the three levels, and the "data is optional" branch. tests/subscribe.test.ts builds a FakeChannel that records on calls and exposes pushStatus and emit, then drives the helper through eight scenarios: default channel-name derivation, custom overrides, the channel.opening log, each of the four status transitions, a pg.change payload round-trip, and the close() path.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 18 pass
 0 fail
 41 expect() calls
Ran 18 tests across 4 files. [131.00ms]

Eighteen passing tests across four files — the seven from step 1 plus three logger tests and eight subscribe tests — completed in about 130 ms. The fence around the new observability is now green.

What we built

We now have a structured, timestamped log line for every event that matters on a Realtime channel: the moment we open it, the SUBSCRIBED handshake, every postgres_changes payload, every status transition, and the explicit close. Because the sink is an injectable function, the same helper can drive a console in development, a file in a reproduction harness, and an in-memory buffer in tests without any code change.

The subscribe helper stays honest by talking to structural interfaces instead of the concrete Supabase types. That means a change in @supabase/supabase-js's internal shape cannot silently break our observability — the test suite catches the mismatch first, and a real socket is never required to prove the behaviour.

Two invariants are now enforceable end to end. The channel.opening event is always emitted before any network call, so a synchronous failure inside the client still leaves a trace. And CHANNEL_ERROR is the only status that logs at error level, which lets us tune alerting on that single line rather than filtering a noisy warning stream.

With this in place, the "dropout" bugs stop being invisible. Every subsequent step can drive traffic through subscribeToTable, read the JSON log lines back, and reason about them like any other event stream — which is exactly the foundation we need before we start injecting latency, closing sockets, or replaying missed events.

Repository

The state of the code after this step: 64a4027

Step 3: Turning Status Transitions into Measurable Dropout Intervals

Step 2 gave us structured log lines for every channel status transition, but the log is a stream of points — nothing in it tells us how long the subscription was actually live, or how long a dropout lasted before recovery. When a support ticket says "we missed events between 14:03 and 14:07", we still have to reconstruct that window by eye.

In this step we bolt a small stateful detector onto the subscribe helper. It remembers the last joinedAt and leftAt timestamps, pairs them into intervals when the channel comes back, and emits three new event types — channel.join, channel.leave, channel.recovered — plus a snapshot API for whatever consumer wants to poll the current state. The transport code is untouched; the detector only reads the same status callback the logger already listens to.

Setup

One new source file and one new test file join the step 2 layout. The detector is a pure function of (status, timestamp) pairs, so it needs no runtime dependencies beyond the Logger interface and the SubscribeStatus union we already export:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts
│   ├── dropout.ts       # new
│   ├── subscribe.ts     # updated — wires detector into subscribeToTable
│   └── index.ts
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts
    ├── subscribe.test.ts
    └── dropout.test.ts  # new

package.json and tsconfig.json are unchanged from step 2. The whole detector fits inside a single closure, so there is no new module boundary to configure and no new type dependency to pin.

Implementation

Start with src/dropout.ts. The public surface is intentionally tiny: a DropoutRecord value type that snapshots the current interval state, a DropoutDetector with observe(status, err?) and snapshot(), and a factory that closes over a channel name, a logger, and an injectable clock:

export interface DropoutRecord {
  channel: string;
  joinedAt: string | null;
  leftAt: string | null;
  sessionMs: number | null;
  dropoutMs: number | null;
  dropoutCount: number;
}

export interface DropoutDetector {
  observe(status: SubscribeStatus, err?: Error): void;
  snapshot(): DropoutRecord;
}

const LEAVE_STATUSES: ReadonlyArray<SubscribeStatus> = [
  "CLOSED",
  "TIMED_OUT",
  "CHANNEL_ERROR",
];

The LEAVE_STATUSES constant is the whole "what counts as a dropout" policy in one place. When we later want to treat CHANNEL_ERROR differently — say, escalate it to a distinct metric — we edit this array, not the status callback and not the tests.

The factory itself keeps five pieces of state inside a closure: the last joinedAt, the last leftAt, the last sessionMs, the last dropoutMs, and a dropoutCount counter. A subscribed flag guards the transitions so that a CHANNEL_ERROR that arrives before we ever hit SUBSCRIBED cannot be misread as a dropout:

export function createDropoutDetector(
  channel: string,
  logger: Logger,
  now: () => Date = () => new Date(),
): DropoutDetector {
  let joinedAt: Date | null = null;
  let leftAt: Date | null = null;
  let sessionMs: number | null = null;
  let dropoutMs: number | null = null;
  let dropoutCount = 0;
  let subscribed = false;

The now parameter is the same trick the logger uses: default to new Date() in production, inject a scripted clock in tests. Every duration the detector reports flows through this single function, so a test that hands over a sequence of ISO strings can assert on exact millisecond values without racing a real timer.

The SUBSCRIBED branch has two shapes. If we have never left before, this is a fresh join and we emit channel.join. If we did have a previous leftAt, this is a recovery, and we compute dropoutMs as the delta between now and that leave timestamp:

  const emitJoin = (at: Date) => {
    logger.info("channel.join", {
      channel,
      joinedAt: at.toISOString(),
    });
  };

  const emitRecovered = (at: Date, previousLeave: Date) => {
    dropoutMs = at.getTime() - previousLeave.getTime();
    logger.info("channel.recovered", {
      channel,
      joinedAt: at.toISOString(),
      dropoutMs,
      dropoutCount,
    });
  };

  const handleSubscribed = () => {
    const at = now();
    if (leftAt) {
      emitRecovered(at, leftAt);
    } else {
      emitJoin(at);
    }
    joinedAt = at;
    subscribed = true;
  };

The two emit helpers stay separate on purpose. channel.join is the "cold start" line — useful when tailing a fresh session — and channel.recovered is the "we came back after N ms of blackout" line that a dashboard actually wants to alarm on. Merging them into one event type would force every downstream consumer to re-implement the "did dropoutMs come along?" branch itself.

The leave branch is the mirror image. It only fires when we were previously subscribed, computes sessionMs from the current joinedAt, records the leave timestamp, bumps the running count, and logs at warn:

  const handleLeave = (status: SubscribeStatus, err?: Error) => {
    if (!subscribed) {
      return;
    }
    const at = now();
    sessionMs = joinedAt ? at.getTime() - joinedAt.getTime() : null;
    leftAt = at;
    subscribed = false;
    dropoutCount += 1;
    logger.warn("channel.leave", {
      channel,
      status,
      leftAt: at.toISOString(),
      sessionMs,
      error: err?.message ?? null,
    });
  };

The subscribed guard is the invariant that keeps the count honest. A CHANNEL_ERROR during the initial handshake — for example, a bad JWT that fails before the socket ever confirms — is a boot failure, not a dropout, and inflating dropoutCount for it would poison every metric downstream. The status callback still logs the error at step 2's error level; the detector simply refuses to count it as an outage.

The dispatch inside the returned object stays flat: SUBSCRIBED goes one way, any member of LEAVE_STATUSES goes the other, everything else is ignored. snapshot() returns a JSON-friendly copy of the current state so callers never hold a reference to the mutable dates:

  return {
    observe(status, err) {
      if (status === "SUBSCRIBED") {
        handleSubscribed();
        return;
      }
      if (LEAVE_STATUSES.includes(status)) {
        handleLeave(status, err);
      }
    },
    snapshot() {
      return {
        channel,
        joinedAt: joinedAt ? joinedAt.toISOString() : null,
        leftAt: leftAt ? leftAt.toISOString() : null,
        sessionMs,
        dropoutMs,
        dropoutCount,
      };
    },
  };
}

Wiring the detector into subscribeToTable is a two-line change: build one per channel, hand it the same clock the helper uses, and call detector.observe(status, err) inside the status callback before the existing logStatus call. The Subscription shape gains a detector field so callers can pull a snapshot without knowing where the detector lives:

export interface Subscription {
  channel: ChannelLike;
  detector: DropoutDetector;
  close(): Promise<void>;
}

// inside subscribeToTable:
const detector = createDropoutDetector(channelName, logger, now);
// ...
.subscribe((status, err) => {
  detector.observe(status, err);
  logStatus(logger, channelName, status, err);
});

Ordering matters: observe runs first so that a downstream sink reading the log stream sees channel.leave (from the detector) before channel.status (from the logger) when a dropout happens. That ordering will make correlation trivial when we start replaying logs in later steps.

The tests in tests/dropout.test.ts cover seven scenarios end to end: a cold SUBSCRIBED emits channel.join, a SUBSCRIBED → CLOSED pair emits channel.leave at warn with the right sessionMs, a full SUBSCRIBED → CLOSED → SUBSCRIBED cycle emits channel.recovered with the correct dropoutMs, a TIMED_OUT after a live session records both the status and the error message, a CHANNEL_ERROR before the first SUBSCRIBED is ignored, snapshot() reflects the running count across two full cycles, and a double-SUBSCRIBED without an intervening leave does not double-emit recovery.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 26 pass
 0 fail
 58 expect() calls
Ran 26 tests across 5 files. [189.00ms]

Twenty-six passing tests across five files — the eighteen from step 2 plus seven new dropout tests and the updated subscribe wiring — completed in about 190 ms. The dropout detector is now fenced in green.

What we built

We now have a first-class model of a subscription's availability on top of the raw status stream. Every time the channel joins, we know when. Every time it drops, we know when and for what reason. Every time it comes back, we know how long the blackout lasted and how many times it has happened this session.

The detector is stateless from the outside — a caller only interacts with observe() and snapshot() — but internally it enforces the invariants that turn a stream of transitions into meaningful intervals. A boot-time failure never inflates the dropout count. A duplicate SUBSCRIBED never produces a phantom recovery. A leave without a prior join is silently ignored.

Because the detector is wired into subscribeToTable, every future step that uses that helper gets the availability signal for free. The Subscription.detector.snapshot() handle is the seed for the next step's alert threshold logic, and the same JSON events feed straight into any log aggregator that already understands channel.status lines.

Most importantly, the "dropout" story is no longer a hunch. When someone asks "did we lose the subscription at 14:03?", we point at a channel.leave line with a precise leftAt and a sessionMs, and — if the channel recovered — a matching channel.recovered with the exact dropoutMs. That is the observable primitive the rest of the article will build reconnection, replay, and alerting on top of.

Repository

The state of the code after this step: 93a9796

Step 4: Scripting Network, JWT, and RLS Dropouts for Deterministic Replay

Step 3 turned the raw status stream into join / leave / recovered intervals, but every test we wrote for it hand-rolled its own sequence of transitions. If we want to reason about why a channel drops — not just that it did — we need named, reusable fixtures for the failure modes that actually happen in production.

In this step we add a scenarios module that encodes the three most common Realtime dropout causes: a short Wi-Fi flap that resubscribes on its own, a JWT that expires mid-session and tears the channel down permanently, and an RLS policy tightening that denies the subscription outright. Each scenario ships as a scripted timeline of (atMs, status, error?) tuples, a mutable clock that lets us replay them without touching real time, an error classifier that maps messages to causes, and a runScenario driver that plays a scenario through the step 3 detector and returns a structured result.

Setup

Two files land in the codebase for step 4, and src/index.ts is extended to re-export the new surface. No new runtime dependencies — every scenario is a pure data structure, and the runner reuses the logger and detector we already have:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts
│   ├── subscribe.ts
│   ├── dropout.ts
│   ├── scenarios.ts       # new
│   └── index.ts           # updated — re-exports the scenarios surface
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts
    ├── subscribe.test.ts
    ├── dropout.test.ts
    └── scenarios.test.ts  # new

package.json, tsconfig.json, and the transport layer are all untouched. The scenarios module deliberately imports only from ./dropout.ts, ./logger.ts, and the SubscribeStatus type — it stays a leaf on the dependency graph so nothing that runs in a real client depends on it.

Implementation

Start with the shared vocabulary at the top of src/scenarios.ts. We name the four cause labels the rest of the article will reason about, define a StatusSink interface that mirrors the shape of the detector's observe, and give a scenario the minimal structure it needs: a name, a cause, a human description, and an ordered list of scripted steps.

export type DropoutCause = "network" | "jwt" | "rls" | "unknown";

export interface StatusSink {
  push(status: SubscribeStatus, err?: Error): void;
}

export interface ScenarioStep {
  atMs: number;
  status: SubscribeStatus;
  error?: Error;
}

export interface Scenario {
  name: string;
  cause: DropoutCause;
  description: string;
  steps: ReadonlyArray<ScenarioStep>;
}

Keeping atMs as a plain offset from a scenario-local origin — not a wall-clock timestamp — is the key choice here. The scenario data becomes portable across machines and time zones, and the mutable clock below can convert every offset into a deterministic ISO string when the runner needs one.

The error classifier is next. It scans the message against three ordered pattern groups and returns "unknown" when nothing matches. Order matters: JWT errors sometimes mention "connection" as collateral damage, so we test JWT and RLS patterns first and only fall through to the network group once the more specific causes have been ruled out.

const JWT_PATTERNS: ReadonlyArray<RegExp> = [/jwt/i, /expired/i, /access token/i];
const RLS_PATTERNS: ReadonlyArray<RegExp> = [/rls/i, /permission/i, /policy/i, /42501/];
const NETWORK_PATTERNS: ReadonlyArray<RegExp> = [
  /socket/i, /disconnect/i, /transport/i, /econnreset/i, /enotfound/i, /network/i,
];

export function classifyError(err?: Error): DropoutCause {
  if (!err) return "unknown";
  const msg = err.message;
  if (JWT_PATTERNS.some((p) => p.test(msg))) return "jwt";
  if (RLS_PATTERNS.some((p) => p.test(msg))) return "rls";
  if (NETWORK_PATTERNS.some((p) => p.test(msg))) return "network";
  return "unknown";
}

Each of the three scenarios is a tiny factory that returns a fresh Scenario value. The network flap simulates a four-second Wi-Fi drop that recovers on its own, the JWT expiry fires a CHANNEL_ERROR an hour into the session and then a CLOSED a hundred milliseconds later, and the RLS scenario fails five seconds after subscription and never recovers.

export function networkFlapScenario(): Scenario {
  return {
    name: "network-flap",
    cause: "network",
    description: "Wi-Fi drops for ~4 seconds and re-associates...",
    steps: [
      { atMs: 0, status: "SUBSCRIBED" },
      { atMs: 3000, status: "CLOSED",
        error: new Error("socket disconnect: transport closed") },
      { atMs: 7000, status: "SUBSCRIBED" },
    ],
  };
}

The two failure-only scenarios follow the same shape but omit the final SUBSCRIBED step, which is exactly the signal runScenario uses to decide whether the channel actually recovered. Both keep a CHANNEL_ERROR immediately followed by a CLOSED so the timeline matches what the Supabase client emits when the server tears a channel down after rejecting it.

export function jwtExpiryScenario(): Scenario {
  return {
    name: "jwt-expiry",
    cause: "jwt",
    description: "The access token TTL elapses mid-session...",
    steps: [
      { atMs: 0, status: "SUBSCRIBED" },
      { atMs: 3_600_000, status: "CHANNEL_ERROR", error: new Error("JWT expired") },
      { atMs: 3_600_100, status: "CLOSED", error: new Error("JWT expired") },
    ],
  };
}

The mutable clock is a two-method interface: now() returns the current instant, and setMs(atMs) advances the clock to startAt + atMs. It gives the runner exact control over what timestamp the detector sees at every step, which is what makes the assertions on sessionMs and dropoutMs in the tests exact rather than approximate.

export function createMutableClock(
  startAt: Date = new Date("2026-07-22T06:15:00.000Z"),
): MutableClock {
  const startMs = startAt.getTime();
  let currentMs = startMs;
  return {
    now: () => new Date(currentMs),
    setMs: (atMs: number) => { currentMs = startMs + atMs; },
  };
}

The runScenario driver ties everything together. It creates a fresh logger with a captured event buffer, spins up a mutable clock, hands both to a new dropout detector, then walks the scripted steps — advancing the clock before each observe so the detector's own now() returns the scripted instant.

export function runScenario(scenario: Scenario, options: RunScenarioOptions = {}): ScenarioResult {
  const channel = options.channel ?? "pg:public.todos:*";
  const captured: LogEvent[] = [];
  const logger = options.logger ?? createLogger((event) => { captured.push(event); });
  const clock = createMutableClock(options.startAt);
  const detector = createDropoutDetector(channel, logger, clock.now);

  for (const step of scenario.steps) {
    clock.setMs(step.atMs);
    detector.observe(step.status, step.error);
  }

  const snapshot = detector.snapshot();
  return {
    scenario: scenario.name,
    cause: scenario.cause,
    detectedCause: classifyError(lastErrorOf(scenario)),
    recovered: didRecover(scenario, snapshot),
    dropoutCount: snapshot.dropoutCount,
    dropoutMs: snapshot.dropoutMs,
    sessionMs: snapshot.sessionMs,
    snapshot,
    events: captured,
  };
}

didRecover returns true only when the final step is a SUBSCRIBED and the snapshot has a non-null dropoutMs. That second half of the predicate matters: a scenario whose only step is SUBSCRIBED is a cold start, not a recovery, and we do not want it to count as one. lastErrorOf walks the steps backwards and returns the newest error it finds, which is the message the classifier should key on when a scenario surfaces multiple errors in a row.

The tests in tests/scenarios.test.ts cover every branch we care about. classifyError gets a table-driven test per cause label plus an unknown fallback. CLASSIC_SCENARIOS is asserted to contain exactly the three causes, with unique names, non-empty descriptions, and a SUBSCRIBED opening step. Each of the three scenarios then gets its own describe block: the network flap must recover with dropoutMs === 4000 and sessionMs === 3000, the JWT expiry must produce a leave whose payload contains "JWT" and no recovered event, and the RLS scenario must leave with a "permission denied" error and never recover. Two integration tests confirm that applyScenario faithfully forwards steps to any sink and that piping a scenario directly through a real detector produces the same snapshot numbers the runner does.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 43 pass
 0 fail
 107 expect() calls
Ran 43 tests across 6 files. [267.00ms]

Forty-three passing tests across six files — the twenty-six from step 3 plus seventeen new scenario and classifier tests — finish in about 267 milliseconds. Every scenario is green, and the detector's numbers match the scripted timeline to the millisecond.

What we built

The dropout detector from step 3 now has a library of named, deterministic failure modes to observe. networkFlapScenario, jwtExpiryScenario, and rlsPolicyChangeScenario each script a realistic Realtime failure as a sequence of (atMs, status, error?) tuples, and CLASSIC_SCENARIOS gathers all three into a single roster the rest of the article can iterate over.

The classifyError function turns the raw error message on a channel.leave event into one of four cause labels — network, jwt, rls, or unknown — using ordered pattern groups that resolve the ambiguous cases correctly. Because it operates on the message alone, we can point it at real production logs later without any change of shape.

createMutableClock and runScenario together give us a fully deterministic replay harness. The mutable clock hands the detector exact ISO timestamps on demand, so sessionMs and dropoutMs come out as fixed integers rather than jittery millisecond ranges. The runner returns a ScenarioResult that pairs the snapshot with the classified cause and the captured event stream, which is the shape the next step will feed into recovery-strategy tests.

Most importantly, we now have a vocabulary for why a dropout happened, not just that one did. When a later step wants to test a JWT-refresh strategy it will replay jwtExpiryScenario; when it wants to test a network-jitter backoff it will replay networkFlapScenario. The article's cause-and-effect chain is now scriptable end to end.

Repository

The state of the code after this step: 075a190

Step 5: Catching Silent Channel Deaths with a Data-Plane Heartbeat

Step 3 gave us a detector that reacts to the Supabase client's own SUBSCRIBED / CLOSED / CHANNEL_ERROR callbacks, and step 4 scripted the three classic dropout causes. Between them, every failure we can currently catch is one the SDK has already noticed and told us about. The nastier class of failure — the one that hurts the most in production — is the one the SDK never surfaces at all: the socket is technically alive, the last status we saw is still SUBSCRIBED, but no postgres_changes payloads have arrived for minutes.

This step adds a heartbeat monitor that treats each Postgres change payload as a pulse on the data plane, tracks the age of the last pulse against a configurable silence threshold, and emits a distinct channel.silent-death log event the first time that threshold is crossed. It also emits channel.pulse-resumed when payloads start flowing again, and it plugs into subscribeToTable as an opt-in heartbeat option so existing call sites keep the exact behaviour they had after step 4.

Setup

Two new files land in codebase/ this step, and two existing files pick up a small integration diff. No new runtime dependencies — the monitor is pure TypeScript over Date, and the tests reuse the scripted-clock pattern from steps 3 and 4:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts
│   ├── subscribe.ts        # updated — accepts a heartbeat option, pulses on every pg.change
│   ├── dropout.ts
│   ├── scenarios.ts
│   ├── heartbeat.ts        # new
│   └── index.ts            # updated — re-exports the heartbeat surface
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts
    ├── subscribe.test.ts   # updated — three integration tests for the heartbeat wiring
    ├── dropout.test.ts
    ├── scenarios.test.ts
    └── heartbeat.test.ts   # new

package.json and tsconfig.json are untouched. The heartbeat module deliberately depends only on the logger — it knows nothing about channels, sockets, or the Supabase SDK — which is what lets it stay a leaf on the dependency graph and, later, be reused for any other stream that needs a data-plane liveness signal.

Implementation

The public shape of the monitor lives at the top of src/heartbeat.ts. A snapshot always reports the last pulse timestamp, the age of that pulse, whether we are currently in a silent episode, when that silence started, and running counters for both silent deaths and resumes. Exposing counters — not just a boolean — is what makes silentDeathCount and resumedCount assertable in tests and observable in dashboards.

export interface HeartbeatSnapshot {
  channel: string;
  lastPulseAt: string | null;
  ageMs: number | null;
  silent: boolean;
  silentSinceMs: number | null;
  silentDeathCount: number;
  resumedCount: number;
}

export interface HeartbeatMonitorOptions {
  channel: string;
  logger: Logger;
  intervalMs?: number;
  silenceThresholdMs?: number;
  now?: () => Date;
}

The options accept a heartbeat interval and a silence threshold separately. The interval is the cadence the caller intends to poll at; the threshold is how long we wait before we declare silence. Making the threshold default to 1.5 * intervalMs gives us one grace period for late payloads without flapping, which matches the "one missed heartbeat is fine, two is not" heuristic most alerting systems already use.

const DEFAULT_INTERVAL_MS = 30_000;
const DEFAULT_TOLERANCE_MULTIPLIER = 1.5;

const intervalMs = options.intervalMs ?? DEFAULT_INTERVAL_MS;
const silenceThresholdMs =
  options.silenceThresholdMs ?? Math.round(intervalMs * DEFAULT_TOLERANCE_MULTIPLIER);

Internal state is four variables and one helper: lastPulseAt, silent, silentSinceMs, and the two counters, plus an ageAt helper that returns the millisecond distance between a given instant and the last pulse. The helper returns null when there is no prior pulse, which is what makes check() before start() a safe no-op instead of a false-positive silent-death.

const ageAt = (at: Date): number | null => {
  if (!lastPulseAt) {
    return null;
  }
  return at.getTime() - lastPulseAt.getTime();
};

The two emit helpers keep the state transitions and the log calls in one place. emitSilentDeath flips silent to true, records the age at which we crossed the threshold, bumps silentDeathCount, and logs at error level with every field an on-call operator would want. emitResumed mirrors it in the other direction and logs at info, since a resumption is good news that still needs a timeline entry.

const emitSilentDeath = (ageMs: number) => {
  silent = true;
  silentSinceMs = ageMs;
  silentDeathCount += 1;
  logger.error("channel.silent-death", {
    channel, lastPulseAt: lastPulseAt?.toISOString() ?? null,
    ageMs, intervalMs, silenceThresholdMs, silentDeathCount,
  });
};

const emitResumed = (at: Date, silentForMs: number) => {
  silent = false;
  silentSinceMs = null;
  resumedCount += 1;
  logger.info("channel.pulse-resumed", {
    channel, pulseAt: at.toISOString(), silentForMs, resumedCount,
  });
};

The exported monitor is a five-method object. start records the initial pulse when the channel goes SUBSCRIBED; pulse records a data-plane payload and, if we were silent, emits channel.pulse-resumed; check computes the current age and, if it has crossed the threshold and we are not already silent, emits channel.silent-death; reset clears state when the channel leaves; and snapshot gives read-only access without mutating counters.

return {
  intervalMs, silenceThresholdMs,
  start() {
    lastPulseAt = now();
    silent = false;
    silentSinceMs = null;
  },
  pulse() { recordPulse(); },
  check() {
    const at = now();
    const age = ageAt(at);
    if (age !== null && !silent && age > silenceThresholdMs) {
      emitSilentDeath(age);
    }
    return buildSnapshot(at);
  },
  reset() {
    lastPulseAt = null;
    silent = false;
    silentSinceMs = null;
  },
  snapshot() { return buildSnapshot(now()); },
};

The !silent guard inside check is the single most important line in this file. Without it, every subsequent poll during a long silent episode would re-emit channel.silent-death and turn one real incident into a torrent of duplicate alerts. With it, we get exactly one silent-death per episode, matched by exactly one pulse-resumed when the data plane comes back.

Wiring the monitor into subscribeToTable is a small, deliberately additive change. A new HeartbeatOptions type surfaces intervalMs and silenceThresholdMs on the subscription options, a buildHeartbeatMonitor helper returns either a real monitor or null, the postgres_changes callback calls heartbeat?.pulse() before it logs, and the status callback routes SUBSCRIBED to heartbeat.start() and everything else to heartbeat.reset().

export interface HeartbeatOptions {
  intervalMs?: number;
  silenceThresholdMs?: number;
}

function handleHeartbeatOnStatus(
  heartbeat: HeartbeatMonitor | null,
  status: SubscribeStatus,
): void {
  if (!heartbeat) {
    return;
  }
  if (status === "SUBSCRIBED") {
    heartbeat.start();
    return;
  }
  heartbeat.reset();
}

Because heartbeat is null when the caller omits the option, every step 4 test still passes without modification — the new behaviour is strictly additive. The Subscription return type gains a heartbeat: HeartbeatMonitor | null field so callers can hold the reference, call check() on their preferred cadence, and read snapshot() for dashboards.

The test suite in tests/heartbeat.test.ts covers the whole state machine: snapshot-before-start reports the empty state, the default silence threshold is 1.5 * intervalMs, start records the initial pulse, check past the threshold logs channel.silent-death exactly once, repeated checks during the same silence do not re-emit, a pulse after silence logs channel.pulse-resumed with silentForMs, a pulse while healthy stays quiet, separate silence episodes both increment the counter, reset wipes the state, and check before start is a no-op. Three integration tests in tests/subscribe.test.ts then confirm the wiring: no option means subscription.heartbeat === null, the monitor fires when no pulses arrive past the threshold, a postgres_changes payload resets the silence and emits channel.pulse-resumed, and a CLOSED status clears lastPulseAt so a stale channel cannot look silent forever.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 57 pass
 0 fail
 148 expect() calls
Ran 57 tests across 7 files. [170.00ms]

Fifty-seven passing tests across seven files — the forty-three from step 4 plus fourteen new tests split between the heartbeat unit suite and the subscribeToTable integration cases — finish in about 170 milliseconds. Every silent-death path, every resume path, and every heartbeat-off default is green.

What we built

The system now has a data-plane liveness signal that the Supabase SDK does not provide out of the box. createHeartbeatMonitor tracks the age of the last postgres_changes payload, declares a silent death when the age crosses a configurable threshold, and declares a resume when payloads start flowing again — each transition logged as its own structured event with counters that downstream dashboards can page on.

subscribeToTable is now the single entry point for a fully-observable Realtime channel. Callers who want the old behaviour keep passing the same options they always have; callers who want the new behaviour add heartbeat: { intervalMs, silenceThresholdMs } and hold on to the returned subscription.heartbeat for polling. The monitor is opt-in, so no existing subscriber pays for a feature it did not ask for.

Between the step 3 dropout detector and this heartbeat monitor, our channel now emits five distinct classes of event: channel.join, channel.leave, channel.recovered, channel.silent-death, and channel.pulse-resumed. Each event names its own cause and carries the numbers — sessionMs, dropoutMs, ageMs, silentForMs — an on-call engineer needs to reason about what happened without opening a debugger.

The next step will feed real production-shaped traffic through this instrumented channel and demonstrate an alert that fires within one silence window of a real stall, closing the loop from "the SDK told us nothing was wrong" to "we knew within 45 seconds."

Repository

The state of the code after this step: 0ee7c31

Step 6: Reconnecting with Exponential Backoff, Full Jitter, and an Exhaustion Guard

After step 5 the channel could tell us when it had silently died, but it still could not heal itself: a CLOSED, TIMED_OUT, or CHANNEL_ERROR status just sat there, waiting for a human or a wrapping process to call subscribe again. Any real production client has to close that loop automatically, and it has to do so without hammering the server every millisecond and without pretending a permanently misconfigured channel is going to come back on attempt number ten thousand.

This step introduces two new modules — backoff.ts and reconnect.ts — that together give subscribeToTable an opt-in reconnection loop. The scheduler doubles the wait between attempts up to a cap, adds full jitter so simultaneous drops do not synchronise into a thundering herd, and returns null once a configurable maxAttempts ceiling is hit. The controller sits on top, listens for leave statuses, schedules the timer, fires the resubscribe, resets on SUBSCRIBED, and emits four new structured log events an operator can plot on a timeline.

Setup

Two new source files and two new test files land under codebase/ this step, and subscribe.ts, tests/subscribe.test.ts, and index.ts pick up small integration diffs. No new runtime dependencies — the scheduler is pure TypeScript over Math.pow, and the controller takes an injectable schedule function so tests can drive time without setTimeout:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts
│   ├── dropout.ts
│   ├── scenarios.ts
│   ├── heartbeat.ts
│   ├── backoff.ts         # new — pure exponential + jitter scheduler
│   ├── reconnect.ts       # new — controller that wires the scheduler to statuses
│   ├── subscribe.ts       # updated — accepts a reconnect option, exposes subscription.reconnect
│   └── index.ts           # updated — re-exports the backoff + reconnect surface
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts
    ├── subscribe.test.ts  # updated — four integration tests for the reconnect wiring
    ├── dropout.test.ts
    ├── scenarios.test.ts
    ├── heartbeat.test.ts
    ├── backoff.test.ts    # new
    └── reconnect.test.ts  # new

package.json and tsconfig.json are untouched. backoff.ts has zero project imports on purpose — it is a leaf that knows nothing about Supabase, channels, or timers — which is what lets us test its arithmetic in isolation and reuse it later for any other retry-with-backoff surface.

Implementation

The scheduler in src/backoff.ts is a small state machine over four knobs — baseMs, maxMs, maxAttempts, factor — plus a jitter mode and an injectable random. The public surface exposes next(), reset(), snapshot(), and read-only echoes of the configuration so tests and dashboards can assert on the exact behaviour without reaching into private state.

export type JitterMode = "full" | "none";

export interface BackoffOptions {
  baseMs?: number;
  maxMs?: number;
  maxAttempts?: number;
  factor?: number;
  jitter?: JitterMode;
  random?: () => number;
}

export interface BackoffSnapshot {
  attempts: number;
  totalDelayMs: number;
  lastDelayMs: number | null;
  exhausted: boolean;
}

Defaults land at baseMs=1000, maxMs=30_000, maxAttempts=8, and factor=2, which give the classic 1s → 2s → 4s → 8s → 16s → 30s → 30s → 30s ceiling. Full jitter is on by default because the AWS Architecture Blog's canonical "Exponential Backoff And Jitter" analysis showed full jitter dominates every alternative when many clients might drop at the same instant. Injecting random is what makes the jittered path deterministic under test.

const cappedExponential = (attempt: number): number => {
  const raw = baseMs * Math.pow(factor, attempt);
  return Math.min(raw, maxMs);
};

const computeDelay = (attempt: number): number => {
  const cap = cappedExponential(attempt);
  if (jitter === "none") {
    return cap;
  }
  return Math.floor(random() * cap);
};

next() is the only method that mutates state. If attempts has already reached maxAttempts it returns null; otherwise it computes the delay, records lastDelayMs, adds to totalDelayMs, bumps attempts, and hands the delay back. That single return-value contract — number for "wait this long" and null for "give up" — is what keeps the calling controller free of branching on internal counters.

next() {
  if (attempts >= maxAttempts) {
    return null;
  }
  const delay = computeDelay(attempts);
  lastDelayMs = delay;
  totalDelayMs += delay;
  attempts += 1;
  return delay;
},

The controller in src/reconnect.ts translates channel statuses into scheduler calls. Its options extend BackoffOptions with a channel name, a logger, a resubscribe callback that the caller supplies, and an injectable schedule that defaults to setTimeout. The controller stays completely decoupled from the Supabase SDK — it never touches a channel object directly, it just says "please subscribe again" when the timer fires.

export interface ReconnectOptions extends BackoffOptions {
  channel: string;
  logger: Logger;
  resubscribe: () => void;
  schedule?: ScheduleFn;
}

const LEAVE_STATUSES: ReadonlyArray<SubscribeStatus> = [
  "CLOSED",
  "TIMED_OUT",
  "CHANNEL_ERROR",
];

The scheduling core is a single function protected by two guards. cancelled is a hard stop set by close(); pending is the "already have a timer in flight" flag that keeps a burst of three leave statuses in the same tick from stacking three timers. If the scheduler returns null, we emit channel.reconnect-exhausted at error level and return without scheduling anything further — the loop is over.

const scheduleReconnect = () => {
  if (cancelled || pending) {
    return;
  }
  const delay = scheduler.next();
  if (delay === null) {
    emitExhausted();
    return;
  }
  const attempt = scheduler.snapshot().attempts;
  emitScheduled(attempt, delay);
  pending = true;
  schedule(() => fireTimer(attempt), delay);
};

Status routing is intentionally tiny. SUBSCRIBED clears the counter and, if we had been retrying, emits channel.reconnect-recovered with the number of attempts it took — the metric an on-call engineer actually wants when they ask "did that channel heal itself?". Every leave status funnels into the same scheduleReconnect, which is what keeps the four event types (scheduled, attempt, recovered, exhausted) in a one-to-one mapping with the four things the controller can do.

const handleSubscribed = () => {
  const snap = scheduler.snapshot();
  if (snap.attempts > 0) {
    emitRecovered(snap.attempts);
  }
  scheduler.reset();
};

const handleStatus = (status: SubscribeStatus) => {
  if (status === "SUBSCRIBED") {
    handleSubscribed();
    return;
  }
  if (LEAVE_STATUSES.includes(status)) {
    scheduleReconnect();
  }
};

Wiring into subscribeToTable follows the same additive pattern step 5 used for the heartbeat. A new ReconnectSubscribeOptions type surfaces the backoff knobs on the subscription options, a buildReconnectController helper returns either a real controller or null, and the status callback forwards every status into reconnect?.onStatus(status). The subscription.close() method calls reconnect?.cancel() first so a pending timer that fires after the channel is gone cannot resurrect it.

export interface ReconnectSubscribeOptions extends BackoffOptions {
  schedule?: ScheduleFn;
}

let statusCallback: (status: SubscribeStatus, err?: Error) => void = () => {};
const reconnect = buildReconnectController(
  channelName,
  options.reconnect,
  logger,
  () => channel.subscribe(statusCallback),
);

The one-line subtlety worth calling out is the let binding for statusCallback. The controller's resubscribe closure has to call channel.subscribe(statusCallback) with the same callback the initial subscribe used, so future reconnect cycles keep observing statuses. Declaring the callback with let before the controller is built, then assigning it before channel.subscribe, keeps that reference identity intact without a mutable wrapper object.

The test suites cover both layers. tests/backoff.test.ts locks down the arithmetic — default constants, exact exponential progression with jitter: "none", jittered delays under an injected random, the null return after maxAttempts, snapshot counters, reset() behaviour, and a bound-check that no full-jitter delay ever exceeds its capped exponential. tests/reconnect.test.ts covers the state machine — CLOSED schedules and logs channel.reconnect-scheduled at warn, the timer firing emits channel.reconnect-attempt at info and calls resubscribe, SUBSCRIBED after retries emits channel.reconnect-recovered and resets the counter, SUBSCRIBED without prior leaves stays silent, exhausting maxAttempts emits channel.reconnect-exhausted at error and stops scheduling, all three leave statuses can trigger a reconnect, injected random drives the jittered delay, cancel() prevents pending timers from firing, and back-to-back leave statuses only schedule one timer. Four integration tests in tests/subscribe.test.ts then confirm the wiring: without a reconnect option subscription.reconnect === null, a CLOSED schedules the resubscribe and a follow-up SUBSCRIBED clears the counter, maxAttempts exhaustion stops further scheduling, and close() cancels a pending timer so a late fire is a no-op.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 77 pass
 0 fail
 224 expect() calls
Ran 77 tests across 9 files. [176.00ms]

Seventy-seven passing tests across nine files — the fifty-seven from step 5 plus twenty new tests split between the pure backoff suite, the reconnect controller state machine, and the four subscribeToTable integration cases — finish in about 176 milliseconds. Every reconnect-scheduled, reconnect-attempt, reconnect-recovered, and reconnect-exhausted path is green, and no existing step-5 assertion needed to change.

What we built

The client now heals itself from transient dropouts without any external supervisor. When CLOSED, TIMED_OUT, or CHANNEL_ERROR arrives, the controller schedules a jittered exponential-backoff timer, emits channel.reconnect-scheduled, and — when the timer fires — calls the exact same status callback the initial subscribe used, so the dropout detector, the heartbeat monitor, and the reconnect controller all continue to observe the new lifecycle together.

The maxAttempts guard converts "we cannot connect right now" into a distinct, observable outcome. Instead of an unbounded loop hiding a permanent misconfiguration — a rotated JWT that will never come back, an RLS policy that has locked us out — the loop stops, channel.reconnect-exhausted fires at error level with the total delay budget spent, and the operator gets a page that names the exact channel and the exact ceiling that was crossed.

Full jitter is the specific choice that keeps a cluster of clients from re-synchronising every time the fleet drops together. Without jitter, a thousand clients that all lost their socket at the same second would all try to reconnect at exactly baseMs later, then exactly 2 * baseMs later, and so on — the retry loop itself becomes the outage. With Math.floor(random() * cap), each client's next attempt is uniformly spread across its capped window, and the reconnection storm smears out into a manageable curve.

The channel now emits nine distinct event classes end to end: channel.join, channel.leave, channel.recovered, channel.silent-death, channel.pulse-resumed, channel.reconnect-scheduled, channel.reconnect-attempt, channel.reconnect-recovered, and channel.reconnect-exhausted. Each one names its own cause and carries the numbers an operator needs — sessionMs, dropoutMs, ageMs, silentForMs, attempt, delayMs, totalDelayMs, attempts — to reason about what happened without opening a debugger.

Repository

The state of the code after this step: 2a696d3

Step 7: Closing the Dropout Gap with a Since-Timestamp REST Catch-Up

After step 6 the channel can heal itself from any transient dropout: the exponential-backoff controller waits, resubscribes, emits channel.reconnect-recovered, and the socket is live again. What it cannot do is tell you what happened to the rows that were committed while the socket was down. Realtime is a live stream, not a durable queue — every INSERT and UPDATE that landed in the gap between the last delivered event and the new subscription is simply missing from the stream, and downstream state silently drifts out of sync with the database.

This step adds a catchup module that fixes exactly that gap. It observes the commit_timestamp of every live payload to keep a running high-water mark, exposes a trigger("reconnect") entry point the reconnect controller calls right after channel.reconnect-recovered, and issues a caller-supplied REST fetch for every row committed since that mark. Recovered rows are then re-emitted through the same pg.change pipeline used for live events, so subscribers do not have to distinguish "live" from "recovered" — they just see events, in order, and the count they receive matches the count Postgres actually committed.

Setup

One new source file and one new test file land under codebase/ this step; subscribe.ts, reconnect.ts, index.ts, and tests/subscribe.test.ts pick up small integration diffs, and tests/reconnect.test.ts gains one test for the new onRecovered callback. No new runtime dependencies — the coordinator is pure TypeScript over Date plus an injected fetch closure, so the caller decides whether to talk to PostgREST, supabase-js, or a stubbed harness under test:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts
│   ├── dropout.ts
│   ├── scenarios.ts
│   ├── heartbeat.ts
│   ├── backoff.ts
│   ├── reconnect.ts       # updated — accepts onRecovered callback fired after channel.reconnect-recovered
│   ├── catchup.ts         # new — since-timestamp coordinator with fetch + emit contracts
│   ├── subscribe.ts       # updated — wires the coordinator into the change pipeline and the reconnect hook
│   └── index.ts           # updated — re-exports the catchup surface
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts
    ├── dropout.test.ts
    ├── scenarios.test.ts
    ├── heartbeat.test.ts
    ├── backoff.test.ts
    ├── reconnect.test.ts  # updated — onRecovered callback test
    ├── subscribe.test.ts  # updated — four integration tests for the catchup wiring
    └── catchup.test.ts    # new — nine unit tests for the coordinator

package.json and tsconfig.json are untouched. catchup.ts has zero project imports other than the Logger and PostgresChangePayload types — no Supabase SDK, no HTTP client, no setTimeout. The fetch and now seams are what let the whole file be unit-tested with deterministic strings.

Implementation

The coordinator in src/catchup.ts is built around a single invariant: lastSeenCommitTs must be the highest commit_timestamp we have ever observed on this channel, whether the payload arrived live or through a catch-up fetch. The advanceLastSeen helper is the only place that mutates it, and it refuses to move backward — an out-of-order live event with an older timestamp is ignored, which prevents a stale row from re-opening a gap we already closed.

export interface CatchupOptions {
  channel: string;
  logger: Logger;
  fetch: CatchupFetcher;
  emit: CatchupEmit;
  schema?: string;
  table?: string;
  initialSince?: string;
  now?: () => Date;
}

export interface CatchupSnapshot {
  channel: string;
  lastSeenCommitTs: string | null;
  lastCatchupAt: string | null;
  runCount: number;
  totalRowsRecovered: number;
  lastRunRowCount: number;
  lastRunDurationMs: number | null;
  failureCount: number;
}

The snapshot is what an operator actually consumes — runCount, totalRowsRecovered, and failureCount become obvious dashboard counters, and lastRunDurationMs is the number that tells you whether a catch-up window is starting to blow past your reconnect budget. Everything is derived from monotonic increments inside trigger, so a snapshot taken mid-flight can never observe a torn write.

const advanceLastSeen = (ts: string | undefined | null): void => {
  if (!ts) {
    return;
  }
  if (!lastSeenCommitTs || ts > lastSeenCommitTs) {
    lastSeenCommitTs = ts;
  }
};

Comparing ISO 8601 strings with > works here because Postgres emits commit_timestamp in a fixed lexicographically-sortable UTC format. That saves a Date parse on every payload — for a busy channel that fires thousands of events per minute, that is a real cost — and it keeps the coordinator dependency-free.

The trigger method is the entry point the reconnect controller calls. It skips the fetch entirely if we have no baseline (a channel that recovered before its first live event has nothing to catch up on, and we log channel.catchup-skipped so an operator can see the decision), records startedAt, awaits the caller's fetch, and hands the rows to finalizeSuccess. A thrown error from the fetch is caught once, converted into a channel.catchup-failed log at error level, and returned as a snapshot — trigger never re-throws, so a bad REST endpoint cannot poison the reconnect controller's own state machine.

async trigger(reason: CatchupTriggerReason = "manual") {
  if (!lastSeenCommitTs) {
    emitSkipped("no-baseline", reason);
    return buildSnapshot();
  }
  const since = lastSeenCommitTs;
  const startedAt = now();
  emitStarted(since, reason);
  let rows: CatchupRow[];
  try {
    rows = await runFetch(since);
  } catch (err) {
    failureCount += 1;
    const durationMs = now().getTime() - startedAt.getTime();
    emitFailed(err, durationMs);
    return buildSnapshot();
  }
  finalizeSuccess(rows, startedAt);
  return buildSnapshot();
},

finalizeSuccess is where the ordering discipline lives. It records lastCatchupAt, computes lastRunDurationMs, bumps the counters, and only then calls applyRows. applyRows walks the fetched rows in the order the caller returned them, calls emit for each one, and lets advanceLastSeen push the high-water mark forward as it goes — so if the fetch returns rows out of order, the mark still tracks the largest timestamp, and a later reconnect will not re-fetch anything we already handled.

const emitRow = (row: CatchupRow): void => {
  const payload: PostgresChangePayload = {
    eventType: row.eventType ?? "UPDATE",
    schema,
    commit_timestamp: row.commit_timestamp,
    new: row.new ?? null,
    old: row.old ?? null,
  };
  if (table) {
    payload.table = table;
  }
  emit(payload);
};

Wiring into subscribeToTable follows the additive pattern the previous three steps established. A new CatchupSubscribeOptions type carries the fetch closure and an optional initialSince, a buildCatchupCoordinator helper returns either a real coordinator or null, and the payload handler calls catchup?.observePayload(payload) on every live change so the high-water mark tracks live traffic without any extra callback from the caller. initialSince is threaded through observePayload so callers that already know a baseline (say, from a restarted process that persisted a checkpoint) can prime the coordinator without a synthetic first payload.

catchup = buildCatchupCoordinator(
  channelName,
  schema,
  table,
  options.catchup,
  logger,
  handlePayload,
  now,
);

let statusCallback: (status: SubscribeStatus, err?: Error) => void = () => {};
const reconnect = buildReconnectController(
  channelName,
  options.reconnect,
  logger,
  () => channel.subscribe(statusCallback),
  () => triggerCatchupAfterRecovery(catchup, logger, channelName),
);

The reconnect controller in src/reconnect.ts gains a single optional onRecovered(attempts) callback that fires immediately after channel.reconnect-recovered. That is the exact seam step 7 needs: the controller already knows when the channel is healthy again, so it hands the coordinator a "you can safely fetch now" signal without either module knowing about the other's internals. triggerCatchupAfterRecovery calls catchup.trigger("reconnect") and attaches a .catch so an unexpected rejection lands as channel.catchup-failed at error level rather than an unhandled promise.

function triggerCatchupAfterRecovery(
  catchup: CatchupCoordinator | null,
  logger: Logger,
  channelName: string,
): void {
  if (!catchup) {
    return;
  }
  catchup.trigger("reconnect").catch((err: unknown) => {
    const message = err instanceof Error ? err.message : String(err);
    logger.error("channel.catchup-failed", {
      channel: channelName,
      error: message,
      stage: "trigger",
    });
  });
}

Two behavioural details are worth calling out. First, the coordinator is intentionally not triggered on the initial SUBSCRIBED — the reconnect controller only fires onRecovered when it has actually retried, so a first-time subscribe stays quiet and no fetch runs against a channel that has no gap to close. Second, handlePayload calls catchup?.observePayload before it logs pg.change, so a recovered row that flows back through emit still advances the high-water mark exactly once and cannot trigger a second catch-up on the next reconnect.

The tests cover both layers. tests/catchup.test.ts locks down the coordinator arithmetic in nine cases: trigger without a baseline skips and logs channel.catchup-skipped, observePayload advances monotonically and ignores older timestamps, initialSince primes the baseline, recovered rows re-emit as PostgresChangePayload values with the coordinator's schema and table, duration is measured against an injected clock, channel.catchup-started fires at info before the fetch and channel.catchup-complete fires after, a thrown fetch increments failureCount at error level without re-throwing, back-to-back triggers use the highest timestamp seen from streaming payloads, trigger() with no argument defaults to reason: "manual", and an empty result set still emits complete while leaving lastSeenCommitTs untouched. tests/subscribe.test.ts adds four integration cases: subscription.catchup === null when the option is absent, a full SUBSCRIBED → live event → CLOSED → reconnect → SUBSCRIBED cycle drives a fetch with the correct since and re-emits the missed row through pg.change, the initial SUBSCRIBED does not fire a catch-up, and initialSince becomes the baseline when no live event has arrived. tests/reconnect.test.ts gains one case that proves onRecovered fires with the attempt count exactly once per recovery.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 92 pass
 0 fail
 267 expect() calls
Ran 92 tests across 10 files. [177.00ms]

Ninety-two passing tests across ten files — the seventy-seven from step 6 plus fifteen new tests split between the nine catchup coordinator cases, one new reconnect callback case, and five new subscribe integration cases — finish in about 177 milliseconds. Every catch-up-skipped, catch-up-started, catch-up-complete, and catch-up-failed path is green, and no existing step-6 assertion needed to change.

What we built

The client now closes the last outstanding gap in its dropout story. When the reconnect controller announces channel.reconnect-recovered, the coordinator issues a REST fetch for every row committed since the highest commit_timestamp the channel had ever observed, re-emits those rows through the same pg.change pipeline the live socket uses, and advances the high-water mark so the next dropout will pick up exactly where this one ended.

The channel.catchup-skipped, channel.catchup-started, channel.catchup-complete, and channel.catchup-failed events give an operator a full ledger of every recovery attempt: which channel ran, from what since timestamp, how many rows came back, how long the fetch took, and whether it failed. Combined with the reconnect controller's own four events, a single dropout now produces a readable timeline — leave → reconnect-scheduled → reconnect-attempt → reconnect-recovered → catchup-started → catchup-complete — that an on-call engineer can plot without opening a debugger.

The fetch seam is what keeps the coordinator honest as a library. The caller decides whether to talk to PostgREST directly, go through supabase-js, or route through a private HTTP endpoint that already applies row-level authorization — the coordinator does not care, because it never constructs a URL or a query. That also means the RLS story stays exactly the same as for the live subscription: whatever policy governs postgres_changes also governs the REST catch-up, as long as the caller passes an authenticated client into their fetch closure.

Downstream state can now converge to the database's state after any transient dropout without any consumer-side reconciliation code. The channel emits nine event kinds and one new payload class end to end, and the four catch-up events plus the nine PostgresChangePayload fields (eventType, schema, table, commit_timestamp, new, old, plus the three snapshot counters an operator reads) carry every fact a reader needs to explain, after the fact, exactly what happened during the outage.

Repository

The state of the code after this step: 5413f19

Step 8: Turning the Log Stream into Metrics, Alerts, and a Prometheus Endpoint

By the end of step 7 the client emitted a rich, structured event stream — channel.join, channel.leave, channel.recovered, the four reconnect events, channel.silent-death, channel.pulse-resumed, and the four catch-up events — that fully described a dropout end to end. What it did not have was a way for anyone other than a human tailing a JSON log to know something was wrong. There were no counters, no gauges, no threshold that flipped a status page, and no webhook that woke an on-call engineer at 03:00 when a channel had quietly stopped recovering.

This step adds the last mile of production observability: a MetricsCollector that folds every log event into per-channel counters and gauges, a prometheus() renderer that spits out the standard # HELP / # TYPE text format an existing scraper can consume, and an AlertEvaluator with four default rules — reconnect-exhausted, catchup-failure, silent-death, and flapping-channel — that watches the snapshot on every tick and dispatches structured AlertPayload values to a caller-supplied sink. An createObservability() bundle wires all three together behind a single object so an application only has to install one thing.

Setup

Three new source files land under codebase/src/ this step, three new test files land under codebase/tests/, and src/index.ts picks up the re-exports for the new public surface. No new runtime dependencies — everything is stdlib TypeScript over the existing Logger and LogEvent types, so the metrics and alerts modules stay dependency-free and can run in a Node cron, a Bun worker, a browser tab, or a Cloudflare Worker without change:

codebase/
├── src/
│   ├── config.ts
│   ├── client.ts
│   ├── logger.ts
│   ├── dropout.ts
│   ├── scenarios.ts
│   ├── heartbeat.ts
│   ├── backoff.ts
│   ├── reconnect.ts
│   ├── catchup.ts
│   ├── subscribe.ts
│   ├── metrics.ts        # new — per-channel counters + gauges + Prometheus renderer
│   ├── alerts.ts         # new — rule engine with cooldowns and a pluggable sink
│   ├── observability.ts  # new — bundles logger + metrics + alerts behind one object
│   └── index.ts          # updated — re-exports the observability surface
└── tests/
    ├── config.test.ts
    ├── client.test.ts
    ├── logger.test.ts
    ├── dropout.test.ts
    ├── scenarios.test.ts
    ├── heartbeat.test.ts
    ├── backoff.test.ts
    ├── reconnect.test.ts
    ├── subscribe.test.ts
    ├── catchup.test.ts
    ├── metrics.test.ts        # new — 15 tests for the collector + Prometheus renderer
    ├── alerts.test.ts         # new — 14 tests for the rule engine + cooldown logic
    └── observability.test.ts  # new — 6 tests for the wired bundle

package.json and tsconfig.json are untouched. The three new modules import nothing but the existing Logger / LogEvent / MetricsSnapshot types, so they can be unit-tested with plain synthetic events and a fixed clock — no Supabase SDK, no HTTP, no timers.

Implementation

The collector in src/metrics.ts is a folded reducer over the log stream. Every LogEvent runs through observe, which bumps the global totals, ensures a ChannelMetrics row exists for the event's channel, increments a counter using a static COUNTER_MSG_MAP lookup, and then delegates to updateFromMessage for the handful of events that also carry a gauge value:

observe(event) {
  bumpTotals(event.level);
  const channel = readString(event.data, "channel");
  if (!channel) {
    return;
  }
  const cm = ensure(channel);
  cm.lastSeenAt = event.ts;
  incrementCounter(cm, event.msg);
  updateFromMessage(cm, event);
  updateLevelTotals(cm, event.level);
},

Splitting the counter map from the gauge extraction keeps the hot path branch-free — a pg.change event on a busy channel touches exactly one map lookup and one integer increment, which matters when the collector sits behind every payload the socket ever delivers. Events without a channel field still bump the global totals so unattributed errors are never silently dropped, they just do not create a phantom empty channel row.

Gauges are lifted out of the payload data by updateFromMessage, which pulls dropoutMs from channel.recovered, sessionMs from channel.leave, durationMs plus rowCount from channel.catchup-complete, and attempts from channel.reconnect-recovered. catchupRowsRecovered is a true counter — every completion adds its rowCount to a running total — so an operator can see both "the last catch-up returned 5 rows" and "we have recovered 8 rows over the lifetime of this channel" without any extra logging.

The prometheus() method walks two static tables — COUNTER_FIELDS and GAUGE_FIELDS — and formats each populated field into the canonical Prometheus text exposition format. Label values run through escapePromLabel first, which escapes backslashes, double quotes, and newlines so a channel name like weird"name\here cannot break the scraper's parser:

function escapePromLabel(value: string): string {
  return value.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\n/g, "\\n");
}

The alert engine in src/alerts.ts is a small rule loop with per-(rule, channel) cooldowns. Four rules ship as defaults: reconnect-exhausted fires at critical the moment the reconnect controller gives up, catchup-failure fires at warning on any failed REST catch-up, silent-death fires at warning when the heartbeat monitor trips, and flapping-channel fires at warning once a channel has left SUBSCRIBED five or more times. The rules are pure predicates over a ChannelMetrics snapshot, so a caller can register their own without touching the loop:

export const reconnectExhaustedRule: AlertRule = {
  name: "reconnect-exhausted",
  severity: "critical",
  when: (cm) => cm.reconnectExhausted > 0,
  message: (cm) =>
    `channel ${cm.channel} exhausted its reconnect budget ${cm.reconnectExhausted} time(s) — the socket is dead and no more retries will fire`,
};

Every fired rule produces an AlertPayload — rule name, severity, channel, human message, a cloned ChannelMetrics snapshot, and an ISO firedAt — that flows to three destinations at once: an alert.fired log line (at error for critical, warn for warning), the caller's optional sink closure, and the return value of evaluate(). The sink dispatch is wrapped in try/catch and lands as alert.dispatch-failed on the logger if the webhook is down, so a broken Slack integration cannot poison the evaluator's internal cooldown table.

Cooldowns are keyed on ${rule}::${channel} and default to 60 seconds so the same critical alert does not spam every tick. Per-rule cooldownMs overrides the default, reset() clears the whole cooldown table for tests, and — crucially — the payload's metrics field is a shallow clone so a downstream consumer that mutates it cannot corrupt the evaluator's view of the world.

Finally, src/observability.ts glues the three modules together. createObservability() builds a MetricsCollector, wraps the caller's LogSink in a teeSink that observes every event before forwarding it downstream, creates a Logger that writes through the tee, and hands the same logger to the alert evaluator so alert.fired and alert.dispatch-failed also flow into the metrics collector. A single tick() call snapshots the metrics and returns the fired alerts, so an operator can wire the whole system to a cron loop with three lines of code.

Verification

Run the full suite from the codebase/ directory:

bun test

bun test v1.2.19 (aad3abea)

 127 pass
 0 fail
 365 expect() calls
Ran 127 tests across 13 files. [242.00ms]

One hundred twenty-seven passing tests across thirteen files — the ninety-two from step 7 plus thirty-five new ones split between fifteen metrics cases, fourteen alerts cases, and six observability cases — finish in about a quarter of a second. Every existing assertion from steps 1 through 7 still passes; the new modules attach to the log stream but never modify a single event that flows through it.

What we built

The client now has a complete observability surface that a production team can plug into their existing stack without writing a single line of glue. The MetricsCollector folds every event the previous seven steps introduced into per-channel counters and gauges — twenty-two fields per channel plus four global totals — that are safe to snapshot, safe to mutate downstream, and cheap enough to update on every pg.change without any measurable overhead.

The prometheus() renderer produces the standard text-format page a Prometheus, VictoriaMetrics, or Grafana Agent scraper can consume with zero configuration beyond a target URL. Seventeen counters and five gauges are exposed with escaped channel labels and correct # HELP / # TYPE headers, so a dashboard for reconnect exhaustion, catch-up row counts, silent deaths, and dropout latency can be built entirely from the standard PromQL vocabulary.

The AlertEvaluator closes the loop from "we have data" to "we page a human". Four default rules cover the four failure modes the earlier steps built defenses against — an exhausted reconnect budget, a failed catch-up fetch, a silent-death breach, and a flapping channel — with per-rule cooldowns, structured alert.fired logs, and a pluggable sink that can post to PagerDuty, Slack, or a private webhook. A broken sink is caught, logged, and cannot corrupt the evaluator's state.

createObservability() bundles all three behind a single { logger, metrics, alerts, tick } object, so an application only has to construct one thing and call bundle.tick() from a cron loop. The library now walks the full path from "raw socket event" to "a dashboard tile turns red and a webhook fires" — every layer is unit-tested, dependency-injected, and free of hidden global state.

Repository

The state of the code after this step: 736a8e0

Repository

Full source at https://github.com/vytharion/supabase-realtime-subscription-dropouts.

Walk the lessons by stepping through the git commits in the repo — each major step has its own commit you can git checkout and rerun.