Nov 8, 2026

Your retry logic is the outage

A retry is a second request to a service that just told you it is struggling. If every client retries at the same instant, your recovery becomes a second outage.

APIsDashboardsTypeScript

A dashboard under load

Photo: Unsplash

A dashboard API goes down for ten seconds. It comes back up. Then it goes down again, harder, for a minute. Nothing changed on the server in between — except that every dashboard that was waiting started retrying at the same moment.

That second outage was not caused by the failure. It was caused by the retry logic.

A retry is a second request to a service that just told you it is struggling. On its own, one retry is harmless. But a dashboard is never one client. When the API blips, every open tab, every polling widget, and every background refresh fails at roughly the same time — and if they all retry after the same fixed delay, they all arrive back together. The recovering service gets hit with a wave bigger than the traffic that knocked it over in the first place. That is a retry storm, and it is the most common way a ten-second blip turns into a ten-minute incident.

Exponential backoff helps, but only halfway. Waiting 200ms, then 400ms, then 800ms spreads one client's retries out. It does nothing about synchronization across clients, because every client computes the same numbers. Two hundred clients that failed together still retry together — just a little later.

The missing piece is jitter: make each wait random. Marc Brooker's 2015 post on the AWS Architecture Blog, "Exponential Backoff and Jitter", compared the variants in a simulation of 100 contending clients, and full jitter — a uniform random wait between zero and the exponential cap — did more than half the total work of plain exponential backoff while finishing faster. Most AWS SDKs now ship it as their standard retry mode.

// Full jitter: the exponential value is only an upper bound.
// Each client samples its own wait inside it, so retries
// spread out instead of arriving as a wave.
function fullJitterDelay(attempt: number, baseMs = 200, capMs = 10_000): number {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * exp);
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

// Only retry failures that can pass: network errors and
// 408 / 429 / 502 / 503 / 504. A 400 or 404 will fail
// identically on every attempt — retrying it is just load.
const RETRYABLE = new Set([408, 429, 502, 503, 504]);

async function fetchWithRetry(url: string, maxAttempts = 4): Promise<Response> {
  const deadline = Date.now() + 10_000;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const res = await fetch(url);
      if (res.ok || !RETRYABLE.has(res.status)) return res;
      if (attempt === maxAttempts - 1) return res;
      const retryAfter = res.headers.get("Retry-After");
      const wait = retryAfter
        ? Number(retryAfter) * 1000
        : fullJitterDelay(attempt);
      if (Date.now() + wait > deadline) return res;
      await sleep(wait);
    } catch (err) {
      if (attempt === maxAttempts - 1) throw err;
      await sleep(fullJitterDelay(attempt));
    }
  }
  throw new Error("unreachable");
}

Three details matter as much as the formula. First, respect Retry-After when the server sends one on a 429 or 503 — it is the server telling you exactly how long to stay away, and your jittered wait should never undercut it. Second, bound the whole thing: a maximum attempt count and a total deadline, so a client never retries forever against a dependency that is not coming back. Third, retry in exactly one layer. If your API gateway already retries, your service client and your ORM driver should fail fast instead — otherwise one logical retry multiplies into three, and your budget math is fiction.

That budget is the last control. Even perfect jitter does not make unlimited retries safe, so cap retries as a share of traffic — Google's SRE guidance puts it around 10%. Past that point, stop retrying and fail fast with a clear state instead of adding load to a fire.

The rule I use now is simple: retry only what can pass, wait longer every time, randomize every wait, and stop on a budget. A retry policy is not a comfort blanket for a flaky dependency. It is load you are choosing to put on a struggling service, and it should be designed like load — spread out, bounded, and pointed only at failures that have a chance of passing.

Your retry logic is the outage

Further reading