Retries with Jitter and Backoff

By Michael Samuel Naeem · November 9, 2024 · 6 min read

A dependency goes down and every client retries every second. The dependency comes back, gets immediately hammered by synchronized retries from ten thousand instances, goes down again, and the cycle repeats for an hour. This is a retry storm — caused not by the original failure but by well-intentioned retry logic with zero jitter. Exponential backoff spaces retries out; jitter randomizes the spacing so clients don't move in lockstep.

Backoff formula

Base delay doubles each attempt, capped at a maximum:

delay = min(cap, base * 2^attempt)

With full jitter (AWS recommended):

delay = random(0, min(cap, base * 2^attempt))

TypeScript implementation

interface RetryOptions {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  retryable?: (error: unknown) => boolean;
}

function fullJitterDelay(attempt: number, base: number, cap: number): number {
  const ceiling = Math.min(cap, base * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);
}

async function withRetry<T>(
  fn: () => Promise<T>,
  opts: RetryOptions
): Promise<T> {
  const { maxAttempts, baseDelayMs, maxDelayMs, retryable = () => true } = opts;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      if (attempt === maxAttempts - 1 || !retryable(error)) throw error;
      const delay = fullJitterDelay(attempt, baseDelayMs, maxDelayMs);
      await sleep(delay);
    }
  }
  throw new Error('unreachable');
}

Usage:

const response = await withRetry(
  () => fetch('https://api.partner.com/inventory'),
  {
    maxAttempts: 5,
    baseDelayMs: 200,
    maxDelayMs: 30_000,
    retryable: (err) => err instanceof HttpError && [502, 503, 504].includes(err.status),
  }
);

Respect Retry-After

On 429 responses, honor the server's hint:

async function fetchWithRetry(url: string): Promise<Response> {
  for (let attempt = 0; attempt < 5; attempt++) {
    const resp = await fetch(url);
    if (resp.status !== 429) return resp;

    const retryAfter = resp.headers.get('Retry-After');
    const delayMs = retryAfter
      ? parseInt(retryAfter, 10) * 1000
      : fullJitterDelay(attempt, 1000, 60_000);

    await sleep(delayMs);
  }
  throw new Error('Rate limited after max retries');
}

Circuit breaker pairing

Retries without a circuit breaker retry into a dead service forever (bounded, but still wasteful). Open the circuit after N consecutive failures; fail fast until a probe succeeds:

Closed (normal) → failures exceed threshold → Open (fail fast)
Open → timeout → Half-open (probe) → success → Closed
                                    → failure → Open

Libraries: opossum (Node), resilience4j (Java), gobreaker (Go).

Message consumer retries

Kafka/SQS consumers should retry with backoff before sending to DLQ:

async function processMessage(msg: Message): Promise<void> {
  const attempt = msg.headers['x-retry-count'] ?? 0;
  try {
    await handleEvent(msg.body);
  } catch (error) {
    if (attempt >= 5) {
      await deadLetterQueue.send(msg);
      return;
    }
    const delay = fullJitterDelay(attempt, 1000, 300_000);
    await retryQueue.send(msg, { delaySeconds: delay / 1000, headers: { 'x-retry-count': attempt + 1 } });
  }
}

What not to retry

• Non-idempotent POST without idempotency key
• Validation errors (4xx except 429)
• Authentication failures (401/403) — rotate credentials first
• Errors that succeeded partially without compensation logic

My default policy: 3–5 attempts, 200ms base, 30s cap, full jitter, circuit breaker at 50% failure rate over 30 seconds.

Decorrelated jitter and AWS guidance

AWS published three jitter strategies beyond full jitter:

Decorrelated jitter works well when previous delay is tracked — each retry independently chooses a new delay based on the last, preventing the "everyone at zero" spike of full jitter on attempt 0 while still spreading subsequent retries.

For infrastructure-level retries (load balancers, service mesh), configure retry budgets — max percentage of requests that can be retries. Envoy's retry_budget prevents retry storms from consuming all connection pool capacity.

Idempotency requirements for retries

Retrying a non-idempotent operation creates duplicates:

// Dangerous — retries on timeout may double-charge
await withRetry(() => stripe.charges.create({ amount: 5000, source: token }));

// Safe — idempotency key makes retries safe
await withRetry(() => stripe.charges.create(
  { amount: 5000, source: token },
  { idempotencyKey: `charge-${orderId}` }
));

Rule: if the operation has side effects (payment, email, inventory decrement), it needs an idempotency key before you wrap it in retry logic. Read operations and idempotent upserts are safe to retry freely.

Retry budgets and hedging

Retry budget — cap total retry traffic as percentage of normal traffic:

max_retries = normal_request_rate * retry_budget_percent

When budget exhausted, fail fast instead of retrying. Prevents a degraded dependency from consuming all client capacity.

Hedged requests — send a second copy of slow requests after a delay (e.g., 95th percentile latency). First response wins; cancel the loser. Useful for latency-sensitive reads where tail latency matters more than load. Don't hedge writes — you'll create duplicates.

Per-dependency retry configuration

Not every dependency gets the same policy:

Centralize retry config in a service mesh or shared library — don't scatter magic numbers across call sites.

Observability for retry storms

Metrics to alert on:

• retry_attempts_total{service, attempt_number} — spike in attempt > 0 indicates dependency trouble
• retry_exhausted_total — retries gave up; likely user-facing errors
• circuit_breaker_state{dependency} — open circuit = fail-fast mode active
• Ratio of retry traffic to primary traffic — exceeds budget threshold

Dashboard the dependency's perspective too — if your retries double their traffic during an outage, you're part of the problem.

Failure modes

• Retrying 500s on POST without idempotency — creates duplicate records
• No cap on attempts or delay — retries forever, masking permanent failures
• Synchronized retries across clients — no jitter, creates thundering herd
• Retrying through an overloaded service — keeps it overloaded; circuit breaker prevents this
• Ignoring Retry-After on 429 — violates rate limit contract, gets IP banned
• Retry at every layer — client retries × service mesh retries × load balancer retries = exponential amplification

Production checklist

• Full jitter on all client-side retry policies
• Idempotency keys on all retried write operations
• Circuit breaker paired with retry (fail fast when open)
• Retry-After header honored on 429 responses
• Per-dependency retry config documented
• Retry budget configured at infrastructure level
• Retry metrics dashboarded with alerts on exhaustion spikes

Resources

• AWS Exponential Backoff And Jitter
• Google Cloud retry strategy
• Polly .NET resilience library
• Envoy retry policies
• gRPC retry design