Back to Blog
Webhooks

GitHub's Webhook Delay Was a Retry Storm. Your Handler Can Start One Too.

On October 30, 2026, GitHub delayed webhooks for over half an hour because a Redis client retried itself into a connection storm. Here is how the same pattern hides inside webhook receivers, why retry layers multiply, and how to keep exactly one of them in charge.

WebhookVault Team · Webhook Infrastructure Experts10 min read
Two rack-mounted network switches above patch panels, with dozens of teal patch cables looping between the ports and yellow fibre cables running down the right side of the rack

Ninety percent of connection attempts failed

At 17:50 UTC on October 30, 2026, GitHub opened an incident for degraded performance in Actions and Webhooks. Pages joined ten minutes later. By 18:06 the status page said Actions workflows, Pages deployments and webhook events were delayed. Services recovered at 18:32 and the incident closed at 19:23. That made three GitHub incidents involving webhooks in twelve days.

This time GitHub published a cause. Maintenance on a cloud network gateway disturbed the connections between an Actions service in GitHub's data centre and Redis clusters in Azure's Central US region. The Redis client then "retried aggressively and redialed those connections." The resulting connection storm saturated the gateway, and about 90% of connection attempts failed. GitHub's list of fixes reads like a checklist: jittered backoff, shorter connect timeouts, retry budgets.

Now picture your own receiver. Nothing in that chain is exotic. A client library reconnecting on its defaults, a shared network hop, lots of instances doing the same thing at once. Your webhook endpoint probably has all of it.

The client did exactly what it was told

Nobody wrote a bug that afternoon. A Redis client that loses its connection is supposed to reconnect. The trouble is that every instance of that service did so, triggered by the same event, at the same moment. Each reconnect attempt added load to the gateway that was causing the failures. So the failures produced more reconnects.

That is what a retry storm looks like. Each retry is correct on its own and the sum is a disaster. You won't spot it by reading one function. You spot it by counting how many places in a request path are allowed to try again, and multiplying.

Count the retry layers in your own receiver

Take a webhook you receive today and walk its path. For a typical Node receiver behind a load balancer, these layers can retry:

  1. The sender. Stripe retries a failed delivery for up to three days in live mode. GitHub doesn't retry automatically at all.
  2. Your ingress. Some load balancers and service meshes retry a request against another backend when the first one resets the connection.
  3. Your handler. Somebody wrapped the database write in retry(3) after a flaky week last spring.
  4. Your SDK. The AWS SDK for JavaScript v3 makes up to three attempts per call by default.
  5. Your connection layer. A Redis or database client that reconnects in the background while commands wait in a queue.

Most teams can name two of these. The other three got added by different people at different times, each with a good reason.

Retries multiply, they do not add

Say a sender gives up after five attempts, your handler retries a write three times, and the SDK under it makes three attempts per call. One event that hits a broken dependency can now produce 5 × 3 × 3 = 45 calls to that dependency. A burst of 2,000 events becomes 90,000 calls aimed at the one component that is already failing.

It gets worse when the thing failing is a connection. New connections are expensive on the server side, with a TLS handshake and authentication for each one. A database that handles 10,000 queries a second comfortably can fall over at a few hundred new connections a second. That matches the GitHub incident. What saturated the gateway was reconnect attempts. A happy-path load test will never show you this.

The sender's clock does not wait for your backoff

GitHub expects a 2xx response within 10 seconds, and most other senders use a similar timeout. Every second your handler spends sleeping in an internal backoff loop counts against that window.

This is where inner retries hurt. A handler that retries a write after 2, then 4, then 8 seconds will never finish inside 10 seconds once the dependency is down. The sender times out and records a failure. If the sender retries, you process the same event again while the first attempt is still asleep in its loop. If the sender is GitHub, the event stays marked as failed until you redeliver it, a cleanup job we walked through in the post on GitHub's hour-long delay.

So your careful backoff bought you nothing. The sender's timeout settled it first.

Give retries to exactly one layer

Keep retries. Decide which layer owns them, and make every other layer fail fast.

The owner has to be durable. It must survive your process dying and be able to wait minutes or hours without holding a connection open. In practice that leaves two choices. The sender owns retries, which works for providers like Stripe that keep trying for days. Or your own queue does, which you need for providers like GitHub that never retry.

Everything in the synchronous request path does the opposite. Accept the event, persist it somewhere cheap, answer quickly. If persisting fails, fail the request right away and let the owner try later. One in-request retry for a transient blip is defensible. Three with exponential backoff inside a 10-second window is asking for trouble. We made the general case for queue-first receivers in the post on queue-based webhook processing, and this failure mode is the strongest argument for it.

Connect timeouts are the setting nobody touches

Shorter connect timeouts were on GitHub's list, and they're the fix people skip most. The defaults are generous or missing. In node-postgres, connectionTimeoutMillis on a Pool defaults to 0, meaning no timeout. A handler waiting on a saturated database will wait until something else kills it.

ioredis is friendlier, but its defaults suit a long-lived process riding out an outage. A request that must answer in 10 seconds is a different situation. By default ioredis queues commands while disconnected, and each command can wait through 20 reconnection attempts before it is rejected.

Configure clients for the request path you actually have:

import Redis from 'ioredis'
import { Pool } from 'pg'

export const redis = new Redis(process.env.REDIS_URL!, {
  connectTimeout: 1_000,
  maxRetriesPerRequest: 1,
  enableOfflineQueue: false,
  retryStrategy(times) {
    // Full jitter: a random delay between 0 and the capped exponential value
    const cap = Math.min(2 ** times * 100, 10_000)
    return Math.random() * cap
  },
})

export const db = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,
  connectionTimeoutMillis: 1_000,
  statement_timeout: 3_000,
})

With enableOfflineQueue off, a command sent while Redis is unreachable fails at once. Nothing piles up in memory. The background reconnect still runs, with full jitter, so a hundred pods that lost Redis at the same instant don't all knock on its door at the same instant either.

Put a retry budget in the handler

Jitter spreads retries out. A budget caps how many there are: retries may only ever be a small fraction of real requests. When the dependency is healthy, almost nothing retries and the budget never kicks in. When it breaks, the budget runs dry within seconds and the handler stops adding load.

export class RetryBudget {
  private requests = 0
  private retries = 0

  constructor(private ratio = 0.1, windowMs = 10_000) {
    setInterval(() => {
      this.requests = 0
      this.retries = 0
    }, windowMs).unref()
  }

  recordRequest() {
    this.requests++
  }

  tryAcquire(): boolean {
    const allowed = Math.max(1, Math.floor(this.requests * this.ratio))
    if (this.retries >= allowed) return false
    this.retries++
    return true
  }
}

Wire it so a failed write gets one retry, and only when the budget allows it:

const budget = new RetryBudget(0.1)

app.post('/webhooks/github', express.raw({ type: '*/*' }), async (req, res) => {
  budget.recordRequest()
  const deliveryId = req.header('X-GitHub-Delivery')

  try {
    await storeRawDelivery(deliveryId, req.body)
  } catch (err) {
    if (!budget.tryAcquire()) return res.status(503).set('Retry-After', '30').end()
    try {
      await storeRawDelivery(deliveryId, req.body)
    } catch {
      return res.status(503).set('Retry-After', '30').end()
    }
  }

  res.sendStatus(202)
})

I left signature verification out to keep this short. It goes before the store call. The budget is per process, and that's fine. Each instance just needs to stop feeding the storm. They don't need to coordinate.

Serverless receivers turn retries into new connections

On Lambda, Cloud Run or similar platforms, the multiplication gets another factor. A slow handler holds its instance longer. The platform scales out to absorb incoming traffic, and every new instance opens its own fresh database connection. When the sender retries the deliveries that timed out, those retries land on more cold instances, each dialling in from scratch.

That's the GitHub incident in miniature. A connection pooler in front of the database helps, and so does a concurrency limit on the function. The most useful step is the dullest one: write the raw delivery to something that shrugs off connection churn, like an object store or a managed queue, and do the database work in a consumer running at a concurrency you pick.

The recovery is the second storm

Outages end, and the ending has its own spike. When the dependency comes back, every client that was backing off reconnects. Every queued event is ready at once. A sender that retries is now delivering its backlog on top of new traffic. GitHub noted that some Actions queues needed extra time to drain, which is the polite version of this.

Jitter matters most here. Fixed backoff keeps clients in sync: everyone who failed at 17:51 retries at 17:52, again at 17:54, again at 17:58. Random delays break the lockstep. On the consumer side, cap the drain rate. A backlog that clears in twenty minutes without tripping anything beats one that clears in five and knocks the database over again.

We looked at that burst from the receiving end in the post on infrastructure failures at scale. The narrower point here: your recovery path needs the same jitter and limits as your failure path. It's the same code, running under worse conditions.

Tell the sender the truth while you are degraded

When your durable layer is down and you can't accept an event, say so. A 503 with Retry-After is the honest answer. Senders that respect it back off. Senders that don't still record a failure you can redeliver later. Don't hang for 9.5 seconds hoping the dependency comes back and then time out. The sender learns nothing and your instance stays busy for no reason.

Know which senders listen. Some honour Retry-After on 429 and 503, some ignore it, and GitHub won't retry either way. Receivers still have only a thin vocabulary for steering senders, which we went through in the post on response signals. With providers that don't retry, the 503 just leaves a mark in their delivery log. Your post-incident redelivery job is the actual retry.

Test the storm before a gateway does it for you

You can reproduce all this in an afternoon. Put a fault-injection proxy such as Toxiproxy between your receiver and Redis or Postgres. Add latency, cut the connection for thirty seconds, and replay a recorded burst of webhook traffic through the receiver meanwhile. Then count connection attempts per second against the dependency, responses that took longer than the sender's timeout, and how many times each event was processed.

Connection attempts climbing steeply while the dependency is down means a storm is waiting for you. Response times bunched just under 10 seconds mean your inner retries are racing the sender's clock. Events processed twice mean your idempotency key isn't where you think it is. Much cheaper to learn on a Tuesday afternoon than during someone else's gateway maintenance.

Frequently asked questions

Should a webhook handler ever retry a failed database write inline? Once at most, and only when a retry budget allows it. A single quick retry absorbs a transient blip without risking the sender's timeout. Anything more belongs to the sender or your own queue, which can wait minutes without holding a request open.

Does GitHub retry a delivery that my endpoint answered with a 503? No. GitHub records the delivery as failed and does not retry it automatically. You have to redeliver it yourself through the REST API, and only recent deliveries are eligible, so run the redelivery job the same day as the incident.

Is exponential backoff enough without jitter? Not when many clients fail at the same moment. Without jitter they all compute the same delays and retry in lockstep, which recreates the original spike at every step. Full jitter, a random delay between zero and the exponential cap, spreads them out.

What ratio should a retry budget use? Ten percent of requests over a short window, such as ten seconds, is a common starting point. That leaves room for normal transient failures and runs out within seconds during a real outage. Tune it by watching how often the budget is exhausted on healthy days, which should be close to never.

Where does idempotency fit if retries are already limited? Everywhere retries still happen. Fewer retries means less load, but the sender, your queue and your redelivery job can all hand you the same event twice. Key processing on the delivery ID the sender provides and only run the side effect if that key hasn't been seen.

Related posts