GitHub's Webhooks Ran an Hour Late. Late Is Not the Same as Lost.

Sixty-seven minutes of nothing, then everything

At 15:49 UTC on October 19, 2026, GitHub opened an incident for degraded performance in Actions and Pages. A minute later, Webhooks joined the list. The status page named the impact: delays in check and workflow webhook events generated by Actions runs. Recovery started at 16:30 and the incident closed at 16:56. There is no root cause yet.

If you run a bot, a deploy orchestrator, a merge gate or a CI dashboard on top of check_run, check_suite, workflow_run or workflow_job, your receiver did nothing wrong that afternoon. It returned 200 to everything it got. It got very little for an hour, then a lot at once.

This post is about that kind of incident. Every event eventually arrives, and your system still ends up in the wrong state.

Late is a different failure from lost

Most webhook resilience advice assumes events get lost. You make the handler idempotent and keep a reconciliation job, or poll the provider's events API as a backstop. We covered that safety net in the post on events APIs, and you should have one.

A delay breaks other assumptions. A dropped event leaves you with missing data, and a reconciler can fill that in. A delayed event does damage through what you did when it finally showed up. Your handler acts on something that was true an hour ago. It posts a red status on a pull request that has since gone green, or starts a deploy for a commit that is no longer the head of main.

A reconciler won't catch that. It compares your state with the provider's, both look valid, and nobody notices the deploy that shouldn't have happened.

Your dashboards stayed green the whole time

Most webhook monitoring watches error rate, p95 handler latency, 5xx count and queue depth. During the GitHub delay all four looked healthy. Nothing was failing, and there was hardly any traffic to be slow on.

Monitoring built around your endpoint can't see a problem upstream of it. Only the volume changed, and a drop between 15:50 and 16:30 UTC looks like a quiet afternoon.

So you need two signals that don't depend on errors. One is absence: how long since the last event of a given type, compared with what that type normally does at this hour. For a busy organisation, ten minutes without a workflow_job event during working hours is an incident. The other is lag.

Measure lag against event time, not arrival time

GitHub doesn't send a timestamp header on webhook deliveries. You get X-GitHub-Event, X-GitHub-Delivery and X-Hub-Signature-256, and none of them says when the event happened. The payload does. A workflow_run carries updated_at, a workflow_job carries completed_at once it is done, and a check_run has started_at and completed_at.

Subtract that from the moment your receiver accepted the request and you have delivery lag. Record it as a histogram per event type, before you enqueue anything:

import { Histogram } from 'prom-client'

const lag = new Histogram({
  name: 'webhook_delivery_lag_seconds',
  help: 'Seconds between the event happening at the sender and our receiver accepting it',
  labelNames: ['provider', 'event'],
  buckets: [1, 5, 15, 60, 300, 900, 1800, 3600, 7200],
})

function eventTime(event: string, body: any): string | undefined {
  switch (event) {
    case 'workflow_run':
      return body.workflow_run?.updated_at
    case 'workflow_job':
      return body.workflow_job?.completed_at ?? body.workflow_job?.started_at
    case 'check_run':
      return body.check_run?.completed_at ?? body.check_run?.started_at
    default:
      return undefined
  }
}

export function recordLag(event: string, body: any, receivedAt = Date.now()) {
  const at = eventTime(event, body)
  if (!at) return
  const seconds = (receivedAt - Date.parse(at)) / 1000
  lag.observe({ provider: 'github', event }, Math.max(0, seconds))
}

Normally this sits in the low seconds. On October 19 it would have jumped to tens of minutes and stayed there until the backlog drained. Alert on p95 lag per event type. The average gets pulled down by the few on-time events that still trickle through.

The number is rough. Clocks drift, updated_at moves when a run is re-run, and some event types have no clean event-time field. That's fine. Telling two seconds apart from twenty minutes is all it has to do.

Your replay window will not catch a late event

You might expect your signature check to reject stale deliveries. It won't.

GitHub's X-Hub-Signature-256 is an HMAC over the body and covers no timestamp. Senders that do sign one, like Stripe with the t= value in Stripe-Signature or anyone on Standard Webhooks with webhook-timestamp, sign the time of the delivery attempt, not the time of the event. An event that waited an hour in the sender's pipeline gets a fresh timestamp when it finally goes out, and sails through a five-minute tolerance check.

That's by design. Replay protection stops an attacker from resending a captured request, as we covered in the replay attacks post. Whether the information inside is still current is a question for your business logic, and you have to write that check yourself.

The backlog arrives as one burst

When the sender recovers, it drains the backlog as fast as its workers allow. Your receiver goes from near silence to several times its normal peak in a few minutes.

That burst is the most dangerous part of the incident. Database connection pools fill up. Downstream rate limits trip, especially if your handler calls back into the GitHub API with an installation token. An autoscaler that reacts to CPU is three minutes behind a spike that lasts five.

If your receiver acknowledges fast and processes from a queue, the burst lands in the queue and drains at a pace you pick. If it processes inline, the burst lands on your handler. And your handler has ten seconds.

Draining is where late turns into lost

GitHub expects a 2xx response within 10 seconds, and it doesn't retry failed deliveries automatically. Put those together and the recovery phase is where you actually lose events.

A handler that answers in 300 milliseconds on a normal day can take 12 seconds when it's competing with a few thousand backlogged deliveries for the same database. Each of those deliveries is recorded as failed, and unless you act, it's gone. Same trap as in the post on GitHub's longer installation tokens: with GitHub, recovering a dropped delivery is your job.

The delay itself costs you nothing. The burst at the end can quietly cost you dozens of events. Acknowledge first: verify the signature, write the raw delivery to durable storage, return 202, and do the real work later. During a burst your endpoint should accept and store. Thinking can wait.

Act on current state, not on the event you just got

Once events can arrive an hour late, every handler with a side effect needs a staleness guard. It's the pattern behind fetch-on-notify: the event tells you something changed, and then you go check what is true now.

For Actions events the check is cheap. Before you deploy on a successful workflow_run, confirm the run's commit is still the head of the branch and the run is still the latest attempt:

import { Octokit } from '@octokit/rest'

export async function shouldDeploy(octokit: Octokit, payload: any): Promise<boolean> {
  const run = payload.workflow_run
  const { owner, name: repo } = payload.repository
  const owner_login = owner.login

  const { data: current } = await octokit.rest.actions.getWorkflowRun({
    owner: owner_login,
    repo,
    run_id: run.id,
  })
  if (current.run_attempt !== run.run_attempt) return false
  if (current.conclusion !== 'success') return false

  const { data: branch } = await octokit.rest.repos.getBranch({
    owner: owner_login,
    repo,
    branch: run.head_branch,
  })
  return branch.commit.sha === run.head_sha
}

Two API calls per event will hurt during a burst. One more reason to process from a queue with a concurrency limit. Coalescing helps too: if six delayed workflow_run events for the same branch sit in the queue, only the newest one matters.

Same for anything that posts back to GitHub. A bot that comments "CI failed" on a pull request should first check that the failing run is still the latest one. Otherwise your contributors get a pile of comments about failures that were fixed forty minutes ago.

Ordering gets worse as well. A backlog doesn't drain in event order, so a completed event can beat the matching in_progress. If your handlers still assume order, read the post on out-of-order delivery.

Timeouts that assume the webhook is coming

A quieter failure lives in systems that wait for a webhook. A deploy orchestrator starts a workflow and waits up to 30 minutes for the workflow_run completion event. A merge queue waits for a check_suite. An internal tool shows "running" and marks the job failed after a deadline.

During a 67-minute delay, every one of those timers fires. The workflow succeeded and the event is on its way, but your system has already recorded a failure. Maybe it rolled something back. Maybe it paged someone.

A missing webhook is no answer. When a deadline expires, ask the provider's API for the actual state of the run before you declare anything. The webhook is the fast path, the API call is the truth, and a timeout means you go and look.

Recovering what failed during the drain

After the incident you want two lists: deliveries GitHub recorded as failed during the burst, and events you accepted but haven't processed yet. The second lives in your own queue or storage. GitHub keeps the first one for you.

For repository webhooks the endpoints are GET /repos/{owner}/{repo}/hooks/{hook_id}/deliveries and POST /repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id}/attempts. Organisation webhooks and GitHub Apps have equivalents. The list endpoint takes a status filter of success or failure. Don't rely on the filter alone, though. A delivery can fail once and succeed on a later redelivery, and you don't want to send it a third time. Group by guid and redeliver only what never succeeded:

import { Octokit } from '@octokit/rest'

export async function redeliverFailures(
  octokit: Octokit,
  owner: string,
  repo: string,
  hook_id: number,
  since: Date,
) {
  const byGuid = new Map<string, { failedId?: number; succeeded: boolean }>()

  for await (const page of octokit.paginate.iterator(
    octokit.rest.repos.listWebhookDeliveries,
    { owner, repo, hook_id, per_page: 100 },
  )) {
    for (const d of page.data) {
      if (new Date(d.delivered_at) < since) continue
      const entry = byGuid.get(d.guid) ?? { succeeded: false }
      if (d.status_code >= 200 && d.status_code < 300) entry.succeeded = true
      else entry.failedId ??= d.id
      byGuid.set(d.guid, entry)
    }
  }

  for (const [guid, entry] of byGuid) {
    if (entry.succeeded || entry.failedId === undefined) continue
    await octokit.rest.repos.redeliverWebhookDelivery({ owner, repo, hook_id, delivery_id: entry.failedId })
    console.log(`redelivered ${guid}`)
  }
}

A redelivery keeps the original X-GitHub-Delivery value, so idempotency keyed on that header protects you from events you already handled. Redeliveries are late by definition, so the staleness guard applies to them too. And GitHub only lets you redeliver recent deliveries. Run this the same day.

Pace them. Firing a few hundred at once recreates the burst you just got through.

A runbook for the next provider delay

The next delay may not be GitHub's, and it won't look the same. Write the runbook while this one is fresh.

Detect it with absence and lag alerts per event type, routed to the same channel as your error alerts. Confirm it against the provider's status page, through its API if it has one, so the alert can say "the provider reports a webhooks incident" and the on-call engineer doesn't have to guess. While the delay lasts, pause automations with irreversible side effects, such as deploys and merges, or switch them to polling. During recovery, watch queue depth and lower consumer concurrency if downstream rate limits start to bite. Afterwards, redeliver failures, run your reconciler, and check what your automations did between the start of the incident and the end of the drain.

Then count the timers in your system that wait for a webhook and would have fired during a 67-minute delay. That's your exposure.

Frequently asked questions

Did GitHub drop any webhooks during the October 19 incident? The status page reports degraded performance and delays for check and workflow events generated by Actions runs. It mentions no dropped events. Deliveries that failed during the recovery burst because a receiver was slow are a separate problem, and they show up as failures in your webhook's delivery log.

Can I use the X-GitHub-Delivery header to measure how late an event is? No. It's a GUID that identifies the delivery and stays the same across redeliveries, and it carries no time information. Use an event-time field from the payload, such as updated_at on a workflow run or completed_at on a check run, and compare it with the time your receiver accepted the request.

Should I reject webhook events that are older than some threshold? Rarely. A late event still describes something that really happened, and dropping it turns a delay into a loss. Accept and store it, then let the handler check current state before it does anything with side effects.

How long does GitHub keep deliveries available for redelivery? The window is short, days and not weeks, so treat recovery as a same-day task after an incident. Check GitHub's current documentation for the exact limit before you depend on it in a runbook.

Does the same advice apply to providers that retry automatically? Mostly. A sender with automatic retries will resend what failed during the burst, but those retries arrive even later than the backlog did. Lag alerts, staleness guards and an acknowledge-first handler matter just as much there.