Back to Blog
Security

Your retries carry a signature that expired four minutes ago

A webhook sender that signs once and then retries hands every receiver a timestamp that ages past the five-minute tolerance by the third attempt. Here is why the bug is the natural shape of a retry loop, how to spot it from the receiving end, and why widening the window is the wrong fix.

WebhookVault Team · Webhook Infrastructure Experts9 min read
MacBook Pro screen seen at an angle in a dark room, showing a code editor with HTML markup in two panes and a yellow vertical guide line

Three retries in, every delivery is a 401

On October 21, 2026, a bug report landed on an open-source payments project with a title that should make every webhook sender check their own code: refresh webhook signatures on every delivery attempt. The service computed the timestamp and the HMAC once, before entering its retry loop. Then it slept between attempts: one minute, then five, then longer.

The report lists the age of the signature at each of the five attempts: 0, 60, 360, 1,260 and 4,860 seconds. Receivers enforce a 300-second replay window. So the first two attempts could succeed. Attempt three arrived six minutes after signing and was rejected. So were attempts four and five. At that point the retry schedule was just a slow way of collecting 401s.

The fix moved one line into the loop. What stays with me is how long a bug like this can live unnoticed, and how much of the damage lands on a receiver who did nothing wrong.

The bug is the natural shape of a retry loop

Nobody writes this on purpose. You build the request and send it. If it fails, you send it again. The request is a value, so why would you rebuild it?

import { createHmac } from 'node:crypto'

function sign(secret: string, id: string, ts: number, body: string) {
  const key = Buffer.from(secret.replace(/^whsec_/, ''), 'base64')
  const sig = createHmac('sha256', key).update(`${id}.${ts}.${body}`).digest('base64')
  return {
    'content-type': 'application/json',
    'webhook-id': id,
    'webhook-timestamp': String(ts),
    'webhook-signature': `v1,${sig}`,
  }
}

// Broken: the timestamp is frozen before the first attempt
async function deliver(url: string, secret: string, event: { id: string }) {
  const body = JSON.stringify(event)
  const headers = sign(secret, event.id, Math.floor(Date.now() / 1000), body)

  for (const delaySeconds of [0, 60, 300, 900, 3600]) {
    await sleep(delaySeconds * 1000)
    const res = await fetch(url, { method: 'POST', headers, body })
    if (res.ok) return
  }
}

That is the Standard Webhooks scheme: HMAC-SHA256 over id.timestamp.body, the key being the base64 part of a whsec_ secret. The delays happen to reproduce the ages in the bug report exactly. It looks fine in review. Unit tests pass too, because unit tests don't sleep for an hour.

The fixed version differs by where one line sits:

async function deliver(url: string, secret: string, event: { id: string }) {
  const body = JSON.stringify(event)

  for (const delaySeconds of [0, 60, 300, 900, 3600]) {
    await sleep(delaySeconds * 1000)
    // Sign at send time. Same id, fresh timestamp, fresh signature.
    const headers = sign(secret, event.id, Math.floor(Date.now() / 1000), body)
    const res = await fetch(url, { method: 'POST', headers, body })
    if (res.ok) return
  }
}

The body stays identical and so does webhook-id. Only the timestamp and the signature move.

Five minutes against a schedule measured in hours

Every timestamped signature scheme comes with a tolerance. Stripe's libraries default to 300 seconds for the t= value in Stripe-Signature. The Standard Webhooks reference libraries use five minutes too. Slack tells you to reject requests whose X-Slack-Request-Timestamp is more than five minutes old.

Now look at retry schedules. Stripe retries undelivered events for up to three days. Most senders that bother with retries run for hours. Put those two numbers side by side. If a sender signs once, only the attempts inside the first five minutes can ever verify. Everything after that is noise with a valid-looking header.

The worst case is the outage that motivated retries in the first place. Your endpoint is down for ten minutes. The sender's first two attempts hit the outage. By the time you are back up, every remaining attempt carries a signature you are obliged to reject. The retry policy exists to survive exactly that outage. It can't, because the one thing the sender reused was the thing with a clock in it.

Stripe gets this right and says so: each delivery attempt gets a new timestamp and a new signature. If you send webhooks, that sentence belongs in your docs too, because receivers will want to know.

The signed timestamp is attempt time, not event time

Part of the confusion is about meaning. People read webhook-timestamp as "when did this happen" and conclude it should be fixed at event creation. Wrong field for that. The signed timestamp is there for replay protection. It answers one question: was this exact request produced recently? An hour-old event that was signed a second ago is fresh. A one-second-old event signed an hour ago is stale.

When the event happened lives in the payload. That is the field to use when you measure delivery lag, which we argued for in the post on GitHub's hour-long delay. Keep the two apart in your head and in your code. Conflating them leads senders to freeze the timestamp, and it leads receivers to reject legitimately late events for being "too old" when the signature was in fact fresh.

Your outbox may have baked the signature in

The retry loop is the obvious place for this bug. The less obvious place is your queue.

A transactional outbox is the right way to send webhooks reliably, as covered in the post on delivery guarantees. But look at what you store in the outbox row. If a well-meaning engineer decided to store the fully prepared request, headers included, so the worker "just sends it", the timestamp was frozen at insert time. Every retry the worker makes replays it.

That version fails even without retries. GitHub had webhook delays on October 19 and again on October 21, 2026. A backlog like that, in a sender that signs at enqueue, means the first attempt arrives already outside the window. Nothing fails until the queue backs up, and then everything fails at once.

Store the event and the endpoint reference. Sign in the worker, right before the socket write. The same applies to a "resend" button in your dashboard: it must re-sign with the current time and the current secret, not replay a stored request. A request replayed from a log with its original headers should fail verification. That is the replay protection doing its job.

What this looks like from the receiving end

As a receiver you can't read the sender's code, but you can see the symptom. Two things give it away.

First, your rejections are split wrong. If your verifier returns one generic "invalid signature" error, you can't tell a wrong secret from a stale timestamp. Separate them. A wrong secret fails every request. A stale timestamp fails only the retries.

Second, the age of the signed timestamp grows with each attempt. Record it. Count how many times you have seen each webhook-id, and plot the age by that count:

import { Histogram } from 'prom-client'

const signatureAge = new Histogram({
  name: 'webhook_signature_age_seconds',
  help: 'Receipt time minus the signed timestamp, by observed attempt number',
  labelNames: ['provider', 'attempt'],
  buckets: [1, 5, 30, 120, 300, 900, 3600, 14400],
})

async function recordAge(provider: string, id: string, signedTs: number) {
  // Diagnostics only. Never use this counter for idempotency:
  // it runs before verification and an attacker can inflate it.
  const attempt = await redis.incr(`wh:attempts:${provider}:${id}`)
  await redis.expire(`wh:attempts:${provider}:${id}`, 7 * 24 * 3600)

  const age = Math.floor(Date.now() / 1000) - signedTs
  signatureAge.observe({ provider, attempt: String(Math.min(attempt, 6)) }, age)
}

For a healthy sender, the age sits near zero for every attempt bucket. For a sender that signs once, attempt one sits near zero and attempt three sits in the hundreds. One look at that chart and you know whom to email.

No status code rescues a stale signature

Once you know, you will want to handle it. You mostly can't.

Return 401 or 400 and some senders stop retrying immediately. Knock, for example, never retries 400 through 405, as we covered in the post on its retry rewrite. Return 503 instead and the sender retries, but with the same frozen timestamp, so the next attempt fails the same way. Either way the event never verifies.

So the realistic path has two parts. Report the bug to the sender with your age-by-attempt numbers attached. Then backfill from their events API. The events still exist on their side; they just could not be delivered with a valid signature. The post on pull-based backstops covers how to do that without double-processing.

Widening the tolerance is the wrong fix

The tempting workaround is to raise your tolerance to a day for that one provider. It does make the retries verify. It also means any captured request can be replayed against you for a day.

So what does the tolerance buy you? Replay protection is really two mechanisms. The timestamp bounds how old a request can be. Deduplication on the event id stops the same event from being processed twice inside that bound. The tolerance decides how long you must keep processed ids around. A five-minute window with a dedup store that keeps ids for a week is solid. A 24-hour window with a dedup store that expires ids after an hour is a hole.

If you widen the window for one sender, keep processed ids for longer than the new window and limit the change to that sender's endpoint. Put a removal date on it. And make sure your handler is genuinely idempotent first. The post on replay attacks goes through why the timestamp alone was never enough.

Clock skew produces the same rejection

Not every stale-timestamp rejection is the sender's retry loop. A receiver whose clock drifts produces the same error, and so does a sender whose clock drifts. The difference shows up in the chart: skew affects every attempt equally, including the first, and usually sits at a fixed offset.

Skew also runs the other way. Timestamps from the future fail too, and the reference libraries reject those as well. Containers inherit the host clock, so a node with broken time sync breaks every pod on it. Check time sync before you blame the sender. Alerting on it costs almost nothing.

Test the sender with a clock you control

The reason this bug ships is that no test waits long enough. Fake timers fix that. Drive the delivery loop with a mock endpoint that verifies like a strict receiver, fail the first attempts, and advance the clock past the tolerance:

import { describe, it, expect, vi } from 'vitest'
import { createHmac, timingSafeEqual } from 'node:crypto'

const SECRET = 'whsec_' + Buffer.from('test-secret-key-0123456789').toString('base64')

function strictVerify(headers: Record<string, string>, body: string): boolean {
  const ts = Number(headers['webhook-timestamp'])
  if (Math.abs(Date.now() / 1000 - ts) > 300) return false
  const key = Buffer.from(SECRET.slice(6), 'base64')
  const expected = createHmac('sha256', key)
    .update(`${headers['webhook-id']}.${ts}.${body}`)
    .digest('base64')
  const got = headers['webhook-signature'].split(',')[1]
  return timingSafeEqual(Buffer.from(expected), Buffer.from(got))
}

describe('deliver', () => {
  it('produces a verifiable signature on a late retry', async () => {
    vi.useFakeTimers()
    const results: boolean[] = []
    let calls = 0

    vi.stubGlobal('fetch', async (_url: string, init: any) => {
      calls += 1
      results.push(strictVerify(init.headers, init.body))
      return { ok: calls >= 4 }
    })

    const done = deliver('https://example.test/hook', SECRET, { id: 'evt_1' })
    await vi.advanceTimersByTimeAsync(2 * 3600 * 1000)
    await done

    expect(results).toEqual([true, true, true, true])
    vi.useRealTimers()
  })
})

Against the broken version, the third result comes back false. Against the fixed one, every attempt verifies. Without a test like this, you find out during somebody else's outage.

Frequently asked questions

Should the webhook-id change on each retry? No. The id identifies the event, and receivers use it as their idempotency key. If it changes per attempt, a receiver that processed attempt two will process attempt three as a new event. Keep the id and body stable and regenerate only the timestamp and the signature.

Does GitHub have this problem? Not in this form. GitHub's X-Hub-Signature-256 is an HMAC over the body only, with no timestamp in the signed content, so there is no window for a retry to fall outside. The flip side is that the signature gives you no replay protection at all, and deduplication on X-GitHub-Delivery has to do that job alone.

How large should my tolerance window be? Five minutes is the common default in Stripe's and the Standard Webhooks libraries, and it is a sensible one if your clocks are synced. Going shorter buys little and turns minor clock drift into rejected deliveries. Going longer is defensible only if your stored event ids outlive the window.

Can I verify a stale request anyway to avoid losing the event? You can check the HMAC while ignoring the timestamp for diagnostics, which tells you the secret is right and the sender is reusing signatures. Do not act on that body. Take the event id and fetch the event from the provider's API, so what you process comes from an authenticated call you made yourself.

Related posts