A sender that never retried just started
Most webhook advice assumes the sender retries. Stripe retries for days, Shopify for hours, so you build idempotent handlers. Fine. But plenty of integrations were written against senders that delivered once and moved on. Against those, a handler that isn't idempotent works. It always has. Nothing arrived twice, so nothing broke.
Then the sender publishes a changelog entry, and an assumption nobody ever wrote down stops being true.
Epilot did exactly that on October 7, 2026. Its Webhooks API changelog says: "Webhook events are now retried automatically." It's the right call for the platform. It also means every consumer that leaned on at-most-once delivery now has a duplicate problem it has never seen in production.
Epilot is the example here. The pattern is general. When a sender goes from fire-and-forget to retries, receivers break in several places at the same time.
What the October 7 entry actually says
Read the entry before you plan anything. It's short.
Events can now report a new status value, retrying. They also carry three new optional fields: attempts, max_retry_attempts and next_retry_at. The change touches GET /v1/webhooks/configs/{configId}/events, GET /v1/webhooks/configs/{configId}/events/{eventId} and POST /v2/webhooks/configs/{configId}/events, and the v2 endpoint accepts retrying as a status filter. The one instruction to consumers is blunt: they "must be prepared for this additional status."
What the entry leaves out matters as much. No retry count. No backoff curve. No list of response codes that trigger a retry, and no word on how long an attempt waits for your response before it counts as failed. Several decisions below depend on those numbers, and right now the only way to learn them is to watch the new fields on real events.
The duplicate you never had to handle
The obvious break first. Your handler gets an event, does the work and responds. Then something goes wrong from the sender's side: a 5xx, a dropped connection, a response that took too long. Before, that event was marked failed and left alone. Now it comes back.
If the work already happened, the second delivery does it again. Two emails. Two tickets. The same record pushed into the CRM twice.
The slow success is the nasty one. Your handler did everything right but took long enough that the sender gave up on the connection. You logged a success. The sender logged a failure and scheduled another attempt. The post on processed-but-errored deliveries explains why that gap exists. With a sender that never retried, it produced a misleading red row in a dashboard. With one that retries, it produces real duplicate side effects.
Speed won't save you here. What saves you is a second delivery that does no harm.
Retries also change what a slow handler costs
Teams miss this one. Without retries, a slow handler is a latency problem. With retries, it multiplies load.
Say a downstream dependency degrades and your p99 climbs past whatever timeout the sender uses. Each of those slow requests is now a failure in the sender's eyes, and each failure becomes another request later, sometimes while you're still chewing on the first attempt. If your handler retries its own downstream calls too, the layers stack. Same mechanics as the GitHub retry storm we wrote up, smaller scale.
The fix is the one you've heard before. Verify the request, write it durably, return 2xx, and process from a queue. Once retries exist, the time between request and response is the only thing the sender sees. Keep it short and boring.
A fourth status your code has never seen
The second break is quieter, and the changelog warns about it directly.
Many integrations also call the provider's event API. Maybe a dashboard, maybe an alert on failed deliveries, maybe a nightly job that finds failures and replays them. That code parses status. If it parses strictly, an unfamiliar value throws.
Here is the shape that breaks:
import { z } from "zod";
// Written against the statuses observed before October 2026.
const EventStatus = z.enum(["succeeded", "failed"]); // illustrative values
const WebhookEvent = z.object({
id: z.string(),
status: EventStatus,
});
// The first event with status "retrying" throws here,
// and the whole page of results goes with it.
const events = z.array(WebhookEvent).parse(body.data);
The status values are placeholders. Use whatever your code really matches on. The point is the failure mode: one event in a new state fails validation, the array parse fails with it, and your failed-delivery alert goes silent. A monitor that crashes on new data looks exactly like a monitor with nothing to report.
A switch that throws in default does the same. So does a database column with a check constraint, or a TypeScript union the compiler trusts and the network never promised.
Parse provider enums like they will grow
Provider enums are open sets. Treat them as open at the boundary and close them inside your own code.
const KNOWN = ["succeeded", "failed", "retrying"] as const;
type KnownStatus = (typeof KNOWN)[number];
type ParsedStatus =
| { kind: "known"; value: KnownStatus }
| { kind: "unknown"; raw: string };
function parseStatus(raw: string): ParsedStatus {
return (KNOWN as readonly string[]).includes(raw)
? { kind: "known", value: raw as KnownStatus }
: { kind: "unknown", raw };
}
// Unknown values are logged and counted, never thrown.
The unknown branch keeps one new value from taking down the page around it. It shouldn't hide anything, though. Emit a metric for unknown statuses and alert when it goes above zero. Then the next changelog entry lands on your dashboard, and you hear about it at a normal hour.
Check storage too. If event status lives in a Postgres enum type or behind a check constraint, the insert fails just like the parse did. Store text, or migrate the type before the new value shows up. Here it already has.
Make the side effect idempotent, not the endpoint
Back to duplicates. "Dedupe on the event ID" is the standard answer, and it's half right. What matters is which part of the handler the dedupe protects.
Dedupe at the HTTP layer is fragile. Record "seen" before the work and the work fails: you dropped the event. Record it after and the process dies in between: you process it twice. The version that holds up puts the dedupe record and the side effect in one transaction:
BEGIN;
INSERT INTO processed_webhooks (provider, delivery_key)
VALUES ('epilot', $1)
ON CONFLICT DO NOTHING
RETURNING delivery_key;
-- If no row came back, this key was already processed: COMMIT and return 2xx.
-- Otherwise apply the business change here, in the same transaction.
COMMIT;
Picking delivery_key is the part you can't skip. It must be identical across attempts of one event and different between events. A provider-assigned event ID usually fits. A hash of the body usually doesn't, because some senders stamp each attempt with a fresh timestamp or counter. Compare two deliveries of the same event in your logs before you decide. Don't go on the docs alone. The idempotency keys post goes deeper.
For side effects outside your database, pass the same key downstream as an idempotency key wherever the API accepts one. Where it doesn't, write an outbox row in the same transaction and let a separate worker send it once.
Your reconciliation job now has a third answer
The third break combines the first two.
Teams integrating with a fire-and-forget sender often built their own safety net: a scheduled job lists failed events from the provider's API and replays them. It was the only way to recover a missed event, and it was correct, because nobody else would resend it.
Now that job races the sender. An event that failed once sits in retrying, and the provider still plans to deliver it. Treat every not-yet-succeeded event as lost and you get your replay plus the sender's retry. Look only for failed with a strict parser and you crash on retrying.
So the replay rule needs three branches. retrying: hands off, the sender owns it. Terminal failure: yours to recover. Unknown: log it and leave it. A terminal failure is the point where the sender has given up, and that's where the pull-based backstop from the post on Events APIs earns its keep.
Run replays through the same idempotent path as live deliveries anyway. Retries and replays will overlap at the edges no matter how carefully you filter.
Use the retry fields instead of guessing the schedule
No published schedule, but the events now carry the next best thing. attempts is how many tries an event has had. max_retry_attempts is the ceiling. next_retry_at is when the sender plans to try again. Check the API reference for the exact field types before you parse them.
That turns guessing into measurement:
- Alert on events where
attemptsis close tomax_retry_attempts. Those are about to become terminal failures you have to recover yourself. - Plot the gaps between
next_retry_atvalues across a few retrying events. That's the backoff curve the changelog doesn't state. - Count events in
retryingper webhook config with the v2 status filter. A growing count means your endpoint is failing, even when your own access logs look calm.
The last one is worth the most. Before October, a failing endpoint was a pile of failed events you might check once a day. Now it's a live backlog with a timer on each item, and you can see it before the timers run out.
If you verify signatures with a timestamp tolerance, watch retried deliveries closely. Some senders sign once and resend the same headers, and those deliveries age out of a five-minute window. The timestamp tolerance post covers that. The changelog doesn't say how epilot signs retries. Test it.
Audit every integration for the same assumption
Epilot isn't unusual. Senders add retries because customers ask for them, and nobody writes "we used to deliver at most once" in a changelog. The risk sits in integrations built against a sender that never redelivered.
Go through your webhook consumers. For each one, ask whether the sender retries today and how you actually know. Ask what a second delivery would do if retries started tomorrow. Ask whether any code parses that sender's status or event-type enums strictly.
You'll usually end up with a short list. A few handlers that were never idempotent because they never had to be. One or two monitoring scripts that would crash on the first new enum value. Fix those now, while it's a chore and not an incident.
Frequently asked questions
Does epilot say how many times it retries or how long it waits between attempts? Not in the October 7, 2026 changelog entry. It adds the max_retry_attempts and next_retry_at fields to events but gives no count, no backoff curve, no timeout and no list of response codes that trigger a retry. Read those values off real events and don't hardcode a guess.
Can I just return 200 immediately and ignore duplicates? Returning 200 fast cuts down how many retries you trigger. It doesn't stop them. Network failures and sender-side timeouts still cause redelivery, so the side effect itself has to be safe to run twice.
Should my reconciliation job replay events that are in the retrying state? No. An event in retrying still belongs to the sender and will be delivered again, so replaying it yourself creates a duplicate. Only recover events that reached a terminal failed state, and send those replays through the same idempotent handler as live traffic.
Is it safe to store provider status values in a database enum type? Only if you're willing to migrate the type before the provider ships a new value, and you rarely get warned in time. Storing the raw value as text and mapping it to your own internal states in application code is much easier to keep working.
What is the quickest signal that a sender has started retrying? Two deliveries with the same provider event ID in your access logs, minutes or hours apart. If you log a stable delivery key per request, a group-by on that key with a count above one will show it on day one.