The blackout nobody paged you for
Friday afternoon you ship a deploy. It drops connections for about four seconds while the old containers drain and the new ones come up. Nobody notices. Health checks go green, the graphs settle, you close the laptop.
What you did not see is that during those four seconds a handful of webhook deliveries hit a connection that was already gone. The sender marked them failed and queued a retry. Over the weekend a few more attempts landed during other brief hiccups, a load balancer failover here, a certificate renewal there. By Monday the sender has decided your endpoint is unreliable, switched it off, and sent an email to an address nobody reads. Events stop arriving. Your handler logs are clean because your handler never runs. Your monitoring is green because there is nothing failing. The first real signal is a customer asking why their upgrade never went through.
This is the failure mode people forget exists. You spend your time hardening the handler against bad payloads and duplicate processing, and meanwhile the sender quietly reserves the right to stop talking to you altogether.
Retries are a countdown, not a safety net
Most people treat retries as insurance. The sender will keep trying, so a blip is fine. That is half true, and the dangerous half.
Retries are a fixed budget with a hard deadline. Stripe attempts delivery for up to three days with exponential backoff in live mode, and that is generous compared with many senders. GitHub does not retry at all: a failed delivery is a missed delivery unless you open Recent Deliveries and redeliver it by hand. Once you understand retries as a countdown rather than a promise, the whole picture changes. Every failure spends part of a budget you cannot refill, and when the budget runs out the sender does not keep the event politely waiting. It gives up.
The backoff schedule that protects the sender works against you here. Early attempts come quickly, then the gaps widen to hours. If your outage is short but recurring, each recurrence can land on a retry that the widening schedule placed right in the danger zone, and you burn attempts on events you would have accepted fine thirty seconds later. If you have never mapped exactly how your senders back off, our breakdown of retry strategies is worth an hour, because the schedule is not the same on both sides of the connection.
What "disabled" actually means
There is a difference between a dropped event and a disabled endpoint, and it is the difference between a scratch and a severed cable.
A dropped event is one delivery lost. Annoying, recoverable, usually invisible until reconciliation catches it. A disabled endpoint is every future event lost until a human intervenes. The sender stops attempting delivery entirely. New events are not queued for you, not retried, in some cases not even retained against your endpoint the way a pending delivery would be. The blast radius is everyone, for as long as the endpoint stays off.
Stripe spells this out: if your destination has been disabled or deleted when it attempts a retry, it prevents future retries of that event. Re-enable the endpoint quickly, before the next scheduled retry, and those attempts resume. Miss that window and the events that would have retried are simply gone from the push channel. Disabling does not pause the stream. It abandons it.
Why a four-second blip becomes a three-day outage
The reason a tiny outage escalates into a total blackout is compounding, and it hides inside your deploy pipeline.
Brief connection drops are not rare events you can engineer away. They are a normal cost of rolling deploys, autoscaling, spot-instance churn, and certificate rotation. Each one is harmless in isolation. The problem is that senders do not evaluate a single failure, they evaluate a trend. A string of failures spread across a weekend of routine infrastructure noise reads, from the sender's side, exactly like an endpoint that has quietly died. It cannot tell the difference between your service being down and your service being briefly unreachable at four unlucky moments.
Volume makes it worse. Stripe warns that the start of the month, when every subscription renews at once, produces a spike large enough to overwhelm an endpoint that was fine all through the quiet middle of the month. A spike is when your handler is slowest, which is when timeouts push your failure rate up, which is the trend that gets you disabled. The failure clusters at the worst possible moment because the same load causes both the surge and your inability to absorb it.
4xx, 5xx, timeouts, and redirects are not the same failure
Senders do not see one undifferentiated "it failed." They see status codes, and they treat them very differently. If you want to avoid being disabled, you need to send the right signal.
A 5xx says your server broke and might recover, so the sender retries. A 4xx says you refuse to process the request, and some senders treat that as a permanent condition that counts hard against your endpoint's health. A timeout, where you accepted the connection but never answered in time, is counted as a failure too and is the most common cause of surprise disabling, because it only shows up under load. And a redirect is a trap most people never think about: Stripe treats any 3xx response to a webhook as a failure, full stop. If your framework quietly 301s from the non-www host to the www one, or upgrades bare HTTP to HTTPS with a redirect, every delivery fails and you never see a single request reach your handler.
(Unable to connect) ERR host not publicly reachable
(302) ERR redirect treated as a delivery failure
(4xx) ERR server refuses the request
(5xx) ERR server errored, will be retried
(Timed out) ERR accepted the connection, never responded in time
The subtle one is deciding when to return 4xx on purpose. Sometimes you genuinely want the sender to stop retrying, and the status code is your only lever, a real vocabulary covered in taking control of webhook retries through your response. The failure to avoid is returning a status you did not mean, and letting the sender's disabling logic read it as a verdict on your reliability.
Return 2xx first, do the work later
The single most effective defence against disabling is also the oldest advice in the webhook handbook, and most handlers still ignore it. Acknowledge fast, process later.
Your endpoint has one job at request time: verify the signature, persist the raw event, and return 2xx. That is it. Everything else, the database writes, the third-party calls, the email that goes out, belongs on a queue that runs after you have already answered. Stripe's own guidance is explicit that you must return the 200 before the complex logic, not after, because any work you do inline is work that can time out and turn a healthy delivery into a counted failure.
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), (req, res) => {
const event = verifyAndParse(req.body, req.headers['stripe-signature'])
// Persist first, acknowledge immediately.
enqueue('webhook-events', { id: event.id, payload: event })
// The sender sees success in milliseconds, before any real work runs.
res.status(202).end()
})
The moment you split acknowledgement from processing, a slow database or a struggling downstream API can no longer drag your delivery success rate down. Your handler stays fast under the month-start spike because it is doing almost nothing, and the heavy lifting happens on a worker pool that can fall behind without the sender ever knowing. If your handler does its work inline, moving to a queue-based ingestion tier is the change that keeps your endpoint alive when traffic triples.
Watch the sender's view of your health, not just your own
Here is the uncomfortable part. Your monitoring watches your service. Disabling is a decision made inside the sender's service, based on data you never see unless you go looking. The two views do not agree, and the gap between them is exactly where the blackout hides.
Your dashboards can be entirely green while the sender has already flagged your endpoint as unhealthy. Your success rate only counts requests that reached you. It cannot count the deliveries that failed at the connection, because those never became requests. So you need a second signal from the sender's side: the delivery success rate as the provider records it, the endpoint's enabled or disabled state, and the email that says your endpoint has been turned off. That email is an incident. It should page someone the way a downed database would. Route it to an inbox a human actually watches, or better, parse it and fire an alert.
Most senders expose their view through an API. Poll the endpoint's status on a schedule and treat any state that is not enabled as a hard alarm.
curl -s https://api.stripe.com/v1/webhook_endpoints/we_123 \
-u "$STRIPE_SECRET_KEY:" \
| node -e "const e=JSON.parse(require('fs').readFileSync(0));console.log(e.status)"
None of this replaces your own metrics. It sits beside them. The principle behind monitoring webhook health properly is that generic uptime checks miss webhook-specific failure modes, and endpoint disabling is the most webhook-specific failure mode there is. Nothing on your side of the wire will ever tell you it happened.
When the endpoint comes back, the events do not
Say you catch it. You re-enable the endpoint, the sender starts delivering again, and the graphs recover. You are not done. You are missing everything that happened during the blackout.
Re-enabling only reopens the push channel from now forward. Some in-flight retries may resume if you acted quickly, but every event whose retry budget expired while you were off is gone from the push side. Teams skip this part because the dashboard looks healthy again. The dashboard is measuring the present. The hole is in the past, in the hours between the moment the sender gave up and the moment you turned the endpoint back on, and nothing in the live stream will surface it.
Treat recovery as two separate tasks. First, stop the bleeding by re-enabling the endpoint. Second, and only after, reconcile the gap. Conflating them is how a blackout that lasted a weekend leaves a permanent hole in your data that surfaces months later as a mystery discrepancy nobody can explain.
Backfilling what the blackout swallowed
The good news is that for most serious senders the events were never truly lost. They are sitting in the provider's event log, and you can pull them back.
Stripe keeps its events retrievable through a list endpoint for 30 days, and it hands you the exact filter you need for this job. Ask for the events your endpoint failed to receive, walk them in order, and replay each one through the same handler your live webhook uses.
curl -G https://api.stripe.com/v1/events \
-u "$STRIPE_SECRET_KEY:" \
-d limit=100 \
-d delivery_success=false \
-d "created[gte]=1755734400"
delivery_success=false returns precisely the events that are still pending or have failed every delivery attempt, which during a blackout is your entire missing set. Bound it by the time window of the outage so you are not paging through the whole 30 days. Because your handler is idempotent, replaying an event you did somehow receive is harmless, so you can backfill aggressively without fear of double-processing. This pull-based recovery is the backstop that every serious webhook consumer eventually builds, and a disabling incident is the day you are grateful it exists.
One caution: the 30-day retention is a hard wall. If your endpoint sits disabled for longer than that, the oldest events age out of the log and no backfill can recover them. The clock on your blackout is not the retry budget, it is the retention window, and it is longer but not infinite.
Designing for the day the sender stops trusting you
Stop treating disabling as an edge case and start treating it as a state your system will eventually enter. When you assume it, the design writes itself.
Build the fast-acknowledge, process-later split so a slow downstream cannot spend your retry budget. Pin your endpoint URL so no redirect ever turns a delivery into a 3xx failure, and test that the registered URL answers with a 2xx and no hop. Monitor the sender's view of your health separately from your own uptime, and treat the disabled state and the disabling email as pageable incidents. Keep a reconciliation job that pulls from the provider's event log on a schedule, so a gap closes within hours whether or not anyone noticed. And know your retention window cold, because it sets the real deadline for recovery.
The teams that get burned are the ones who believe the push channel is the whole system. It is not. It is the fast path, and the fast path is allowed to fail. The sender told you so the moment it reserved the right to switch you off. Build the slow path that survives when it does.
Frequently asked questions
How long does an endpoint have to fail before a sender disables it? It varies by provider and is rarely a clean number. Stripe retries for up to three days with exponential backoff and can disable an endpoint after sustained failure across that window. GitHub does not retry at all and treats each failed delivery as immediately missed. The safe assumption is that a few days of repeated failures, even intermittent ones spread across routine infrastructure noise, is enough to get you switched off. Do not rely on a threshold you read somewhere, because the sender can change it and your only reliable signal is the endpoint's actual status.
If I re-enable a disabled endpoint, do the missed events get redelivered automatically? Only partially. Re-enabling reopens the channel for new events and, if you act before the next scheduled retry, some in-flight retries resume. Everything whose retry budget expired while the endpoint was off is gone from the push side and will not come back on its own. You have to pull those events yourself from the provider's event log and replay them, which is why an idempotent handler and a reconciliation job matter so much.
Why does returning a 3xx redirect count as a failure?
Because the sender is delivering a payload, not following a link, and a redirect means the payload never reached a handler. Stripe and most other senders treat any 3xx as a failed delivery. This bites hardest when a framework or proxy silently redirects bare HTTP to HTTPS or the apex domain to www, so every delivery fails while your handler code looks perfectly correct. Always register the exact URL that answers with a 2xx and no hop.
Will queueing events instead of processing them inline actually stop me getting disabled?
It removes the most common cause. Disabling is driven by timeouts and error responses, and both usually come from doing slow work, database writes, downstream API calls, inside the request. If you verify, persist, and return 2xx in milliseconds, then process on a worker afterward, your delivery success rate stops depending on how healthy your downstream systems are. A genuine outage of the endpoint itself still fails deliveries, so it is no total guarantee, but it kills the slow-handler failures that catch teams by surprise under load.
Can I detect that I have been disabled without waiting for the email? Yes, and you should. Poll the sender's API for your endpoint's status on a schedule and alarm on any value that is not enabled, so you learn about a disabled endpoint in minutes rather than whenever someone happens to read the notification inbox. Pair that with monitoring the provider's own delivery success rate, which reflects failures that never reached your server and therefore never appeared in your logs. The email is a backstop, not a primary signal.
