You changed one field and lost an afternoon of orders
The migration plan fit on one slide. On Tuesday morning you deployed the receiver at hooks.new-infra.example.com, opened the Shopify app configuration, pasted the new URL and saved. Traffic started landing on the new service within seconds. At lunch you shut down the old one, because paying for two clusters felt silly.
On Thursday finance asked why eleven orders from Tuesday morning never reached the warehouse system. Nothing in the new receiver's logs. Nothing in the old one either, because the old one no longer existed.
Those eleven orders were mid-retry when you changed the URL. The old receiver had been flaky that morning, which was half the reason for moving. Shopify kept retrying the failed deliveries, and it sent every retry to the address it had on file when the event fired. You had just switched that address off.
Retries are addressed when the event fires
Shopify states this outright in its developer changelog: retried webhooks go to the subscription address that was configured when the webhook was triggered. Update the address during a retry cycle and the pending retries still go to the old one. Its troubleshooting docs fill in the rest. A delivery fails if your app takes longer than five seconds to respond, Shopify retries up to eight times within four hours, and if the failures keep coming the subscription is removed.
Shopify wrote down what most senders do and few document. A delivery attempt is a job in somebody else's queue, and the destination was copied into it when the job was created. Editing the endpoint in a dashboard changes what new jobs get. The ones already queued keep the old address.
Some senders do look the URL up again at send time. Assume yours doesn't, and treat the change as applying only to events created after you hit save.
Three kinds of traffic still headed for the old URL
At cutover, three streams of traffic are still moving, and your new receiver's dashboards show none of them.
First, events created in the seconds before you saved. Their first attempt may still be sitting in the sender's dispatch queue, however quiet your old endpoint looked.
Second, the retry backlog. Every delivery that failed in the last few hours or days is waiting for its next attempt. If your old receiver was struggling, and old receivers usually are, the backlog is bigger than you think. Stripe retries live-mode events for up to three days with exponential backoff, so a Stripe backlog can easily outlive the old endpoint.
Third, manual redeliveries. Somebody on your team or the provider's support team clicks "resend" on a failed event from last week. Depending on the sender, that resend goes to whatever URL the original delivery recorded. GitHub doesn't retry failed deliveries on its own, so every GitHub catch-up after an outage is a redelivery somebody triggered by hand or by script.
Work out your drain window before you touch anything
The question that decides your whole plan is how long the old endpoint has to keep answering. Call that the drain window.
For each sender, take the longest retry horizon it documents and add a margin. Shopify: four hours. Stripe: three days. A sender that retries for 24 hours: a day. Then take the maximum across every sender that points at this endpoint. The slowest one sets the deadline.
Write the number in the migration ticket. Once it's there, "turn off the old cluster at lunch" stops looking like a plan.
If a sender doesn't document its retry horizon, don't guess low. Use a week, or ask its support team. A few extra days of an old endpoint cost very little. Eleven vanished orders cost a lot more.
Keep the hostname and move what sits behind it
The cheapest migration never changes the URL at all. If the sender delivers to hooks.example.com, keep that name and change what it resolves or routes to. In-flight retries were addressed to a hostname, so they follow it wherever you point it.
So register webhook URLs on a hostname you own. A cloud provider's generated domain or a tunnel URL somebody pasted in during a hackathon ties every sender to that one piece of infrastructure. A stable name lets you swap the whole receiving stack without telling any sender anything.
Moving behind a stable name has two traps. DNS caches mean some deliveries keep hitting the old target until TTLs expire, so lower the TTL a day ahead and keep the old target alive past it. And the new target must present a valid certificate for that hostname before the first delivery arrives, because a TLS failure is a failed delivery and goes back into the retry queue. If you allowlist sender IPs, the new edge needs the same allowlist on day one.
When the hostname has to change, run both endpoints
Sometimes the URL has to change anyway. A rebrand, a tenant split, a sender with fussy path rules. Then you run two endpoints at once, and the old one stays up for the full drain window.
Where the sender allows several subscriptions per topic, there is a cleaner sequence. Register the new endpoint alongside the old one. For a while every new event goes to both. Confirm the new receiver processes them correctly. Then remove the old subscription and let its backlog drain.
That overlap gives you a test with real traffic and a way back. It also guarantees duplicates.
One dedupe store for both receivers
During a migration you are going to get every event twice. That was the plan. It stays harmless only if both receivers check the same idempotency store before doing any work.
The usual mistake is to give the new receiver a fresh database. Its dedupe table is empty, so it happily processes events the old receiver already handled. Per-instance memory caches are no better. Put the processed-event table somewhere both can reach and key it on the sender's event ID. Delivery IDs won't do, because a forwarded request and a direct request can carry different delivery IDs for the same event.
CREATE TABLE IF NOT EXISTS processed_webhook_events (
provider text NOT NULL,
event_id text NOT NULL,
received_by text NOT NULL, -- 'legacy' or 'new'
received_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (provider, event_id)
);
-- Claim the event before doing any work. Zero rows inserted means
-- the other receiver (or an earlier retry) already has it.
INSERT INTO processed_webhook_events (provider, event_id, received_by)
VALUES ($1, $2, $3)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING event_id;
Write the claim in the same transaction as the business effect if you can. The received_by column feeds the drain metric later. If you already handle duplicates, this is the idempotency work you did before, now shared by two deployments.
A new endpoint usually means a new secret
Many senders issue a signing secret per endpoint. Create a second endpoint and it gets a fresh secret, while deliveries to the old one keep using the old secret.
A new receiver that only knows the new secret rejects every forwarded legacy delivery. One that only knows the old secret rejects every direct delivery. During the drain window the verifier accepts both and records which one matched. The pattern is the same one we described for retiring an old signature after dual-signing: accept two, count each, remove one only once its counter has been at zero for long enough.
Store the old secret with an expiry date in config that your code checks. A comment saying "remove after the 14th" fails no build.
Turn the old endpoint into a dumb forwarder
You don't need the old receiver's full stack to keep accepting deliveries. Replace it with a small forwarder that passes each request to the new receiver and returns whatever status comes back. If the new receiver fails, the forwarder fails too, so the sender retries.
Leave the body alone. Signatures cover the exact bytes the sender transmitted. Parse the JSON and serialize it again and you have changed key order or whitespace, so the signature no longer matches. We went through this failure in detail in the post on transforming payloads at the gateway. Forward the raw bytes and the headers that verification needs.
import express from 'express'
const app = express()
const TARGET = process.env.NEW_RECEIVER_URL! // e.g. https://hooks.new-infra.example.com/shopify
const KEEP = ['content-type', 'x-shopify-hmac-sha256', 'x-shopify-topic',
'x-shopify-shop-domain', 'x-shopify-webhook-id', 'x-shopify-event-id',
'x-shopify-triggered-at', 'x-shopify-api-version']
app.post('/webhooks/shopify', express.raw({ type: '*/*', limit: '2mb' }), async (req, res) => {
const headers: Record<string, string> = { 'x-forwarded-by': 'legacy-endpoint' }
for (const name of KEEP) {
const value = req.get(name)
if (value) headers[name] = value
}
try {
const upstream = await fetch(TARGET, {
method: 'POST',
headers,
body: req.body, // the untouched Buffer
signal: AbortSignal.timeout(4000), // stay under the sender's 5s budget
})
res.status(upstream.status).end()
} catch {
res.status(502).end() // let the sender retry
}
})
app.listen(Number(process.env.PORT ?? 8080))
Mind the four-second timeout. Shopify gives you five seconds in total and the forwarder adds a hop. When the new receiver is slow, fail fast and let the sender retry.
Do not answer 410 on the old URL
When the old path goes quiet, it's tempting to make it return 410 Gone so stragglers get a clear message. Senders only read the status code, and several treat a run of 4xx responses as a reason to stop retrying or to disable the endpoint. Shopify removes subscriptions after persistent failures. We covered how quickly that can go wrong in When the Sender Gives Up on Your Webhook Endpoint.
For a subscription you already deleted, a 410 does no harm. For a live one, you just told the sender to give up on events you wanted. Keep forwarding until the drain window has passed and the subscription is gone. Then retire the route.
Measure the drain
A drain window is an estimate, so check it against what arrives. With received_by in place, one query tells you whether the old endpoint still does real work.
SELECT date_trunc('hour', received_at) AS hour,
count(*) FILTER (WHERE received_by = 'legacy') AS via_legacy,
count(*) FILTER (WHERE received_by = 'new') AS via_new
FROM processed_webhook_events
WHERE provider = 'shopify'
AND received_at > now() - interval '7 days'
GROUP BY 1
ORDER BY 1 DESC;
You want via_legacy at zero for longer than the longest retry horizon. One straggler on day three is an event you would have lost with the old endpoint gone. Count requests at the forwarder too, including ones the dedupe table rejected, so you can tell "nothing arrives" from "only duplicates arrive". The first means you can retire it. The second means it's still catching retries.
Backfill whatever fell through anyway
Even a careful migration can drop something: an event that failed every attempt during the cutover, a subscription removed before you noticed. Plan the backfill before you start. Finance asking is a bad trigger.
Most large senders give you a way to pull what you missed. Stripe lets you list events through its API, going back 30 days. Shopify and GitHub let you query the underlying resources, and GitHub lets you list and redeliver recent deliveries for a hook. Cover the whole migration window plus a margin, and push it through the same dedupe table so anything already processed is skipped. The events API backstop is the same job on a schedule.
Then you have two ways to know you're done. The drain metric says nothing came through the old path, or the backfill finds what did. Either way, nobody is guessing on Thursday.
Frequently asked questions
Does editing the URL on an existing endpoint behave differently from creating a new one? Often it does. Editing usually keeps the same endpoint ID and signing secret, so your verifier needs no change, but queued retries may still carry the old address. Creating a new endpoint gives you a clean overlap period but usually a new secret, so check what your sender documents for each case before you choose.
Can I just return a 301 or 308 redirect from the old URL? Don't rely on it. Many webhook senders don't follow redirects and record a 3xx as a failed delivery. Those that do may not repeat the POST body or the signature headers. A forwarder that proxies the raw request is more predictable and works with every sender.
How long should the old endpoint stay up for Stripe specifically? At least three days after the last event that could have been addressed to it, because Stripe keeps retrying live-mode events for up to three days. Add a margin for manual resends and keep the forwarder running until the drain query shows no legacy traffic for longer than that.
What if the old infrastructure is already gone? Put something back at the old address fast, even a tiny forwarder, because retries are still arriving and each failure brings the sender closer to disabling the subscription. Then run a backfill across the whole migration window through your dedupe table to recover what was lost.