The three guarantees webhooks don't make
A team I worked with spent eight months building a customer sync between their SaaS platform and a billing provider. They subscribed to customer webhooks, wrote handlers for every event type, added dedup logic, and kept a local copy of every customer record. Six months after launch, a support ticket came in: a customer who had cancelled three months earlier was still marked active. The billing provider had sent a customer.deleted event. The platform had never received it.
Not a retry failure. The webhook system was working exactly as designed. What failed was the assumption underneath it: that subscribing to webhooks is enough to keep two systems in sync.
A webhook subscription gives you push notifications on changes, a retry window (usually 24 to 72 hours), and delivery confirmation through HTTP status codes. What it cannot give you is the state of the world before you subscribed, any way to verify that nothing was dropped, or reliable handling of deletions. That last one varies wildly by provider, but even the ones that send deletion events have the other two problems. None of this is a bug in any particular implementation. It is how the push notification model is designed.
Gap one: the bootstrap problem
Every webhook integration starts at the same moment of danger: the handoff from initial import to live event stream. You call the provider's REST API to fetch current state, load it into your database, then enable your webhook subscription. By the time the first webhook arrives, some time has passed since your last API call. Changes that happened during that window are gone.
The window is usually short (seconds to a few minutes) but never zero. And the events that fall into it are undetectable. You cannot ask "what changed between my snapshot and my first webhook?" because the provider has no record of your snapshot timestamp. I have seen this bite teams during migrations where the bootstrap query takes 20 minutes to page through 100,000 records. By the time the subscription is active, the imported state is already wrong.
The fix is to subscribe first, then import. Start receiving webhooks and queuing them before the initial load, then replay the queue after the load finishes. This shrinks the gap to near zero. But it requires idempotent handlers from day one and an import that is atomic enough that half-imported state plus queued events still produces something correct. Most bootstrap implementations get this wrong at least once in production.
Gap two: silent deletions
Most webhook providers treat deletions as an afterthought. Stripe sends customer.deleted. GitHub sends repository.deleted. A surprising number of providers send nothing at all. Some send a resource.updated event where one of the updated fields is a status: "deleted" flag you have to know to check.
Even when a deletion event exists, the failure mode is different from a missed update. Miss an update webhook and you have stale data: the wrong value for a field, but the record still exists. Miss a deletion webhook and you have a ghost. A record that is gone from the source system but still shows up in yours. Ghosts are harder to spot because they look like normal data.
The billing team had no dashboard showing stale-vs-fresh records. The ghost customer looked exactly like any active customer until the support ticket arrived.
Gap three: no way to know what you missed
This is what makes state synchronization a different problem from idempotency or dedup. With duplicates, you know a delivery happened and you deduplicate. With a synchronization gap, you do not know whether an event was missed. The gap is invisible.
If the provider's server failed during delivery and eventually gave up, your system has no record of the attempt. Your monitoring shows no failures. From your endpoint's perspective, nothing arrived. The delivery tried, failed, hit the retry budget, and expired. No cursor, no sequence number, nothing to ask afterward.
Some platforms give you a way in. Stripe's Events API lets you list all events by creation date and scroll through them with a cursor. GitHub Webhooks has no equivalent. Most enterprise SaaS providers fall somewhere between: they offer audit logs that you have to parse yourself.
How most teams discover this
Rarely through monitoring. Usually through a support ticket, an anomaly report, or an audit. A payments team I know found a reconciliation gap during a SOC 2 review: the auditor asked for evidence that every invoice had been accounted for, and the team could not produce it. They had been processing Stripe webhooks correctly for two years. Completeness was simply not something their architecture could prove.
Webhooks give you push delivery with a reasonable success rate. "Reasonable" is not "provable." No receipt survives the retry budget expiry.
The nightly reconciliation cron
Every team that discovers these gaps ends up building the same thing: a background job that periodically fetches the full state from the provider and compares it against the local copy. Records that exist in the provider but not locally, records that exist locally but not in the provider, fields that differ. Those are what webhooks let through.
async function reconcileCustomers(
stripe: Stripe,
db: Pool,
since: Date
): Promise<ReconciliationReport> {
const report = { created: 0, updated: 0, deleted: 0, errors: [] as string[] }
for await (const customer of stripe.customers.list({
created: { gte: Math.floor(since.getTime() / 1000) },
})) {
const local = await db.query(
'SELECT * FROM customers WHERE stripe_id = $1',
[customer.id]
)
if (local.rowCount === 0) {
await upsertCustomer(db, customer)
report.created++
} else if (customer.deleted) {
await db.query('DELETE FROM customers WHERE stripe_id = $1', [customer.id])
report.deleted++
} else if (hasChanged(local.rows[0], customer)) {
await upsertCustomer(db, customer)
report.updated++
}
}
return report
}
It works. It catches what the webhook pipeline missed. The problem is it becomes a permanent architectural dependency. You now run two sync paths: webhooks for near-real-time updates, a polling job for correctness. Both indefinitely. Teams that build this also learn quickly that "run at midnight" is the wrong schedule. The reconciliation window should slide -- compare the last N hours of provider events against local state, continuously, not once a day.
Treating webhooks as pokes, not sources of truth
The shift that helps most teams is treating a webhook as a signal rather than a data delivery mechanism. A webhook tells you something changed. What changed is a separate question you answer by calling the REST API.
async function handleCustomerWebhook(
event: Stripe.Event,
stripe: Stripe,
db: Pool
): Promise<void> {
if (event.type.startsWith('customer.')) {
const customerId = (event.data.object as Stripe.Customer).id
// Fetch fresh from the source of truth, ignore the webhook payload
const customer = await stripe.customers.retrieve(customerId)
if ((customer as Stripe.DeletedCustomer).deleted) {
await db.query('DELETE FROM customers WHERE stripe_id = $1', [customerId])
} else {
await upsertCustomer(db, customer as Stripe.Customer)
}
}
}
This means ignoring the webhook payload for state purposes and using the webhook only to trigger a fresh fetch. You lose the convenience of having the new data already in the payload. What you gain is that what goes into your database came directly from the API, not from a webhook that might have been retried 48 hours after the original event carrying stale state.
The cost is real: every webhook triggers an API call, which adds latency and counts against your rate quota. At high volume you will need to batch or debounce those fetches.
Cursor-based polling: how Stripe does it
Stripe's Events API is the closest thing to a real solution here. It lets you list events in reverse chronological order and page forward with a cursor. Scrolling forward from where you left off gives you an incremental change log without re-fetching the full dataset.
async function pollStripeEvents(
stripe: Stripe,
db: Pool,
lastEventId: string | null
): Promise<string> {
let newestEventId = lastEventId ?? ''
const params: Stripe.EventListParams = {
limit: 100,
...(lastEventId ? { starting_after: lastEventId } : {}),
}
let isFirst = true
for await (const event of stripe.events.list(params)) {
if (isFirst) {
newestEventId = event.id
isFirst = false
}
await processEvent(event, db)
}
return newestEventId
}
It does not replace webhooks. The event list only goes back 30 days, and you still need webhooks for near-real-time delivery. But it is an actual reconciliation mechanism you can run continuously. WorkOS and Linear expose similar cursored streams. Most providers do not.
What to monitor for drift
Running webhooks and a reconciliation job in parallel means you want to notice when the reconciliation job starts doing more work than it should. A spike in discovered differences means the webhook pipeline has a hole.
Track three numbers per reconciliation run: records created (events the pipeline missed entirely), records updated (stale data delivered with incorrect content), records deleted (ghosts the pipeline never cleaned up). Chart those numbers. A healthy system shows near-zero on all three between sweeps. A count that trends upward is a structural problem, not noise.
Set an alert: if reconciliation-discovered creations or deletions exceed 0.5% of your total record count in a single sweep, something is wrong with the webhook path itself, not just occasional network turbulence.
Choosing your architecture
How much drift your use case can tolerate is the actual design question. A payment platform where a ghost "active" customer gets charged is a different problem from a statistics dashboard that is slightly stale.
For low drift: use webhooks as pokes, always fetch from the REST API on each event, and add a short reconciliation sweep (one to four hours) for anything that slips through.
For moderate drift: process webhook payloads directly, but run daily reconciliation against recently modified records using updated_since params where the provider supports them. Webhooks are the fast path; polling is the correctness check.
For zero drift: webhooks are the wrong tool. Use the provider's change data stream if one exists, or a queue-based architecture with delivery guarantees. Stripe's data export and Sigma features exist exactly for this case. For the queue-based side of things, the scalability patterns post covers the infrastructure decisions.
Frequently asked questions about webhook state synchronization
Why don't providers just send a complete snapshot in every webhook payload?
Payload size and delivery latency. A complete customer object with all subscriptions, payment methods, and metadata on every field change would multiply payload sizes by 10x to 100x and make large-object webhooks unreliable to deliver within timeout windows. Sending a thin notification and letting the receiver call the API for the full state is a deliberate tradeoff. The assumption is that receivers who need everything will fetch it.
Can I rely solely on a reconciliation cron job and skip webhooks entirely?
For some use cases, yes. If 15 minutes of stale state is acceptable, a polling job on a short interval is simpler than a webhook pipeline plus reconciliation on top. The limit is API rate quotas: polling aggressively enough to catch changes within 60 seconds burns through provider limits fast on any non-trivial account. Webhooks stay the right choice for near-real-time updates; polling is the correctness layer.
What should I do when a provider sends no deletion events at all?
Pull the full list on a schedule, compare against local state using a set difference, and treat anything present locally but absent from the API as deleted. How often depends on your tolerance for ghost records. Once a day covers most cases. Store the detected-at timestamp alongside the deletion so you know when your system found out, because you will not know when it actually happened.
How do I handle the bootstrap race condition safely?
Subscribe to the webhook endpoint before starting the initial import. Queue incoming events without processing them. Once the import finishes and local state is consistent, replay the queued events in delivery order. Any webhook that arrives before its object exists locally should be handled as an upsert rather than an insert, so the import and the queued event do not conflict.
Is there a standard protocol for cursored event streams across providers?
Not yet. The SCROLL protocol proposed in mid-2026 formalizes cursor-based polling into a standard contract with tombstone events for deletions and optional checksums for replica verification. Stripe's Events API is the closest production implementation today. Standard Webhooks covers delivery and signature conventions but does not address the bootstrap or deletion gaps.
