The events arrived. Just not in the order you sent them.
A customer updates their subscription three times in ten seconds. They upgrade to Pro, add a seat, then downgrade back to Starter because they misclicked. Your platform fires three webhooks in that exact order. The receiving system applies them and lands on Pro with an extra seat. The customer is now paying for a plan they cancelled.
Nobody wrote a bug. The three customer.subscription.updated events left in the right order and arrived in the wrong one. The downgrade got retried once after a transient timeout, and by the time it landed the upgrade behind it had already been processed. The final state is whatever event won the race, not whatever the customer actually did last.
This is the ordering problem, and it bites almost everyone who treats a webhook stream as a reliable sequence. You cannot file it as a delivery bug against your provider. It is a property of how webhooks work, and the sooner you design around it the less it costs you.
Why "just send them in order" doesn't fix anything
The obvious fix is to make the sender emit events in strict sequence and call it done. Providers mostly do try to send in order. It changes almost nothing, because ordering breaks on the receiving side just as easily as on the wire.
Say two events leave the sender one millisecond apart, in order, and arrive at your endpoint in order. Your handler for the first event does real work. It reads a row, calls a downstream billing API, writes back. The handler for the second event is a cheap metadata update that finishes in twenty milliseconds. The second event is done before the first one commits. Two events, delivered in order, processed backwards, and no network reordering was involved.
Now add the thing every serious consumer does. You accept the webhook, push it onto a queue, return 200, process asynchronously. The moment you have more than one worker draining that queue, order is gone. Worker A picks up event one, worker B picks up event two, and whichever finishes first writes first. Svix put the ceiling bluntly in their own guidance: if you wait for each webhook to finish before starting the next, you are "essentially limited to one webhook per second." Nobody runs production that way, so nobody preserves order for free.
Sending in order is necessary and nowhere near sufficient. The order you care about is the order of effects, and effects are ordered by your consumer, not by the sender.
The three places order gets lost
It helps to know exactly where the sequence falls apart, because each spot has a different fix.
The first is retries. A webhook that fails and gets retried lands later than events that were emitted after it. A retry schedule with exponential backoff can push a single failed delivery minutes behind its neighbours. This is the same machinery that makes delivery reliable in the first place. See webhook retry strategies for how backoff and dead-letter queues actually behave. Reliability and ordering pull against each other directly. You cannot have aggressive retries and strict order at the same time without blocking.
The second is parallelism. Any fan-out, any multi-worker consumer, any load-balanced set of endpoint replicas processes events concurrently. Concurrency is the entire point of scaling a consumer, and concurrency has no opinion about order.
The third is handler duration variance. Even single-threaded, in-order intake produces out-of-order commits when handlers take different amounts of time, as in the billing example above. Fast events overtake slow ones.
None of these are the sender's fault, and none are fixable by the sender. Two of them are things you chose to do to make your consumer fast and reliable.
Timestamps and version counters beat sequence numbers
The durable fix is to stop relying on arrival order. Rely instead on data in the payload that tells you what is newer. Svix's recommendation is direct: include "the entity's modification date (or modification counter) in the payload, so the customer can check whether this event is newer or older than what they currently have stored."
A monotonic version counter is the stronger of the two. Every mutation to an entity increments a version field. Your handler applies an event only if its version is greater than the version you already have stored:
UPDATE subscriptions
SET plan = $1, seats = $2, version = $3
WHERE id = $4
AND version < $3;
If the row's stored version is already at or beyond the incoming event, the WHERE clause matches nothing and the stale event is a no-op. The downgrade that arrives after the upgrade simply does not apply, because its version is lower. No coordination, no locking, no ordering guarantee required. This is a conditional write, and it is the single highest-leverage change you can make.
Timestamps work the same way but carry a caveat. Two events in the same millisecond, or clock skew between sender nodes, can make "newer" ambiguous. If the provider gives you a version counter, prefer it. If all you get is a timestamp, use it and accept that same-instant events need a tiebreaker, usually the event ID.
Do not use the provider's per-message sequence number as your ordering key unless the provider explicitly guarantees it is gapless and per-entity. Most sequence numbers are per-stream, not per-object, and a global sequence tells you nothing about which of two events touching your row is newer.
Thin payloads turn ordering into a non-problem
There is a design that sidesteps ordering almost entirely. Send thin payloads. Instead of packing the full changed entity into the webhook, the provider sends "identifiers and some additional metadata (like which properties have changed)," and the consumer fetches the current state from the API when it processes the event.
Think about what this does to the out-of-order downgrade. With thin payloads, all three subscription events say the same thing: "subscription sub_123 changed, go look." When your handler processes any of them, it calls GET /subscriptions/sub_123 and reads the true current state right now. It does not matter which event you process first, last, or twice. Every handler converges on the same answer, because the answer comes from the source of truth and not from a payload frozen at emit time.
The cost is read amplification. Every event becomes an API call, and a burst of events for one entity becomes a burst of redundant reads you can coalesce. For high-value state where correctness matters more than webhook-only throughput, like billing or access control, that trade is almost always worth it. It also closes the state-sync gap that fat-payload webhooks leave open, because you pull truth rather than reconstruct it from a sequence of deltas.
Idempotency is the floor, not the fix
People reach for idempotency as an ordering solution and are disappointed. Idempotency and ordering solve different problems, and you need both.
Idempotency makes processing the same event twice safe. Ordering makes processing different events in the wrong sequence safe. An idempotency key deduplicates the retried downgrade so you do not apply it twice. It does nothing about the fact that the downgrade landed after the upgrade. Dedup on event ID, order on version counter. They are complementary. If you only have room in your head for one rule about consuming webhooks, make it this: dedup by ID, decide by version. The mechanics of the first half are covered in idempotency keys and duplicate processing. This post is the second half.
The reason the two get conflated is that a well-built idempotent handler and a well-built order-tolerant handler look similar. Both are conditional writes guarded by something in the payload. But one guards on "have I seen this ID" and the other guards on "is this newer than what I have." Write both conditions and stale duplicates and out-of-order events both become no-ops.
When you actually need strict ordering
Sometimes the effects genuinely cannot commute. A document collaboration stream where operations must apply in sequence. A financial ledger where each entry depends on the running balance. Cases where "apply newest, ignore older" is wrong because every event matters and every event builds on the last.
For those, providers have started offering FIFO endpoints. Svix ships them for Dispatch, for operational webhooks, and for all Ingest customers. The guarantee is exactly what it says. Strict first-in-first-out delivery, per endpoint, with no code changes on the sender side. The mechanism is blunt but effective: "every call to the receiver endpoint is blocked until the previous one is successful." Message N+1 does not leave the queue until message N has been acknowledged.
That blocking is the whole guarantee and the whole cost. It converts your delivery pipeline from parallel to serial, which is the only way to preserve order across a network with retries. If order is load-bearing, this is the right tool, and building the equivalent yourself, a per-entity serialized queue with exactly-once handoff, is a lot of infrastructure to get correct. Reaching for FIFO connects to the broader question of delivery guarantees, where the same at-least-once-with-idempotent-consumers reasoning applies.
What FIFO costs you
Strict ordering is not free, and the bill comes as throughput and fragility. Because each delivery blocks the next, a FIFO endpoint's rate is capped by your handler's per-message latency. Slow handler, slow endpoint, and there is no parallelism to hide behind.
Worse is head-of-line blocking. If message N cannot be delivered, because your endpoint is down or that one payload trips a handler bug, every message behind it waits. A single poison message stalls the entire ordered stream, not just itself. With regular best-effort webhooks, a stuck event is one failed delivery on its own retry schedule while everything else flows past. With FIFO, one stuck event is a dam.
Svix softens the throughput hit with configurable batching. FIFO endpoints "deliver webhooks in configurable batch sizes," so you amortize the round trip over many messages instead of paying it per message. That helps throughput. It does not remove head-of-line blocking, and it makes your handler responsible for processing a batch atomically and in order internally.
Here is the honest framing. FIFO endpoints trade throughput and failure isolation for ordering. If your effects commute, you are paying that price for a guarantee you did not need. Reach for FIFO when order is genuinely load-bearing. Reach for version counters and thin payloads everywhere else, which is most places.
Designing consumers that don't care about order
The goal is a consumer that produces the same correct final state no matter what order events arrive in. That property has a name, commutativity, and it is worth designing for on purpose.
Start by asking, for each event type, whether applying it out of order can produce a wrong result. State-replacement events ("the subscription is now X") commute naturally when guarded by a version check. Apply the newest, ignore the rest. Delta events ("add one seat") do not commute, because the order of additions and removals matters and a lost or duplicated delta corrupts the count. Where you can, model events as absolute state rather than deltas. Where you cannot, carry a version so you can detect a gap and reconcile.
Then make every handler a conditional write. Guard on the idempotency key to kill duplicates, guard on the version to kill stale writes, and let both be no-ops when their condition fails. A handler built this way does not need the stream ordered, does not need FIFO, and does not care whether a retry landed late. It is the cheapest ordering strategy there is, because it is not an ordering strategy at all. It is a consumer that survives disorder. Stop trying to make the stream behave, and build a consumer that is right regardless of how it behaves. The stream will never fully behave.
Frequently asked questions
If my provider says it sends webhooks in order, can I rely on that order? No. In-order sending is best-effort and, more importantly, only controls when events leave the sender. Retries, parallel workers, and handlers that take different amounts of time all reorder the effects on your side. Treat "sent in order" as a courtesy, not a guarantee, and build your consumer to tolerate reordering with a version check.
What is the difference between idempotency and ordering, and do I need both? Idempotency makes processing the same event twice safe. Ordering makes processing different events in the wrong sequence safe. They solve different failure modes and you need both. Deduplicate on the event ID to handle retries, and gate writes on a version counter or modification timestamp to ignore stale events. A handler that does both is safe against duplicates and disorder at once.
Should I use FIFO endpoints to guarantee correct order? Only when your events genuinely cannot commute, such as a ledger or a collaborative document stream. FIFO guarantees strict order by blocking each delivery until the previous one succeeds, which caps throughput at your handler's latency and means one stuck message halts everything behind it. If applying the newest event and ignoring older ones is correct for you, version-guarded conditional writes are cheaper and more resilient.
What are thin payloads and why do they help with ordering? A thin payload carries only identifiers and a hint about what changed, and the consumer fetches current state from the API when it processes the event. Because every handler reads the source of truth at processing time, the order the events arrive in stops mattering. They all converge on the same current state. The cost is an extra API read per event, which you can coalesce for bursts on the same entity.
Can I use the provider's sequence number as my ordering key? Usually not. Most sequence numbers are per-stream, not per-entity, so a higher number does not mean "newer version of the row you care about." Use a per-entity modification counter or timestamp from the payload instead. Only rely on a sequence number if the provider explicitly documents it as gapless and scoped to the individual object.
