The Trace Ends at the POST
A customer files a ticket: their subscription upgraded on the billing provider's side, but your app still shows the old plan. You open your tracing tool, and you have a trace. It starts when your handler received the webhook. It shows the database write, the cache invalidation, the confirmation email. Everything is green. The event was processed cleanly forty minutes ago.
So where is the problem? Not in your trace, because your trace begins at the moment the request arrived. Everything the provider did before that is invisible to you: the internal event that fired, the queue it sat in, the delivery attempt that timed out, the retry that landed. And everything you did is invisible to them. Two teams, two tracing systems, one event, and a wall down the middle. That wall is the webhook.
Traceparent in Sixteen Bytes of Hex
Inside a single system, distributed tracing works because one header rides along with every call. The W3C standardized it as Trace Context, a Recommendation since November 2021, and its main field is traceparent. It looks like this:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
Four dash-separated fields. The 00 is the version. The 32-hex-character block is the trace ID, sixteen bytes that identify the whole end-to-end journey. The 16-hex-character block is the parent span ID, eight bytes naming the specific operation that made this call. The trailing 01 is the flags byte, and its low bit is the sampled flag: it says whether anyone bothered to record this trace at all.
There is a companion header, tracestate, that carries vendor-specific key-value pairs so two tracing systems can hand off proprietary data without stepping on each other. For webhook work you mostly care about traceparent. Get that across the boundary and a span on your side can declare the provider's span as its parent.
Two Traces That Never Introduce Themselves
Here is the thing most people miss. The provider is almost certainly tracing their own webhook delivery. Stripe, GitHub, and Shopify run enormous distributed systems, and internally they know exactly which event object produced which delivery attempt against which endpoint. They have a trace ID for it.
You have a trace ID too, minted the instant your framework accepted the connection. Both of these are real, both are queryable, and neither knows the other exists. When you debug a lost or duplicated event, you are looking at half the picture and quietly assuming the other half is fine. Usually it is not fine. The interesting failures live in the gap between "the provider decided to send" and "your code started running," which is exactly the stretch neither trace covers.
The Header You Probably Will Not Receive
The clean fix would be for the provider to inject their traceparent into the outbound webhook request. A few do. Most do not, and the ones that do often regenerate it per delivery attempt, so a retry gets a fresh trace ID unrelated to the first attempt.
Do not count on the header. Even if a provider sends one today, it is not part of any signed payload, so a payload-transformation proxy in front of you can strip it without warning, the same way it can quietly break other headers. Treat an inbound traceparent as a bonus you verify, never as the backbone of your correlation strategy. The backbone has to be something the provider guarantees, and the only thing they truly guarantee is the event ID.
Carrying Context in the Payload Envelope
If you own both ends, and by that I mean your platform sends webhooks to your customers, or one internal service posts to another, put the trace context in the payload, not just the headers. Headers get rewritten by proxies and load balancers. The signed body does not, because rewriting it breaks the signature.
A minimal envelope looks like this:
{
"id": "evt_2Nc8s9",
"type": "invoice.paid",
"trace": {
"traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
},
"data": { "invoice_id": "in_9Kd21" }
}
Now the trace context is covered by your HMAC signature. A receiver that verifies the signature also gets a trustworthy parent to attach to. And because the field is inside the event you retain for replay, you can reconstruct the linkage days later when the live trace has already aged out of your backend's retention window.
Resuming the Span on Your Side
Extracting the context is not a string-parsing exercise you should hand-roll. The OpenTelemetry API ships a propagator that reads a carrier and returns a context you can start a span inside:
import { propagation, context, trace } from '@opentelemetry/api'
const tracer = trace.getTracer('webhook-receiver')
export async function handleWebhook(event: WebhookEvent) {
const carrier = { traceparent: event.trace?.traceparent ?? '' }
const parentCtx = propagation.extract(context.active(), carrier)
await context.with(parentCtx, async () => {
const span = tracer.startSpan(`webhook ${event.type}`)
span.setAttribute('webhook.event_id', event.id)
try {
await process(event)
} finally {
span.end()
}
})
}
The span you start now lists the provider's span as its parent. Open either tracing tool and you can walk from their delivery attempt straight into your database write, across the org boundary, in one continuous picture. That is the whole point. Not prettier dashboards. The ability to answer "what happened to this event" without guessing which side broke.
One Event, Five Retries, Five Traces
Retries are where naive correlation falls apart. A provider that regenerates traceparent per attempt gives you five different trace IDs for what is, to you, one logical event delivered five times. If you key your traces on traceparent alone, you get five unrelated stories and no thread connecting them.
Key on the event ID instead. The event ID is stable across every retry: Stripe's event ID, Shopify's X-Shopify-Webhook-Id, the webhook-id in a Standard Webhooks header. Set it as a span attribute on every attempt, as webhook.event_id, and your tracing tool can group all five attempts even when their trace IDs differ. This is the same identifier your idempotency layer already uses to reject duplicate processing, so you are not inventing a new key, just spending an existing one twice. If you have not built that layer yet, the mechanics are in /blog/webhook-idempotency-keys-duplicate-processing.
The Queue Is Another Boundary
Most serious receivers do not process inline. They verify, enqueue, and return 200 fast, then a worker picks the job up later, for good reasons covered in /blog/webhook-scalability-patterns-queue-based-architecture. That enqueue is a second trace boundary, and it is one you own entirely, so there is no excuse for dropping context here.
Inject the current context into the queue message the same way you extracted it from the webhook:
function enqueue(event: WebhookEvent) {
const carrier: Record<string, string> = {}
propagation.inject(context.active(), carrier)
return queue.send({
event,
traceparent: carrier.traceparent,
})
}
The worker extracts it back out and continues the span. Skip this and every trace stops at the queue's front door, and you are back to two disconnected halves, this time both of them yours. A surprising number of "the provider never sent it" incidents turn out to be an event that arrived fine and then vanished silently between the endpoint and the worker.
Sampling Decides Before You Do
That trailing flags byte matters more than it looks. If the sampled bit is 00, the upstream decided not to record this trace, and a well-behaved receiver honors that decision, so your span is created but never exported. Which means the one webhook you desperately want to inspect might be the one nobody kept, because a sampler forty minutes and one company ago rolled the dice against it.
For webhooks, blanket head-based sampling is the wrong default. The volume is low compared to your API traffic, and the value of each trace during an incident is high. Sample webhook traces at or near 100 percent, or use tail-based sampling that keeps everything with an error or an abnormal duration. Paying to store a few million cheap webhook spans is nothing next to the cost of the one you needed being gone.
When You Cannot Trace, Correlate
Plenty of providers will never send you a traceparent, and you do not own their side. You still are not helpless. Log the event ID, the delivery attempt number if the provider exposes one, and your own trace ID together in one structured line, so a human can jump from a provider's dashboard entry to your logs by event ID alone.
{"level":"info","msg":"webhook received","event_id":"evt_2Nc8s9","attempt":3,"trace_id":"4bf92f35...","event_type":"invoice.paid"}
This is manual correlation, and it is worse than a real trace, but it turns a support ticket from a shrug into a query. The provider's UI gives you the event ID; your logs give you everything you did with it. The discipline is putting the same identifiers on both sides of the wall, which is really just structured logging done with the boundary in mind. The habits in /blog/webhook-logging-best-practices-structured-debugging carry straight over.
Frequently asked questions
Should I trust a traceparent header a provider sends me? You can attach to it, but verify it first. The header sits outside the signed payload, so a proxy can rewrite it and an attacker who reaches your endpoint can forge one. Attach your span to an inbound header for convenience, but key your grouping and idempotency on the signed event ID, which the provider actually guarantees.
What if the provider regenerates the trace ID on every retry? Then the trace IDs are useless for grouping and the event ID is your anchor. Set the event ID as a span attribute on every attempt so your tracing backend can collect all retries of one logical event, even though each attempt carries a different trace ID and looks like a separate story.
Does putting trace context in the payload leak anything sensitive? A traceparent is random hex identifying a trace, not user data, so exposing it is low risk. The real caution is that anything in the body is covered by your signature and retained with the event, so keep the trace object to the standard fields and do not smuggle internal hostnames or account identifiers into tracestate.
Why is the webhook I want to debug missing from my traces? Most likely a sampler upstream cleared the sampled flag, and your receiver honored that decision, so the span was created but never exported. Raise webhook sampling toward 100 percent or switch to tail-based sampling that retains errors and slow deliveries, because webhook volume is low and each trace is worth keeping.
Can I do any of this without a provider that supports tracing? Yes. Full context propagation needs cooperation you often will not get, but manual correlation needs nothing from the provider. Log the provider's event ID next to your own trace ID in one structured line and you can pivot from their dashboard to your logs by event ID, which recovers most of the debugging value on your own.
