The Header That Was Never Meant to Travel
In April 2026, GitHub sent an email to webhook administrators disclosing something that had been silently happening since September 2025. Between September 11 and December 10, 2025, and again briefly on January 5, 2026, webhook deliveries from GitHub included an X-Github-Encoded-Secret header containing each webhook's secret encoded in base64. The bug was fixed on January 26, 2026, about two weeks after its second appearance, and disclosed around three months after that.
The header was an internal routing artefact from a new delivery platform version that slipped out onto the wire. GitHub stated it had no evidence of interception and that GitHub itself was not compromised. What it could not rule out was that receiving systems had logged the incoming request headers, capturing the secret in the process.
If you have ever written a webhook handler, you know how common full-header logging is. The first thing most developers do when debugging a webhook is log the entire incoming request. That habit is exactly what makes a secret-in-header leak extend beyond the platform's control.
What a Leaked Secret Actually Enables
A webhook secret's job is to let the receiver verify that a payload came from the legitimate sender. The typical mechanism is an HMAC: the sender hashes the payload with the shared secret and sends the hash as a header. The receiver recomputes the hash and compares. If the hashes match, the payload is authentic.
Once an attacker has the secret, that falls apart. They can compute a valid HMAC for any payload they construct. They can replay old deliveries with modified fields. If your receiver does not enforce timestamp validation or track seen event IDs, it will accept forged events as genuine. A system processing push events from a leaked secret could be tricked into triggering a deployment pipeline with attacker-controlled content in the payload.
The base64 encoding of the leaked header adds nothing. Base64 decodes in a single function call, and the exposure is equivalent to sending the raw bytes.
Auditing Your Exposure Window
For the GitHub incident, the relevant window ran from September 11, 2025 to January 26, 2026. If your endpoint was active then and your application or infrastructure logs HTTP request headers, those logs may contain X-Github-Encoded-Secret lines.
Start with application logs. Search for the header name:
grep -r "X-Github-Encoded-Secret" /var/log/your-app/ 2>/dev/null
Then check the infrastructure layer. Nginx, Apache, and cloud load balancer access logs record request headers only when configured to do so, but some observability stacks capture them by default when ingesting full HTTP metadata for tracing. Check your APM platform, log aggregation system, and any request-capture middleware added during a debugging session that never got removed.
Check your webhook delivery logs too. Some teams store raw delivery payloads and headers in a database table for replay and debugging. If you do that, the leaked header may be sitting in your own database right now.
How Webhook Secrets End Up in Logs
The GitHub incident is a sender-side leak, but secrets reach logs from the receiver side just as often. Two paths show up repeatedly.
Full request logging gets added during development and almost never removed. A middleware that logs JSON.stringify(req.headers) captures every header, including X-Hub-Signature-256 values from GitHub and Stripe-Signature values from Stripe. Those headers carry HMACs, not the raw secret, so they do not enable forgery directly. But a captured HMAC of a known payload does narrow the search space for anyone attempting to recover the key.
Error reporting is subtler. Many crash-reporting tools capture request context automatically when an exception fires. If your webhook handler throws during processing and your error reporter ships the request headers to an external service, the secret lands in a third-party system you did not intend to trust with it. Worth checking your error reporter's data capture settings before an incident makes you.
Zero-Downtime Rotation: Accepting Two Secrets at Once
The operational problem with secret rotation is that deliveries in flight during the changeover carry the old signature. Replace the secret and immediately start rejecting the old HMAC, and you will reject any delivery the sender queued before the rotation. The sender signed it with the old key, your receiver holds only the new key, verification fails.
A dual-validation window solves this: accept a delivery as valid if it passes with either the old or the new secret for a period after rotation. The length depends on the sender's maximum retry delay. GitHub retries failed deliveries for up to 30 days, but the practical concern is the initial delivery and its first few retries, which happen within minutes.
import * as crypto from 'crypto'
function verifySignature(
payload: Buffer,
signatureHeader: string,
secrets: string[]
): boolean {
const [algorithm, receivedHash] = signatureHeader.split('=', 2)
if (algorithm !== 'sha256') return false
for (const secret of secrets) {
const expected = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex')
const expectedBuffer = Buffer.from(`sha256=${expected}`, 'utf8')
const receivedBuffer = Buffer.from(signatureHeader, 'utf8')
if (
expectedBuffer.length === receivedBuffer.length &&
crypto.timingSafeEqual(expectedBuffer, receivedBuffer)
) {
return true
}
}
return false
}
// During rotation, pass both secrets.
// After the window closes, pass only the new one.
const valid = verifySignature(body, req.headers['x-hub-signature-256'], [
process.env.WEBHOOK_SECRET_NEW!,
process.env.WEBHOOK_SECRET_OLD!,
])
Keep the old secret in configuration for 24 to 48 hours after rotating. Most senders exhaust their immediate retry window well within that period. After it closes, remove the old secret from the array.
Rotating on GitHub
GitHub's webhook secret lives per-webhook under Settings, Webhooks, Edit. Generate a new secret locally before touching anything on GitHub's side:
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
Deploy the dual-validation configuration to your receiver first. Once your receiver accepts both secrets, update the webhook configuration on GitHub. After 48 hours, deploy a follow-up removing the old secret from your receiver's list.
GitHub lets you re-deliver recent webhook events from the webhook settings page. After rotating during a security incident, re-delivering events from the exposure window confirms receipt and closes any gap left by deliveries that failed while you were mid-rotation.
Rotating on Stripe, Shopify, and Others
Stripe's webhook signing secrets begin with whsec_ and are managed per-endpoint in the Stripe Dashboard under Developers, Webhooks. Stripe has no built-in dual-secret API, but you can create a second webhook endpoint pointing at the same URL before deleting the compromised one. Both will deliver to your receiver for long enough to cover the transition.
Shopify webhook secrets are set at the app level, not per-topic. The secret appears in your Partner Dashboard under app configuration. Rotating means updating that value and redeploying before the old secret is invalidated.
The rotation procedure is the same across platforms in structure: update your receiver first, update the platform second, remove the old secret after the retry window. The retry windows differ. Stripe retries for 72 hours, GitHub for 30 days, Shopify for 48 hours. Size your dual-validation window to the longest one that applies to your situation.
Checking Your Own Sender for Header Leaks
If you run a webhook sender, the GitHub incident is a prompt to audit your own delivery pipeline. Internal headers used for routing, tracing, or debugging can escape onto the wire if they are added upstream of the egress filter.
Capture a delivery with a known test endpoint and inspect every header your sender includes. RequestBin or an ngrok tunnel with header inspection works fine for this. Look at any header containing the words secret, key, token, signature, or credential, and any header whose value looks like a base64 string.
Strip internal headers at the egress point of your delivery worker, not at request construction. Headers added by an HTTP client library, a tracing framework, or the platform runtime can survive into the outbound request if you only filter at the construction step.
What Not to Log on the Receiver Side
A receiver that does not log full headers cannot be the source of a secret leak regardless of what the sender does. The practical guidance is selective: log the headers you actually use for debugging, redact or drop the ones carrying secret material.
For a GitHub webhook receiver, the headers worth keeping in logs are X-Github-Event, X-Github-Delivery, and Content-Type. You do not need X-Hub-Signature-256. It is ephemeral per-delivery and carries no debugging value after verification passes. If verification fails, log that fact and the event type, not the signature value.
If your observability stack captures request metadata automatically, check whether it redacts sensitive headers before shipping them to your aggregation system. Most APM vendors support a scrub list. Add X-Hub-Signature-256, Stripe-Signature, Svix-Signature, and any platform-specific signature header to it.
When to Treat Rotation as Urgent vs Routine
Periodic rotation as a hygiene measure every six to twelve months is low-stakes. You set the dual-validation window, deploy both secrets, update the platform, wait, clean up. No incident mode required.
Rotation after a confirmed or suspected exposure is different. Speed matters because an attacker with the secret can act at any time. Notify your team first. Deploy the dual-validation window while evaluating blast radius. Update the platform as fast as your deployment pipeline allows, then audit logs immediately. The 24-to-48-hour dual-validation window stays the same; the urgency of everything around it does not.
The GitHub incident was neither. It was a sender-side leak disclosed months after the fact, with no evidence of active exploitation. Teams active during the exposure window needed to rotate, but the urgency was lower than a live compromise. The specific date range GitHub provided was what made it possible to scope the audit properly. When a platform discloses a leak with a defined window, start there.
Frequently asked questions about webhook secret rotation
Why did GitHub wait three months to disclose the leak?
Disclosure timing after a security incident reflects the platform's internal investigation timeline, remediation confirmation, and legal or regulatory review processes. GitHub fixed the bug on January 26, 2026 and disclosed in April 2026, a gap of roughly two to three months. This is within the range of typical responsible-disclosure timelines for a bug that the platform itself discovered and fixed, though it does mean receiver teams had no opportunity to act during the exposure window. The practical consequence is that any audit of logs or delivery history needs to cover the full window, not just the period after you became aware of it.
Can I verify whether my endpoint actually received deliveries during the exposure window?
GitHub's webhook delivery log, accessible from Settings - Webhooks - Recent Deliveries, shows the last 250 deliveries and their timestamps. If your webhook was less active than that during the window, you can see exactly which deliveries landed. For higher-volume webhooks, the delivery log will not reach back to September 2025, so you will need to rely on your own application or infrastructure logs filtered by the relevant date range.
Does base64-encoding the secret before sending it provide any protection?
No. Base64 is an encoding, not encryption. Anyone who sees the encoded value can recover the original secret in one function call. The encoding was almost certainly incidental, an internal representation that escaped into the outbound header, rather than a deliberate obfuscation attempt. Treat a base64-encoded webhook secret in a header as equivalent to the raw secret being present.
How long should I keep the old secret active during rotation?
Size the dual-validation window to the sender's maximum retry period, not to your own preferences. GitHub retries for up to 30 days, but the vast majority of deliveries either succeed or exhaust their immediate retries within the first hour. A 24-to-48-hour window covers the practical retry tail for GitHub and most other major platforms. After that window closes, remove the old secret. Leaving both secrets active indefinitely means a compromise of the old secret is still a compromise of your receiver.
