Webhook Sender: How to Send Webhooks Reliably, How to Test Them, and How to Receive the Ones You Cannot Reproduce

A webhook sender is the small piece of code on the producer side that posts an HTTP request to a URL when something happens. The interesting work is not the HTTP call (any library can do that). The interesting work is the reliability (retries with backoff, idempotency, dead-letter), the security (HMAC signature so the receiver can verify the sender), and the testing (a tunnel to localhost, a request inspector, a fake receiver). The plain “POST some JSON to a URL” version is five lines; the production version is two hundred. The reason “webhook sender” is a top search is that the five-line version works in development and breaks in production, and the production version is the part the docs skip.

This post is the production version: the reliability layer, the security layer, the testing setup, and the receiver-side story for webhooks you cannot reproduce locally.

Table of contents

• The direct answer: a POST with retries, signing, and a dead-letter queue
• The minimum viable webhook sender
• The reliability layer: retries with backoff
• The security layer: HMAC signature
• The idempotency layer: the event ID
• The dead-letter layer: the queue that catches the unrecoverable
• The testing setup: tunnels, fake endpoints, and request inspectors
• The receiver story: the webhook you cannot reproduce
• FAQ

The direct answer: a POST with retries, signing, and a dead-letter queue

A production webhook sender does four things:

1. Posts the event to a URL with a payload, headers, and a signature.
2. Retries on failure with exponential backoff, for a bounded number of attempts.
3. Signs the payload with a shared secret (HMAC-SHA256), so the receiver can verify the sender.
4. Stores the event in a queue or a database so an unrecoverable failure can be inspected and replayed.

The five-line “POST some JSON” version is fine for a script that runs once. The two-hundred-line version is the right answer for anything that has to survive a network outage, a receiver bug, or a clock skew.

The rest of this post is each of the four layers, in the order they appear in a real production system.

The minimum viable webhook sender

The five-line version, for the record:

import requests

def send_webhook(url, payload):
    requests.post(url, json=payload)

This works. It is also the source of every “the webhook fired but the customer says they did not get it” report. The five-line version has no retries, no signing, no idempotency, no logging, and no dead-letter. The next three layers add those, one at a time.

For a script that runs in a one-off job, the five-line version is fine. For a service that sends webhooks to a customer’s URL as a feature of the product, the production version is mandatory.

The reliability layer: retries with backoff

The reliability layer: every webhook is retried on failure, with exponential backoff, for a bounded number of attempts. The shape:

import time
import requests

def send_webhook_with_retry(url, payload, max_attempts=8):
    for attempt in range(max_attempts):
        try:
            response = requests.post(url, json=payload, timeout=10)
            if 200 <= response.status_code < 300:
                return True  # success
            if response.status_code < 500:
                # 4xx: the receiver said "do not retry"
                return False
        except requests.RequestException:
            pass

        # Exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s
        backoff = 2 ** attempt
        time.sleep(backoff)

    return False  # all attempts failed

The shape: 8 attempts, exponential backoff starting at 1 second, max wait of 128 seconds between attempts. Total max wait: 255 seconds (about 4 minutes). The right answer for most workloads.

The 4xx-versus-5xx distinction matters: a 4xx response means “the receiver said no, do not retry.” A 5xx response or a network error means “the receiver is having a problem, retry.” The code above handles both correctly.

The right max attempts depends on the SLA. For a webhook that has to arrive within 5 minutes, 8 attempts with the backoff above is the right shape. For a webhook that has to arrive within an hour, more attempts with a longer max backoff. For a webhook that has to arrive eventually, a queue-based retry (RabbitMQ, SQS, a database-backed job) is the right answer.

For a hosted equivalent that handles the queue, the retries, and the dead-letter queue, the RunxBuild deploy path is one option. The same pattern as a managed platform: the webhook is a job, the job is queued, the queue handles the retries. The webhook send is the side effect; the platform handles the reliability.

The security layer: HMAC signature

The security layer: every webhook includes an HMAC-SHA256 signature of the payload, computed with a shared secret. The receiver verifies the signature before processing the payload, which prevents an attacker from sending a forged webhook to a receiver who trusts the sender.

The sender:

import hmac
import hashlib

def sign_payload(payload, secret):
    signature = hmac.new(
        secret.encode("utf-8"),
        payload.encode("utf-8"),
        hashlib.sha256,
    ).hexdigest()
    return f"sha256={signature}"

# Send the webhook with the signature header
headers = {
    "Content-Type": "application/json",
    "X-Webhook-Signature": sign_payload(json.dumps(payload), secret),
}
requests.post(url, data=json.dumps(payload), headers=headers)

The receiver:

def verify_signature(payload, signature_header, secret):
    expected = sign_payload(payload, secret)
    return hmac.compare_digest(signature_header, expected)

# In the webhook handler
if not verify_signature(request.body, request.headers["X-Webhook-Signature"], secret):
    return 401

The pattern is symmetric: both sides compute the same HMAC, the receiver compares. The hmac.compare_digest is the right comparison function (constant-time, not vulnerable to timing attacks).

The right secret: a per-receiver shared secret, generated by the sender and given to the receiver out-of-band (in the receiver’s dashboard, in a configuration page). The secret should be at least 32 bytes of random data, and it should be rotated on a schedule (every 90 days is the right default for a high-value webhook).

For a hosted platform that handles the signature automatically (Stripe, GitHub, SendGrid), the pattern is the same. The signature is in a header, the receiver verifies it with the same algorithm, the platform rotates the secret on a schedule. The platform’s webhook is the sender; the developer’s webhook handler is the receiver.

The idempotency layer: the event ID

The idempotency layer: every webhook includes a unique event ID, which the receiver uses to deduplicate. Without the event ID, a retry produces a duplicate webhook, and the receiver has to detect the duplicate by other means (which is fragile and slow).

The shape:

import uuid

event = {
    "id": str(uuid.uuid4()),  # unique per event
    "type": "order.created",
    "created_at": "2026-06-19T12:00:00Z",
    "data": { ... },
}

The receiver stores the event ID and ignores any webhook with an ID it has already processed. The storage is usually a small table (webhook_events with a unique index on id), and the check is a single INSERT ... ON CONFLICT DO NOTHING or the equivalent.

The event ID is the same across retries. A retry of the same event has the same id; a new event has a new id. The receiver dedupes by id, the sender does not have to know which retries the receiver has seen.

The right place to generate the event ID: at the moment the event happens, not at the moment the webhook is sent. The event is the unit of work, and the ID is the event’s identity. The webhook is the delivery mechanism.

The dead-letter layer: the queue that catches the unrecoverable

The dead-letter layer: a webhook that has exhausted all retries is stored in a queue (a database table, a dedicated message queue) for inspection and replay. Without the dead-letter layer, an unrecoverable failure is a silent loss. With the dead-letter layer, the failure is visible, debuggable, and replayable.

The shape:

# After the retry loop returns False
if not success:
    db.execute(
        "INSERT INTO webhook_dead_letter (url, payload, last_error, last_attempt_at) "
        "VALUES (?, ?, ?, ?)",
        (url, json.dumps(payload), str(last_error), datetime.now()),
    )

The dead-letter table is the audit log of webhook failures. The operations team can query it, find the patterns, and either replay the failed webhooks (after fixing the receiver) or contact the customer (if the receiver is permanently broken).

The right place for the dead-letter: the same database the application uses, or a dedicated queue (SQS, RabbitMQ, a Redis list). The right choice depends on the throughput. For most workloads, the database is fine. For high-throughput workloads, the queue is faster.

The testing setup: tunnels, fake endpoints, and request inspectors

The testing setup is the part most teams skip and most teams regret. The setup has three pieces:

A tunnel to localhost. A tool like ngrok, localtunnel, or cloudflared exposes the local dev server to the public internet with a public URL. The webhook sender can post to the public URL, the request lands on localhost, and the developer can debug the receiver locally.

# ngrok
ngrok http 3000

# cloudflared
cloudflared tunnel --url http://localhost:3000

# localtunnel
npx localtunnel --port 3000

The tunnel is the bridge between “the webhook sender lives in the cloud” and “the webhook receiver lives on my laptop.” Without the tunnel, the sender cannot reach the receiver; with the tunnel, the sender can reach localhost as if it were a public service.

A fake endpoint. A tool like webhook.site or requestbin.com is a public URL that records every request it receives. The developer can configure the webhook sender to post to the fake URL and see exactly what the sender sent. The fake endpoint is the right answer for “I want to see what the payload looks like” without involving the real receiver.

A request inspector. The same tools above usually include a request inspector (the headers, the body, the timing). For local development, a tool like httpie or wget can post to the local server and show the request. For production debugging, a tool like tcpdump or mitmproxy can capture the webhook in flight.

The right testing setup is a combination of the three. The tunnel is the bridge; the fake endpoint is the inspector; the request inspector is the verifier. With all three, the developer can debug a webhook in development, in staging, and in production.

The receiver story: the webhook you cannot reproduce

The hardest webhook to debug is the one that fires in production and never in development. The customer reports it; the developer cannot reproduce it; the bug is in the gap between the two.

The diagnostic for the “production only” webhook:

1. Check the production logs for the webhook sender. Every send should be logged with the URL, the payload, the response code, and the timing. If the log says the webhook was sent, the next stop is the receiver.
2. Check the receiver’s logs for the webhook handler. The handler should log every request it receives, with the signature, the body, and the response. If the log says no request was received, the problem is in the network (DNS, firewall, rate limit, geo-block).
3. Check the receiver’s audit trail for the event ID. If the event ID was processed, the receiver did receive the webhook and processed it twice (the deduplication layer failed). If the event ID was not processed, the receiver received the webhook but the handler failed before the dedup write (the handler is the bug).
4. Re-run the webhook from the dead-letter queue. If the dead-letter queue has the event, the developer can replay the webhook and watch the receiver handle it. The replay is the test the production system could not give.

The receiver is where the production-only bugs live. The right answer is a receiver with a structured log, an audit trail, and a replay button. The audit trail is the only record of what happened, and the replay is the only way to reproduce a production bug in development.

For a hosted platform that handles the receiver story (a managed webhook gateway, a tool like Svix), the production-only bugs disappear. The platform logs every webhook, every retry, every response. The replay is a one-click operation. For a team that does not want to build the receiver story, the platform is the right answer. For a team that wants to own it, the patterns above are the floor.

How this fits the rest of the stack

A reliable webhook sender is also a hosting cost — the runtime, the database for the retry queue, the bandwidth, and the storage for the audit log each show up as a line item. The team’s mental model for the webhook cost is the sum of those numbers, and the right answer is the platform that handles the queue and the retry automatically. The RunxBuild hosting calculator is the right place to model that — pick the runtime size, the database tier, the storage, and the expected request rate, and the calculator shows what the webhook sender costs at the team’s actual usage.

Useful related references:

• services layer
• environment variables
• Postgres introduction

FAQ

What is a webhook sender?

A webhook sender is the producer-side code that posts an HTTP request to a URL when something happens. The request includes a payload, headers (often with an HMAC signature), and a unique event ID. The receiver processes the request and deduplicates by event ID.

How do I make webhooks reliable?

Retries with exponential backoff, a bounded number of attempts, a dead-letter queue for unrecoverable failures, and an audit log of every send. The plain “POST some JSON” version is five lines; the production version is two hundred.

How do I test webhooks locally?

A tunnel to localhost (ngrok, localtunnel, cloudflared) is the bridge. A fake endpoint (webhook.site, requestbin.com) is the inspector. A request inspector (httpie, mitmproxy) is the verifier. The combination of the three is the right testing setup.

How do I sign a webhook payload?

Compute an HMAC-SHA256 of the payload with a shared secret, and put the signature in a header (conventionally X-Webhook-Signature). The receiver recomputes the HMAC with the same secret and compares. Use hmac.compare_digest for the comparison, not ==, to avoid timing attacks.

What is an idempotency key in webhooks?

A unique event ID included in the payload. The receiver stores the ID and ignores any webhook with an ID it has already processed. The ID is generated once per event and is the same across retries. The receiver dedupes by ID; the sender does not have to know which retries the receiver has seen.

How do I debug a webhook that fires in production and never in development?

Check the sender’s logs (every send should be logged), then the receiver’s logs (every request should be logged), then the receiver’s audit trail (the event ID was processed or not), then replay the event from the dead-letter queue. The combination of structured logs, an audit trail, and a replay button is the only way to debug a production-only webhook.