Reliability, Idempotency & Queue Patterns

Networks fail, databases lock up, and server deployments happen. Webhook systems guarantee at-least-once delivery, which means your backend will inevitably receive duplicate, delayed, or out-of-order events.

Building a reliable webhook consumer requires designing for failure from day one.

1. The 4 Webhook Failure Scenarios

2. The Idempotency Problem

Idempotency means that performing an operation once produces the exact same outcome as performing it multiple times.

❌ The Real-World Double-Fulfillment Bug:

Customer pays $100 for an order.
Stripe sends payment_intent.succeeded.
Your server takes 35 seconds to process PDF generation and send emails.
Stripe's HTTP client times out after 30 seconds and flags the delivery as failed.
Stripe automatically retries 1 minute later.
Your backend processes the second webhook and ships two items for a single payment.

3. Implementing Idempotent Handlers in Node.js

To make your handlers idempotent, record the unique event.id inside a database transaction or a Redis key before processing:

PostgreSQL Idempotency with Transactions

import pg from "pg";
 
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
 
async function processWebhookIdempotent(event) {
  const client = await pool.connect();
 
  try {
    await client.query("BEGIN");
 
    // 1. Try to record the event ID in processed_events table
    // ON CONFLICT DO NOTHING ensures duplicates are cleanly ignored
    const insertRes = await client.query(
      `INSERT INTO processed_webhooks (event_id, event_type, created_at)
       VALUES ($1, $2, NOW())
       ON CONFLICT (event_id) DO NOTHING
       RETURNING event_id;`,
      [event.id, event.type]
    );
 
    // If 0 rows were inserted, this event was already processed previously
    if (insertRes.rowCount === 0) {
      console.log(`⚠️ Duplicate event detected: ${event.id}. Skipping.`);
      await client.query("ROLLBACK");
      return { status: "duplicate_skipped" };
    }
 
    // 2. Perform business logic safely inside the same transaction
    switch (event.type) {
      case "payment_intent.succeeded":
        await fulfillCustomerOrder(client, event.data.object);
        break;
      case "customer.subscription.deleted":
        await cancelUserSubscription(client, event.data.object);
        break;
    }
 
    await client.query("COMMIT");
    return { status: "processed" };
  } catch (error) {
    await client.query("ROLLBACK");
    throw error;
  } finally {
    client.release();
  }
}

4. The Async Queue Pattern (Respond Fast, Process Async)

Most providers (Stripe, Shopify, GitHub) expect an HTTP 200 response within 5 to 10 seconds. Heavy tasks (video encoding, PDF generation, sending emails) should be pushed to a background queue.

Implementation with Bull and Express

npm install bull ioredis

import express from "express";
import Queue from "bull";
 
const app = express();
const webhookQueue = new Queue("incoming-webhooks", process.env.REDIS_URL);
 
// Webhook endpoint: Verify -> Enqueue -> Return 200 immediately
app.post(
  "/webhooks/stripe",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    // 1. Verify signature
    const event = verifyStripeSignature(req);
 
    // 2. Push to queue with automatic retry policies
    await webhookQueue.add(
      "process-stripe-event",
      {
        eventId: event.id,
        eventType: event.type,
        payload: event.data.object
      },
      {
        jobId: event.id, // Bull deduplicates jobs sharing the same ID!
        attempts: 5,
        backoff: {
          type: "exponential",
          delay: 2000 // 2s, 4s, 8s, 16s...
        }
      }
    );
 
    // 3. Respond in milliseconds
    res.status(200).json({ received: true });
  }
);
 
// Worker processes jobs in the background
webhookQueue.process("process-stripe-event", async (job) => {
  const { eventId, eventType, payload } = job.data;
  console.log(`Processing job ${job.id} for event ${eventType}...`);
 
  // Run business logic
  await handleStripeEvent(eventType, payload);
});

5. Dead Letter Queues (DLQ) for Failed Events

When a webhook fails even after all 5 retry attempts, you must not discard it. Route it to a Dead Letter Queue (DLQ) and trigger an alert.

const deadLetterQueue = new Queue("webhook-dlq", process.env.REDIS_URL);
 
// Move failed jobs to DLQ on final failure
webhookQueue.on("failed", async (job, error) => {
  if (job.attemptsMade >= job.opts.attempts) {
    console.error(`🚨 Webhook ${job.data.eventId} failed all retries:`, error.message);
 
    await deadLetterQueue.add("poison-pill", {
      originalJob: job.data,
      errorMessage: error.message,
      failedAt: new Date().toISOString(),
      attemptsCount: job.attemptsMade
    });
 
    // Send notification to Slack or PagerDuty
    await notifyOpsTeam(`Webhook failed permanently: ${job.data.eventId}`);
  }
});
 
// Admin Endpoint to replay DLQ events
app.post("/admin/webhooks/dlq/:id/replay", async (req, res) => {
  const job = await deadLetterQueue.getJob(req.params.id);
  if (!job) return res.status(404).json({ error: "Job not found in DLQ" });
 
  // Re-enqueue into the main queue
  await webhookQueue.add("process-stripe-event", job.data.originalJob);
  await job.remove();
 
  res.json({ message: "Job re-queued successfully" });
});

6. Handling Out-of-Order Delivery

Suppose a customer updates their subscription plan. It is possible for customer.subscription.updated to arrive before customer.subscription.created due to network routing.

Strategy 1: Fetch Ground-Truth State from the Provider API

Instead of relying on the event payload, use the event as a trigger to fetch the latest state directly from the provider:

async function handleSubscriptionUpdate(event) {
  const subId = event.data.object.id;
  
  // Query Stripe directly for current state
  const liveSubscription = await stripe.subscriptions.retrieve(subId);
 
  await db.query(
    `INSERT INTO subscriptions (id, plan_id, status)
     VALUES ($1, $2, $3)
     ON CONFLICT (id) DO UPDATE SET plan_id = $2, status = $3`,
    [liveSubscription.id, liveSubscription.items.data[0].price.id, liveSubscription.status]
  );
}

Strategy 2: Event Timestamp Comparison

Check the event's created timestamp against the database's last_event_timestamp before overwriting data.

💡 Key Takeaways

Always implement idempotency: Never assume an event will only arrive once.
Acknowledge in under 100ms: Enqueue work with Bull/Redis and immediately return 200 OK.
Use exponential backoff: Retry temporary database hiccups without overloading servers.
Preserve failed events in a DLQ: Never let unprocessable events disappear into the void.

👉 Next Step: Learn how to build both a Webhook Sender and a full-featured Webhook Receiver platform in Building Webhook Sender & Receiver Systems.

Webhook Security & Signatures Building Senders & Receivers