oads, dispatches work asynchronously, and hydrates missing data via the provider's REST API.

Architecture Decisions & Rationale

1. Ingress Layer: A lightweight HTTP server receives incoming events. It must parse raw bytes for signature verification before any JSON deserialization occurs.
2. Security Boundary: HMAC-SHA256 verification prevents spoofed events. The secret is stored in a vault or environment configuration, never hardcoded.
3. Async Dispatch: Synchronous processing blocks the HTTP response, causing provider timeouts and duplicate retries. Events are immediately acknowledged and pushed to a message queue.
4. Hydration Worker: The queue consumer fetches full object details via the API. Webhook payloads often contain only identifiers or partial snapshots; the API call ensures authoritative state.
5. Idempotency Store: A Redis-backed deduplication layer tracks processed event IDs for 7–14 days, matching provider retry windows.

Implementation

// webhook-ingress.ts
import { createServer } from 'node:http';
import { createHmac, timingSafeEqual } from 'node:crypto';
import { Queue } from 'bullmq';
import { Redis } from 'ioredis';

const REDIS_URL = process.env.REDIS_URL!;
const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET!;
const eventQueue = new Queue('integration-events', { connection: new Redis(REDIS_URL) });

const server = createServer(async (req, res) => {
  if (req.method !== 'POST' || req.url !== '/v1/events') {
    res.writeHead(404);
    return res.end();
  }

  const chunks: Buffer[] = [];
  req.on('data', chunk => chunks.push(chunk));
  
  req.on('end', async () => {
    const rawBody = Buffer.concat(chunks);
    const signature = req.headers['x-provider-signature'] as string;

    if (!signature) {
      res.writeHead(401);
      return res.end('Missing signature');
    }

    const computed = createHmac('sha256', WEBHOOK_SECRET)
      .update(rawBody)
      .digest('base64');

    if (!timingSafeEqual(Buffer.from(signature), Buffer.from(computed))) {
      res.writeHead(403);
      return res.end('Invalid signature');
    }

    let payload: any;
    try {
      payload = JSON.parse(rawBody.toString('utf-8'));
    } catch {
      res.writeHead(400);
      return res.end('Malformed JSON');
    }

    await eventQueue.add('process-event', payload, {
      jobId: payload.event_id,
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 }
    });

    res.writeHead(202);
    res.end('accepted');
  });
});

server.listen(3000, () => console.log('Ingress listening on :3000'));

// hydration-worker.ts
import { Worker } from 'bullmq';
import { Redis } from 'ioredis';
import { createClient } from 'redis';

const REDIS_URL = process.env.REDIS_URL!;
const API_BASE = process.env.PROVIDER_API_URL!;
const API_KEY = process.env.PROVIDER_API_KEY!;

const dedupeClient = createClient({ url: REDIS_URL });
dedupeClient.connect();

const worker = new Worker('integration-events', async job => {
  const { event_id, object_type, object_id } = job.data;
  
  const dedupeKey = `dedupe:${event_id}`;
  const alreadyProcessed = await dedupeClient.exists(dedupeKey);
  if (alreadyProcessed) {
    console.log(`Skipping duplicate: ${event_id}`);
    return;
  }

  await dedupeClient.set(dedupeKey, '1', { EX: 1209600 }); // 14 days

  // Hydrate authoritative state via API
  const response = await fetch(`${API_BASE}/${object_type}/${object_id}`, {
    headers: { Authorization: `Bearer ${API_KEY}` }
  });

  if (!response.ok) {
    throw new Error(`API hydration failed: ${response.status}`);
  }

  const fullState = await response.json();
  await applyBusinessLogic(fullState);
}, { connection: new Redis(REDIS_URL) });

async function applyBusinessLogic(state: any) {
  // Database writes, downstream notifications, state machine transitions
  console.log('Processed authoritative state:', state.id);
}

Why This Structure Works

• Raw body consumption before parsing ensures signature verification matches the exact bytes the provider signed. JSON parsers normalize whitespace, escape sequences, and key ordering, which breaks cryptographic signatures.
• Immediate 202 Accepted response prevents provider timeout retries. The ~10-second window is strictly for acknowledgment, not business logic execution.
• Job ID mapping to event_id leverages the queue's built-in deduplication, providing a secondary safety net alongside the Redis store.
• API hydration decoupled from ingestion guarantees that partial webhook payloads never corrupt business state. The API call acts as a reconciliation step, fetching nested relationships, updated metadata, or corrected statuses that may have changed between event generation and delivery.
• 14-day deduplication window aligns with standard provider retry policies. After this period, the probability of legitimate retries drops to near zero, allowing safe cleanup.

Pitfall Guide

1. The Raw Body Trap

Explanation: Frameworks like Express or Fastify automatically parse incoming JSON. If you parse before verifying the signature, the computed hash will never match the provider's header because parsers strip whitespace, reorder keys, or normalize numbers. Fix: Always consume the raw byte stream for signature verification. Only parse JSON after the cryptographic check passes. Configure your router to bypass automatic body parsing for webhook routes.

2. Silent Idempotency Failures

Explanation: Providers retry deliveries when they don't receive a 2xx response within their timeout window. Even if your server processed the event successfully, network latency or GC pauses can cause the provider to assume failure and resend. Without deduplication, downstream systems execute mutations twice. Fix: Maintain an idempotency store keyed by event_id. Check existence before processing, and set a TTL matching the provider's maximum retry window (typically 7–14 days). Use atomic operations to prevent race conditions during concurrent retries.

3. The FIFO Illusion

Explanation: Webhook delivery is not ordered. Network partitions, provider load balancing, and retry queues cause events to arrive out of sequence. An order.shipped event may arrive before order.created, breaking state machines that assume linear progression. Fix: Never advance state based solely on event type. Always fetch the current authoritative state via the API before applying transitions. Design handlers to be idempotent and state-agnostic, relying on the API response as the single source of truth.

4. Synchronous Processing Bottlenecks

Explanation: Performing database writes, external API calls, or heavy computation inside the webhook handler blocks the HTTP response. This triggers provider timeouts, causes retry storms, and degrades system throughput under load. Fix: Acknowledge immediately with a 2xx response. Offload all business logic to an asynchronous queue. Use a worker pool to process jobs at a controlled rate, enabling backpressure and graceful degradation.

5. Cursor Drift in Polling Fallbacks

Explanation: When webhooks fail or providers lack push support, teams fall back to polling. Without a persistent cursor tracking the last processed timestamp or ID, polling either reprocesses old data or misses events that occurred between intervals. Fix: Store the cursor in durable storage. On startup, load the last known cursor. If the cursor lags beyond a threshold (e.g., 5 minutes), trigger a reconciliation job that fetches the gap. Use updated_at or monotonically increasing IDs as cursors, never relying on creation time alone.

6. Unbounded Retry Storms

Explanation: When the API hydration step fails (rate limits, temporary outages), naive retry logic floods the provider with requests, triggering circuit breakers or account suspensions. Fix: Implement exponential backoff with jitter. Set a maximum retry count (typically 3–5). Route permanently failed jobs to a dead-letter queue for manual inspection. Monitor retry rates and trigger alerts when they exceed baseline thresholds.

7. Over-Reliance on Webhook Payloads

Explanation: Webhook payloads are optimized for size and speed, not completeness. They frequently omit nested objects, calculated fields, or recent updates. Building business logic directly on webhook data leads to stale or incomplete records. Fix: Treat webhook payloads as triggers, not data sources. Always hydrate via the API. Cache API responses with short TTLs to reduce latency during high-volume event bursts.

Production Bundle

Action Checklist

• Configure ingress routes to bypass automatic JSON parsing for signature verification
• Store webhook secrets in a vault or encrypted environment configuration
• Implement HMAC-SHA256 verification using timing-safe comparison functions
• Acknowledge all incoming events with a 2xx response within 2 seconds
• Dispatch events to an asynchronous queue with event_id as the job identifier
• Maintain a Redis-backed deduplication store with a 14-day TTL
• Hydrate partial payloads via the provider's REST API before applying business logic
• Route failed hydration jobs to a dead-letter queue with structured error metadata

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-time payment settlement tracking	Webhook trigger + API hydration	Payments require sub-second reaction but authoritative state for reconciliation	Low infrastructure cost, high reliability
End-of-day financial reconciliation	API polling with cursor	Batch operations don't require real-time latency; polling ensures complete dataset	Higher API call volume, predictable compute
Bulk product catalog updates	API batch mutation	Webhooks cannot push outbound changes; API supports atomic batch operations	Rate limit consumption, but optimal for write-heavy workloads
CI/CD pipeline triggers	Webhook only	Build systems only need event notification; state is fetched internally by the runner	Minimal cost, high throughput
User profile synchronization	Hybrid (webhook for changes, API for full sync)	Webhooks catch mutations; API resolves conflicts and fetches missing fields	Moderate cost, prevents data drift

Configuration Template

// config/integration.ts
export const INTEGRATION_CONFIG = {
  ingress: {
    port: parseInt(process.env.WEBHOOK_PORT || '3000', 10),
    path: '/v1/events',
    timeoutMs: 2000, // Hard limit for 2xx response
  },
  security: {
    algorithm: 'sha256',
    headerName: 'x-provider-signature',
    secretEnvVar: 'WEBHOOK_SECRET',
  },
  queue: {
    name: 'integration-events',
    maxAttempts: 3,
    backoffDelay: 2000,
    concurrency: 10,
  },
  deduplication: {
    ttlSeconds: 1209600, // 14 days
    keyPrefix: 'dedupe:',
  },
  api: {
    baseUrl: process.env.PROVIDER_API_URL!,
    authHeader: 'Authorization',
    authPrefix: 'Bearer',
    rateLimitBurst: 50,
    rateLimitInterval: 60000,
  },
  observability: {
    metricsPrefix: 'integration.webhook.',
    logLevel: process.env.LOG_LEVEL || 'info',
    dlqAlertThreshold: 10, // Alert after 10 dead-lettered jobs
  },
};

Quick Start Guide

1. Initialize the ingress service: Deploy the webhook receiver with environment variables for WEBHOOK_SECRET, REDIS_URL, and PROVIDER_API_URL. Ensure the endpoint is publicly accessible over HTTPS.
2. Register the webhook: Use the provider's dashboard or API to register your ingress URL. Select the event types required for your integration. Verify the provider sends a test payload.
3. Deploy the worker: Run the queue consumer on a separate compute instance or container. Configure concurrency based on your API rate limits and database write capacity.
4. Validate the pipeline: Trigger a test event. Confirm the ingress returns 202 within 2 seconds, the worker processes the job, and the deduplication key is created in Redis.
5. Enable monitoring: Attach metrics to queue depth, processing latency, and deduplication hits. Set alerts for dead-letter queue growth and API hydration failures.