API request deduplication

g unbounded growth.

Core Solution

Implementing robust API deduplication requires a middleware layer that intercepts requests, computes a deterministic key, checks a fast storage backend, and manages the lifecycle of the deduplication record.

Step-by-Step Implementation

1. Key Generation: Construct a key that uniquely identifies the request intent. This should combine the HTTP method, path, a client-provided idempotency key (if available), and a hash of the request body.
2. Storage Selection: Use a distributed in-memory store (e.g., Redis) for O(1) lookups and atomic operations. Avoid primary databases for the hot path.
3. Response Caching: Store the response body and status code alongside the key. This allows the middleware to return the cached result on duplicate hits, satisfying the client without re-executing logic.
4. TTL Management: Set a Time-To-Live on deduplication records based on the maximum expected retry window and business requirements.
5. Error Handling: Ensure that if processing fails, the deduplication record is cleared or marked to allow retries, depending on the error type.

Code Implementation (TypeScript / Express)

This implementation uses ioredis for storage and provides a reusable middleware factory.

import { Request, Response, NextFunction } from 'express';
import Redis from 'ioredis';
import { createHash } from 'crypto';

export interface DeduplicationConfig {
  redis: Redis;
  ttlSeconds: number;
  keyPrefix: string;
  maxResponseSizeBytes: number; // Prevent caching massive payloads
}

export const createDedupMiddleware = (config: DeduplicationConfig) => {
  const { redis, ttlSeconds, keyPrefix, maxResponseSizeBytes } = config;

  return async (req: Request, res: Response, next: NextFunction) => {
    // Only deduplicate safe-write operations (POST, PUT, PATCH)
    if (!['POST', 'PUT', 'PATCH'].includes(req.method)) {
      return next();
    }

    // 1. Generate Deduplication Key
    const clientKey = req.headers['x-idempotency-key'] as string;
    const bodyHash = createHash('sha256')
      .update(JSON.stringify(req.body))
      .digest('hex')
      .slice(0, 16);

    const dedupKey = `${keyPrefix}:${req.method}:${req.path}:${clientKey || bodyHash}`;

    // 2. Check Cache
    const cachedResult = await redis.get(dedupKey);
    if (cachedResult) {
      try {
        const parsed = JSON.parse(cachedResult);
        return res.status(parsed.status).json(parsed.body);
      } catch {
        // Corrupted cache entry; proceed to processing
      }
    }

    // 3. Wrap Response to Capture Result
    const originalJson = res.json.bind(res);
    const originalSend = res.send.bind(res);

    const captureResponse = (statusCode: number, body: any) => {
      const responsePayload = { status: statusCode, body };
      const serialized = JSON.stringify(responsePayload);

      if (serialized.length <= maxResponseSizeBytes) {
        // Store result with TTL
        // Use SET with NX to avoid overwriting if race condition occurs
        redis.set(dedupKey, serialized, 'EX', ttlSeconds, 'NX');
      }
    };

    res.json = (body: any) => {
      captureResponse(res.statusCode, body);
      return originalJson(body);
    };

    res.send = (body: any) => {
      captureResponse(res.statusCode, body);
      return originalSend(body);
    };

    // 4. Handle Errors: Clear cache on failure to allow retry
    const originalEnd = res.end.bind(res);
    res.end = function (...args: any[]) {
      if (res.statusCode >= 500) {
        redis.del(dedupKey).catch(() => {});
      }
      return originalEnd(...args);
    };

    next();
  };
};

Architecture Decisions and Rationale

• Response Caching vs. Rejection: Rejecting duplicates with a 409 Conflict forces the client to handle errors. Caching the response allows the client to receive the success result transparently, which is critical for mobile apps where the initial success response may have been lost in transit.
• Hash Slicing: The body hash is sliced to reduce key size and storage overhead. SHA-256 provides sufficient collision resistance for API payloads; slicing to 16 characters balances uniqueness with storage efficiency.
• TTL Strategy: The TTL must exceed the client's maximum retry duration. If the client retries after the TTL expires, the request is processed again, which is acceptable behavior for expired idempotency windows.
• Size Limits: Caching responses without limits can exhaust Redis memory. Enforcing maxResponseSizeBytes ensures that only typical API responses are cached; large payloads should be fetched via subsequent GET requests.

Pitfall Guide

1. Storing Responses in Primary Database: Using a relational table for deduplication storage introduces write amplification and index bloat. This degrades performance for high-write workloads and complicates schema migrations.
   • Best Practice: Use a dedicated cache layer with TTLs. Archive metadata to a data warehouse if audit trails are required.
2. Ignoring Partial Failures: If a request processes partially (e.g., payment captured but notification fails) and the client retries, naive deduplication may block the retry.
   • Best Practice: Implement idempotency keys that support "in-flight" states or use compensating transactions. Ensure the deduplication key covers the entire transaction scope.
3. Weak Hashing Algorithms: Using MD5 or CRC32 for body hashing increases collision probability, leading to false deduplication of distinct requests.
   • Best Practice: Always use SHA-256 or SHA-3 for content hashing.
4. TTL Mismatch: Setting a TTL shorter than the client's retry policy causes duplicates to be processed after the cache expires.
   • Best Practice: Align TTL with the maximum retry window. Document the idempotency window in API specs.
5. Caching PII in Cache: Storing request/response bodies in Redis may violate data retention policies if PII is cached without encryption or scrubbing.
   • Best Practice: Implement a response scrubber that removes sensitive fields before caching. Use Redis ACLs and encryption at rest.
6. Cross-Service Deduplication: In microservices, a duplicate request may hit different services. Service-level deduplication does not prevent duplicate downstream calls.
   • Best Practice: Propagate the idempotency key via headers (e.g., X-Request-ID) and implement deduplication at the orchestration layer or use distributed tracing context.
7. Mutable Payloads: If the request body contains timestamps or nonces that change per retry, hashing the body will generate different keys, defeating deduplication.
   • Best Practice: Require clients to use stable idempotency keys for mutable payloads. Do not rely solely on body hashing for such requests.

Production Bundle

Action Checklist

• Generate Client Keys: Ensure all clients generate and transmit X-Idempotency-Key headers for mutating operations.
• Deploy Deduplication Store: Provision a Redis cluster with sufficient memory and configure persistence policies appropriate for ephemeral data.
• Implement Middleware: Deploy the deduplication middleware to the API gateway or service layer, configuring TTL based on retry policies.
• Configure Response Limits: Set maxResponseSizeBytes to prevent cache exhaustion from large payloads.
• Add Observability: Instrument metrics for dedup.hit_count, dedup.miss_count, and dedup.cache_size to monitor effectiveness.
• Audit PII: Review cached responses for sensitive data and implement scrubbing if necessary.
• Test Retry Scenarios: Validate deduplication behavior under network partition simulation using chaos engineering tools.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Financial Transactions	Distributed Cache + Response Cache	Guarantees no double charges; returns cached result on retry; high integrity.	Medium (Redis nodes)
High-Throughput Logging	Client Debounce + Server Rate Limit	Deduplication overhead is unjustified; focus on volume reduction at edge.	Low
Public API with Mobile Clients	Server-Side Idempotency Key	Mobile networks cause frequent retries; response caching improves UX significantly.	Low-Medium
Legacy Monolith	DB Unique Constraint + Retry Logic	Minimal architectural change; constraints prevent data corruption; acceptable latency.	Low (DB load)
Event Processing Pipeline	Distributed Deduplication + State Store	Ensures exactly-once semantics; handles out-of-order delivery and consumer rebalancing.	High (State storage)

Configuration Template

Use this configuration interface to standardize deduplication settings across services.

// dedup.config.ts
export const dedupConfig = {
  redis: {
    host: process.env.REDIS_HOST || 'localhost',
    port: parseInt(process.env.REDIS_PORT || '6379'),
    password: process.env.REDIS_PASSWORD,
    keyPrefix: 'dedup:v1',
  },
  ttlSeconds: parseInt(process.env.IDEMPOTENCY_TTL || '86400'), // 24 hours default
  maxResponseSizeBytes: 1024 * 1024, // 1MB limit
  hashAlgorithm: 'sha256',
  enabled: process.env.DEDUP_ENABLED === 'true',
  // Paths to exclude (e.g., health checks, webhooks)
  excludePaths: ['/health', '/webhooks/*'],
};

Quick Start Guide

1. Install Dependencies:

   npm install express ioredis
   

2. Add Middleware: Import createDedupMiddleware and apply it to your Express app before route definitions.

   import { createDedupMiddleware } from './dedup-middleware';
   import { dedupConfig } from './dedup.config';
   
   app.use(createDedupMiddleware({
     redis: new Redis(dedupConfig.redis),
     ttlSeconds: dedupConfig.ttlSeconds,
     keyPrefix: dedupConfig.redis.keyPrefix,
     maxResponseSizeBytes: dedupConfig.maxResponseSizeBytes,
   }));
   

3. Configure Client Headers: Update client SDKs to generate UUIDs and attach X-Idempotency-Key headers to all POST/PUT/PATCH requests.
4. Verify Behavior: Send a request, capture the response, and resend the identical request within the TTL. Confirm the second request returns the cached response with negligible latency and no backend processing logs.
5. Monitor: Check Redis keyspace hits/misses and application logs to ensure deduplication is active and effective.