Backend secrets management

rovider, enforces caching, and supports graceful degradation. The following architecture uses a Provider pattern with a time-to-live (TTL) cache.

Architecture Decisions

1. Abstraction Layer: The application interacts with a SecretProvider interface, not a specific SDK. This allows swapping AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault without code changes.
2. Caching Strategy: Secrets are cached in memory with a configurable TTL. This reduces latency and prevents rate-limiting against the secret manager. The cache must support invalidation to handle rotations.
3. Injection Pattern: Secrets are injected via dependency injection at startup or on-demand. On-demand injection with caching is preferred for high-churn environments.
4. Error Handling: The provider must distinguish between transient network errors and permission failures. Retries should be exponential with jitter.

TypeScript Implementation

This implementation provides a robust, production-ready secret provider using AWS Secrets Manager as the reference backend.

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";
import type { GetSecretValueCommandOutput } from "@aws-sdk/client-secrets-manager";

/**
 * Abstraction for secret retrieval.
 * Enables swapping providers without touching business logic.
 */
export interface SecretProvider {
  getSecret(secretName: string): Promise<string>;
}

/**
 * CachedSecretProvider implements SecretProvider with a local TTL cache.
 * 
 * Rationale:
 * - Reduces latency by avoiding network calls for cached secrets.
 * - Prevents throttling on the secret manager API.
 * - Supports dynamic rotation by invalidating cache after TTL expires.
 */
export class CachedSecretProvider implements SecretProvider {
  private client: SecretsManagerClient;
  private cache: Map<string, { value: string; expiresAt: number }>;
  private defaultTTL: number; // in milliseconds

  constructor(options?: { region?: string; defaultTTLSeconds?: number }) {
    this.client = new SecretsManagerClient({ region: options?.region || "us-east-1" });
    this.cache = new Map();
    this.defaultTTL = (options?.defaultTTLSeconds || 300) * 1000; // Default 5 minutes
  }

  async getSecret(secretName: string): Promise<string> {
    const cached = this.cache.get(secretName);
    const now = Date.now();

    if (cached && cached.expiresAt > now) {
      return cached.value;
    }

    try {
      const command = new GetSecretValueCommand({ SecretId: secretName });
      const response: GetSecretValueCommandOutput = await this.client.send(command);

      if (!response.SecretString) {
        throw new Error(`Secret ${secretName} returned no value.`);
      }

      const value = response.SecretString;
      
      // Update cache with TTL
      this.cache.set(secretName, {
        value,
        expiresAt: now + this.defaultTTL,
      });

      return value;
    } catch (error) {
      // If cache exists and we hit a transient error, serve stale data
      // This improves resilience during secret manager outages
      if (cached) {
        console.warn(`Failed to refresh secret ${secretName}, serving stale cache. Error: ${error}`);
        return cached.value;
      }
      throw error;
    }
  }

  /**
   * Explicitly invalidate a secret. Useful for testing or forced rotation triggers.
   */
  invalidate(secretName: string): void {
    this.cache.delete(secretName);
  }
}

/**
 * Usage Example: Dependency Injection in a Database Connection
 */
class DatabaseService {
  constructor(
    private secretProvider: SecretProvider,
    private dbConfig: { host: string; port: number; db: string }
  ) {}

  async getConnectionUri(): Promise<string> {
    // Secrets are fetched on-demand with caching
    const username = await this.secretProvider.getSecret("db/admin/username");
    const password = await this.secretProvider.getSecret("db/admin/password");
    
    // Sanitize logs: Never log the URI containing credentials
    return `postgresql://${username}:****@${this.dbConfig.host}:${this.dbConfig.port}/${this.dbConfig.db}`;
  }
}

Implementation Steps

1. Define Interface: Create the SecretProvider interface to enforce contract compliance.
2. Implement Cache: Build the CachedSecretProvider with TTL logic and stale-caching fallback.
3. Configure Client: Initialize the cloud SDK client with minimal permissions.
4. Inject Provider: Pass the provider to services requiring secrets via constructor injection.
5. Audit Logs: Ensure no secret values are written to standard output or error logs. Use masking utilities.

Pitfall Guide

Production experience reveals recurring patterns of failure in secrets management. Avoid these critical mistakes:

1. Logging Secrets:

   • Mistake: Accidentally logging secret values in error traces, access logs, or metrics.
   • Impact: Logs are often aggregated to third-party tools (Datadog, Splunk) with broader access controls, creating a secondary breach vector.
   • Fix: Implement log scrubbing middleware. Never concatenate secrets into log strings. Use structured logging with redaction rules.

2. Infinite Caching:

   • Mistake: Caching secrets indefinitely or without a TTL.
   • Impact: Rotation becomes impossible. If a secret is compromised, the application continues to use the old credential until restart, extending the window of exposure.
   • Fix: Enforce strict TTLs. Use short TTLs (5-15 minutes) for high-value secrets.

3. Over-Privileged IAM Roles:

   • Mistake: Granting the application role secretsmanager:GetSecretValue on *.
   • Impact: If the application is compromised, the attacker can retrieve all secrets in the account.
   • Fix: Apply least privilege. Scope IAM policies to specific secret ARNs required by the service.

4. Ignoring Rotation Impact:

   • Mistake: Rotating secrets without updating dependent services or connection pools.
   • Impact: Application crashes or database connection failures upon rotation.
   • Fix: Design applications to handle credential changes gracefully. Use connection poolers that support re-authentication. Test rotation in staging.

5. Storing Encrypted Secrets in Databases:

   • Mistake: Encrypting secrets with a hardcoded key and storing them in the application database.
   • Impact: The key management problem is merely shifted, not solved. If the code is compromised, the key is exposed, decrypting all secrets.
   • Fix: Use a dedicated KMS (Key Management Service) for encryption keys. Never hardcode keys.

6. Using .env Files in Production:

   • Mistake: Mounting .env files into containers as the primary secret source.
   • Impact: .env files are static, lack audit trails, and are difficult to rotate. They often end up in version control or image layers.
   • Fix: Use secret managers or workload identity. Reserve .env for local development only.

7. Secret Sprawl:

   • Mistake: Creating unique secrets for every microservice without a naming convention or tagging strategy.
   • Impact: Operational chaos. Secrets become unmanageable, and orphaned secrets accumulate, increasing the attack surface.
   • Fix: Enforce naming conventions (e.g., env/service/secret-type). Implement lifecycle policies to delete unused secrets.

Production Bundle

Action Checklist

• Audit Existing Secrets: Scan codebases, config maps, and infrastructure scripts for hardcoded credentials. Use tools like trufflehog or gitleaks.
• Implement Secret Abstraction: Replace direct env var access with a SecretProvider interface across all services.
• Enable Caching: Deploy the cached secret provider with a TTL appropriate for your rotation policy.
• Scope IAM Policies: Review and restrict IAM roles to access only the specific secrets required by each service.
• Configure Audit Logging: Enable logging on the secret manager to track access patterns and detect anomalies.
• Test Rotation: Perform a live rotation of a production secret in a staging environment to verify application resilience.
• Remove Static Credentials: Eliminate all .env files, config maps, and hardcoded values from production deployments.
• Set Up Alerts: Configure alerts for failed secret retrieval attempts or unauthorized access patterns.

Decision Matrix

Use this matrix to select the appropriate secrets management strategy based on your infrastructure and requirements.

Scenario	Recommended Approach	Why	Cost Impact
Monolith on EC2/VM	AWS Secrets Manager / Vault	Centralized management, rotation support, audit trails.	Low (Per-secret cost)
Kubernetes Microservices	Workload Identity + External Secrets Operator	Ephemeral credentials, no static secrets, native K8s integration.	Medium (Operator complexity)
Serverless Functions	Workload Identity / IAM Roles	Stateless execution, native integration with provider IAM.	None (Included in compute cost)
Multi-Cloud / Hybrid	HashiCorp Vault (Enterprise)	Unified API, secret engines, policy enforcement across clouds.	High (Licensing/Infra)
Local Development	.env + Mock Provider	Developer velocity, offline capability, simplicity.	None

Configuration Template

Copy this TypeScript configuration to bootstrap your secret provider with production settings.

// config/secrets.config.ts
import { CachedSecretProvider } from '../providers/CachedSecretProvider';

export const secretsConfig = {
  region: process.env.AWS_REGION || 'us-east-1',
  // TTL in seconds. Rotate frequently for high-security contexts.
  defaultTTLSeconds: 300, 
  // Retry configuration for resilience
  retryConfig: {
    maxRetries: 3,
    baseDelayMs: 100,
    maxDelayMs: 2000,
  },
  // Stale cache fallback: Serve stale data if provider is unreachable
  allowStaleOnFailure: true,
};

// Factory function for dependency injection
export function createSecretProvider(): CachedSecretProvider {
  return new CachedSecretProvider({
    region: secretsConfig.region,
    defaultTTLSeconds: secretsConfig.defaultTTLSeconds,
  });
}

Quick Start Guide

Get secrets management running in under 5 minutes.

1. Install SDK:

   npm install @aws-sdk/client-secrets-manager
   

2. Create Provider: Add the CachedSecretProvider class to your providers/ directory. Import it into your application entry point.
3. Inject Dependency:

   const secretProvider = createSecretProvider();
   const dbService = new DatabaseService(secretProvider, dbConfig);
   

4. Retrieve Secret:

   const dbPassword = await secretProvider.getSecret('prod/db/password');
   

5. Verify: Run the application and check logs to ensure secrets are fetched successfully and cached. Validate that no secret values appear in log output.