:** Alert on TTT rather than absolute values. Alerting when storage is 90% full is too late; alerting when storage will be 90% full in 7 days allows for intervention.

Technical Implementation: Capacity Modeler

The following TypeScript module provides a robust foundation for calculating capacity projections. It handles linear growth, applies safety margins, and computes days remaining until threshold breach.

export interface CapacityMetric {
  name: string;
  current: number;
  limit: number;
  growthRatePerHour: number; // e.g., MB/hour or connections/hour
  safetyMargin: number; // 0.0 to 1.0, e.g., 0.10 for 10% headroom
}

export interface CapacityForecast {
  metric: string;
  daysRemaining: number;
  projectedThresholdDate: Date;
  status: 'healthy' | 'warning' | 'critical';
  recommendation: string;
}

export class CapacityModeler {
  /**
   * Calculates forecast based on linear growth model.
   * For exponential workloads, replace linear projection with exponential smoothing.
   */
  static forecast(metric: CapacityMetric): CapacityForecast {
    const effectiveLimit = metric.limit * (1 - metric.safetyMargin);
    const remainingCapacity = effectiveLimit - metric.current;
    
    let daysRemaining: number;
    
    if (metric.growthRatePerHour <= 0) {
      daysRemaining = Infinity;
    } else {
      const hoursToThreshold = remainingCapacity / metric.growthRatePerHour;
      daysRemaining = hoursToThreshold / 24;
    }

    const thresholdDate = new Date();
    thresholdDate.setDate(thresholdDate.getDate() + daysRemaining);

    let status: 'healthy' | 'warning' | 'critical';
    let recommendation = '';

    if (daysRemaining > 30) {
      status = 'healthy';
      recommendation = 'Capacity sufficient. Review in 30 days.';
    } else if (daysRemaining > 7) {
      status = 'warning';
      recommendation = `Capacity will breach in ${Math.floor(daysRemaining)} days. Schedule scaling operation.`;
    } else {
      status = 'critical';
      recommendation = `CRITICAL: Capacity breach in ${Math.floor(daysRemaining)} days. Immediate action required.`;
    }

    return {
      metric: metric.name,
      daysRemaining,
      projectedThresholdDate: thresholdDate,
      status,
      recommendation
    };
  }

  /**
   * Simulates workload burst impact on connection limits.
   * Critical for preventing connection pool exhaustion.
   */
  static simulateConnectionBurst(
    currentConnections: number,
    maxConnections: number,
    avgRequestDurationMs: number,
    expectedRPS: number
  ): { projectedConnections: number; burstRisk: boolean } {
    // Little's Law: L = λ * W
    // L = average number of items in a queuing system
    // λ = average arrival rate
    // W = average time an item spends in the system
    const projectedConnections = (expectedRPS * avgRequestDurationMs) / 1000;
    const burstRisk = projectedConnections > (maxConnections * 0.8);

    return {
      projectedConnections: Math.ceil(projectedConnections),
      burstRisk
    };
  }
}

// Usage Example
const storageMetric: CapacityMetric = {
  name: 'Primary Disk',
  current: 4500, // GB
  limit: 6000,   // GB
  growthRatePerHour: 0.5, // GB/hour
  safetyMargin: 0.15 // 15% headroom required
};

const forecast = CapacityModeler.forecast(storageMetric);
console.log(forecast);
// Output: { daysRemaining: 135, status: 'healthy', ... }

Architecture Decisions and Rationale

• Predictive vs. Reactive Scaling: Implement predictive scaling for storage and compute. Reactive scaling is acceptable only for ephemeral workloads with strict cost caps. Predictive scaling requires integration with your CI/CD pipeline to automate infrastructure changes based on forecast data.
• Sharding Strategy: When single-node capacity limits are reached, sharding is inevitable. Plan for sharding early by designing keys that distribute load evenly. Avoid range-based sharding on monotonically increasing keys (e.g., timestamps) to prevent hot spots.
• Connection Pooling: Database connection limits are often the first bottleneck. Implement aggressive connection pooling (e.g., PgBouncer, ProxySQL) at the application layer. This reduces the connection footprint on the database and allows higher concurrency with fewer resources.
• Storage Tiering: Implement automated tiering for time-series or log data. Move cold data to cheaper storage tiers or archive tables to keep the hot dataset within memory/buffer pool limits.

Pitfall Guide

Common Mistakes

1. Ignoring IOPS Throughput Limits: Cloud databases often have IOPS limits tied to storage size or instance class. You may have free disk space but hit IOPS caps, causing severe latency. Always model IOPS consumption alongside storage volume.
2. Assuming Linear Growth for Exponential Data: User-generated content or log volumes often grow exponentially. Linear models will underestimate capacity needs, leading to sudden breaches. Use exponential smoothing for metrics with accelerating growth rates.
3. Neglecting Index Bloat: In MVCC databases like PostgreSQL, indexes can bloat significantly over time, consuming disk space and degrading read performance without a corresponding increase in row count. Capacity plans must account for vacuum/maintenance overhead.
4. Overlooking Backup and Restore Capacity: Backup operations consume IOPS and storage. If your capacity plan does not reserve headroom for backup windows, backup jobs may throttle production traffic or fail due to disk full errors.
5. Single-Metric Monitoring: Focusing only on CPU or Memory ignores bottlenecks like network bandwidth, replication lag, or disk queue depth. A holistic capacity model tracks all resource vectors.
6. Static Threshold Alerts: Setting alerts at fixed percentages (e.g., 80% CPU) fails to account for rate of change. A system at 80% CPU growing at 1% per hour is stable; a system at 80% growing at 5% per hour is minutes from failure. Use rate-of-change alerts.
7. Testing with Synthetic Load: Capacity tests using uniform load patterns often miss real-world burstiness. Production traffic has spikes from marketing events, batch jobs, or cache evictions. Load tests must replicate production traffic distributions.

Best Practices

• Chaos Engineering for Capacity: Regularly inject capacity constraints in staging environments to validate alerting and scaling logic. Simulate disk full scenarios to ensure graceful degradation.
• Capacity Review Cadence: Schedule monthly capacity reviews. Analyze forecast accuracy against actual growth and adjust growth rate models.
• Document Capacity SLOs: Define Service Level Objectives for capacity, such as "Storage must not breach 85% utilization" or "P99 latency must remain under 50ms at 110% projected load."
• Automate Remediation: Where safe, automate scaling actions. For example, automatically provision additional read replicas when replication lag exceeds thresholds during predicted peak hours.

Production Bundle

Action Checklist

• Inventory Resource Vectors: List all critical metrics (CPU, Memory, Disk, IOPS, Connections, Network) for each database instance.
• Instrument Predictive Metrics: Configure monitoring to calculate predict_linear or equivalent time-series forecasts for all resource vectors.
• Define Safety Margins: Set safety margins per metric based on risk tolerance (e.g., 15% for storage, 20% for connections).
• Implement TTT Alerts: Replace absolute threshold alerts with Time-to-Threshold alerts (e.g., alert if storage breach in < 7 days).
• Run Load Validation: Execute load tests that simulate peak production traffic plus 20% growth to verify capacity limits.
• Review Connection Limits: Audit application connection pools and database max_connections settings. Ensure pooling is active.
• Document Scaling Runbooks: Create runbooks for vertical scaling, sharding, and emergency capacity expansion.
• Schedule Monthly Reviews: Establish a recurring meeting to review capacity forecasts and approve scaling actions.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Steady, Predictable Growth	Predictive Vertical Scaling	Simplest path; minimizes architectural complexity.	Moderate; costs rise with instance size.
Spiky, Unpredictable Traffic	Read Replicas + Auto-scaling	Handles bursts without over-provisioning primary.	Higher; replica costs incurred continuously.
Massive Data Volume (>10TB)	Sharding or Distributed DB	Single node limits reached; horizontal scale required.	High; operational complexity and infra costs increase.
Cost-Sensitive Workload	Serverless Database	Pay-per-use model optimizes for variable load.	Variable; can be expensive at sustained high load.
High Availability Requirement	Multi-AZ with Standby	Ensures failover capacity is always available.	High; standby resources are reserved but idle.

Configuration Template

Prometheus Alerting Rule for Predictive Capacity

This rule uses predict_linear to forecast storage exhaustion and triggers an alert if the threshold is breached within 7 days.

groups:
  - name: database_capacity
    rules:
      # Alert if storage will be full in less than 7 days
      - alert: DatabaseStorageExhaustionPredictive
        expr: |
          (
            predict_linear(
              node_filesystem_avail_bytes{mountpoint="/data"}[7d],
              7 * 24 * 3600
            )
          ) < 0
        for: 5m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Database storage exhaustion predicted within 7 days"
          description: "Instance {{ $labels.instance }} is projected to run out of storage on {{ $labels.mountpoint }} within 7 days based on 7-day trend. Current available: {{ $value }} bytes."

      # Alert if IOPS throughput is approaching limit
      - alert: DatabaseIOPSApproachingLimit
        expr: |
          rate(node_disk_io_time_seconds_total{device="nvme0n1"}[5m]) 
          > 0.85 * max_over_time(rate(node_disk_io_time_seconds_total{device="nvme0n1"}[5m])[30d:1h])
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "IOPS throughput approaching historical max"
          description: "IOPS rate on {{ $labels.instance }} is exceeding 85% of the 30-day maximum. Risk of IOPS throttling."

Quick Start Guide

1. Deploy Exporter: Install the database-specific exporter (e.g., postgres_exporter, mysqld_exporter) on your database instances. Ensure metrics are exposed on port 9187.
2. Scrape Configuration: Update your Prometheus scrape_configs to target the exporter endpoints.

   scrape_configs:
     - job_name: 'postgres'
       static_configs:
         - targets: ['db-primary:9187']
   

3. Load Recording Rules: Apply the PromQL recording rules to calculate growth rates and predictions. This reduces query load and standardizes metrics.
4. Verify Dashboard: Import a capacity planning dashboard. Confirm that days_remaining panels are populating correctly and reflect current growth trends.
5. Test Alerting: Simulate a capacity breach by temporarily adjusting the predict_linear window or threshold in a non-production environment to verify alert routing to Slack/PagerDuty.