terraform/dr-infrastructure.tf

• Tier 2: Background jobs, analytics, caches (RTO <4h, RPO <1h)
• Tier 3: Static assets, logs, archives (RTO <24h, RPO <24h)

Tier assignment drives infrastructure replication strategy and automation depth.

Step 2: Implement Immutable Infrastructure with State Reconciliation

Deploy all infrastructure through declarative tooling (Terraform, Pulumi, CDK). State files must be versioned, encrypted, and replicated across regions. Use remote state backends with cross-region replication enabled. Avoid mutable changes outside IaC; enforce drift detection in CI/CD pipelines.

Step 3: Automate Cross-Region Data Replication

Select replication mechanisms aligned with data consistency requirements:

• Relational databases: Managed read replicas with synchronous commit for Tier 0, asynchronous for Tier 1
• Object storage: Cross-region replication with versioning and lifecycle policies
• Message queues: Mirrored topics with consumer offset tracking
• Caches: Ephemeral; rebuild from source on failover

Validate replication lag continuously. Reject cutover if lag exceeds RPO thresholds.

Step 4: Build Automated DR Validation Pipeline

Replace manual testing with scheduled, automated recovery simulations. The pipeline must verify data consistency, deploy infrastructure in the secondary region, route traffic, validate health endpoints, and roll back safely.

// dr-validator.ts
import { S3Client, ListBucketsCommand } from "@aws-sdk/client-s3";
import { RDSClient, DescribeDBInstancesCommand } from "@aws-sdk/client-rds";
import { Route53Client, ListHostedZonesCommand } from "@aws-sdk/client-route-53";
import axios from "axios";

interface DRValidationResult {
  region: string;
  status: "PASS" | "FAIL";
  rpoLagSeconds: number;
  rtoElapsedMs: number;
  healthCheckStatus: number;
}

export async function runDRValidation(
  primaryRegion: string,
  secondaryRegion: string,
  targetEndpoint: string
): Promise<DRValidationResult> {
  const startTime = Date.now();
  const s3Client = new S3Client({ region: secondaryRegion });
  const rdsClient = new RDSClient({ region: secondaryRegion });
  const route53Client = new Route53Client({ region: secondaryRegion });

  // Verify cross-region replication exists
  const buckets = await s3Client.send(new ListBucketsCommand({}));
  const replicatedBuckets = buckets.Buckets?.filter(b => 
    b.Name?.startsWith("app-dr-")
  ) ?? [];

  if (replicatedBuckets.length === 0) {
    throw new Error("No replicated storage targets found in secondary region");
  }

  // Check database replication lag
  const dbInstances = await rdsClient.send(new DescribeDBInstancesCommand({}));
  const replica = dbInstances.DBInstances?.find(d => d.DBInstanceStatus === "available");
  const replicationLag = replica?.ReplicationLag ?? 0;

  // Validate health endpoint after simulated cutover DNS update
  const healthCheck = await axios.get(targetEndpoint, { timeout: 5000 });
  const rtoElapsed = Date.now() - startTime;

  return {
    region: secondaryRegion,
    status: replicationLag <= 60 && healthCheck.status === 200 ? "PASS" : "FAIL",
    rpoLagSeconds: replicationLag,
    rtoElapsedMs: rtoElapsed,
    healthCheckStatus: healthCheck.status,
  };
}

Step 5: Integrate Chaos Engineering for Failure Injection

Schedule controlled failure scenarios: region network partition, database replica promotion, DNS TTL expiration, and IAM credential rotation. Validate that automated recovery pipelines trigger without manual intervention. Record mean time to recovery (MTTR) and compare against RTO targets.

Architecture Decisions and Rationale

• Declarative over imperative: Infrastructure state must be reproducible from code. Imperative scripts introduce drift and break recovery determinism.
• Idempotent deployments: DR execution must handle repeated runs without data corruption or resource conflicts.
• Health-based cutover: Routing decisions must depend on endpoint validation, not time-based assumptions.
• Replication lag gates: Cutover should abort if data consistency falls outside RPO boundaries.
• Automated rollback: Every failover must include a verified rollback path to prevent state divergence.

Pitfall Guide

1. Assuming backups equal disaster recovery Backups protect against data loss; DR protects against service unavailability. A backup restores files; DR restores traffic routing, state consistency, and dependency resolution. Validate recovery paths, not just backup success.

2. Static runbooks with manual execution Runbooks that require human decision-making during incidents introduce latency and error. Automate cutover logic, DNS updates, and health validation. Reserve manual intervention for architectural escalation, not routine failover.

3. Ignoring cross-region dependency mapping Applications depend on DNS, IAM roles, VPC peering, security groups, and external APIs. Replicating compute and storage without replicating network topology and identity permissions guarantees failure. Map and automate all dependency chains.

4. No automated replication lag monitoring Asynchronous replication accumulates lag under load. Without continuous monitoring, RPO targets become theoretical. Implement lag thresholds that block cutover when exceeded.

5. Over-provisioning passive regions without utilization tracking Idle infrastructure incurs cost without validation. Run continuous health checks, scheduled DR tests, and cost attribution. Treat secondary regions as active validation environments, not storage lockers.

6. Missing DNS TTL and caching awareness DNS propagation delays extend RTO beyond infrastructure recovery time. Reduce TTL to 60–300 seconds before failover events. Use health-checked routing policies (Route 53, Cloudflare Load Balancing) instead of manual record updates.

7. Testing only during business hours with low traffic DR tests under nominal load do not reflect production failure conditions. Inject traffic, simulate concurrent writes, and validate replication under stress. Recovery behavior changes significantly under load.

Best practices from production experience:

• Run DR validation on every major infrastructure change, not just quarterly.
• Version control all DR scripts, IaC, and configuration templates.
• Implement circuit breakers that prevent cascading failover during partial outages.
• Log every recovery step with timestamps for post-incident analysis.
• Align RTO/RPO targets with actual business impact, not engineering convenience.

Production Bundle

Action Checklist

• Define RTO/RPO per service tier and document dependency chains
• Migrate all infrastructure to declarative IaC with remote state replication
• Configure cross-region data replication with lag monitoring and alerting
• Build automated DR validation pipeline with health checks and rollback logic
• Reduce DNS TTL to ≤300s and implement health-based routing policies
• Schedule monthly chaos experiments simulating region failure and replica promotion
• Implement cost attribution and utilization tracking for secondary regions
• Run DR validation on every infrastructure change and major release

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Legacy monolith with infrequent updates	Cold Backup + IaC snapshot	Low change velocity justifies slower recovery; minimizes idle infrastructure cost	Low ($800–$1,500/mo)
Customer-facing API with 15-min RTO requirement	Warm Standby with automated cutover	Balances recovery speed and cost; requires replication lag validation	Medium ($3,200–$5,500/mo)
Financial transaction platform with zero data loss tolerance	Hot Standby with synchronous replication	RPO <5m demands real-time consistency; automation prevents human error during failover	High ($8,500–$12,000/mo)
Global SaaS with multi-region traffic distribution	Active-Active with conflict resolution	Eliminates single-region dependency; requires distributed state management and consistent hashing	Very High ($15,000–$22,000/mo)

Configuration Template

# terraform/dr-infrastructure.tf
resource "aws_db_instance" "primary" {
  engine               = "postgres"
  instance_class       = "db.r6g.xlarge"
  multi_az             = true
  backup_retention_period = 30
  storage_encrypted    = true
}

resource "aws_db_instance" "replica" {
  engine               = "postgres"
  instance_class       = "db.r6g.xlarge"
  replicate_source_db  = aws_db_instance.primary.id
  skip_final_snapshot  = true
  publicly_accessible  = false
}

resource "aws_s3_bucket" "primary_storage" {
  bucket = "app-primary-data"
}

resource "aws_s3_bucket_replication_configuration" "replication" {
  bucket = aws_s3_bucket.primary_storage.id
  role   = aws_iam_role.replication.arn

  rule {
    status = "Enabled"
    destination {
      bucket        = aws_s3_bucket.secondary_storage.arn
      storage_class = "STANDARD"
    }
  }
}

resource "aws_route53_health_check" "app_health" {
  fqdn              = "api.example.com"
  port              = 443
  type              = "HTTPS"
  request_interval  = 30
  failure_threshold = 3
}

resource "aws_route53_record" "api_routing" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  set_identifier = "primary"
  health_check_id = aws_route53_health_check.app_health.id
  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }
}

Quick Start Guide

1. Install dependencies: npm install @aws-sdk/client-s3 @aws-sdk/client-rds @aws-sdk/client-route-53 axios
2. Define tier mapping: Create a recovery-tiers.json file mapping services to RTO/RPO targets and dependency lists.
3. Deploy baseline infrastructure: Apply the Terraform template to establish primary/replica resources and health-checked routing.
4. Run validation: Execute ts-node dr-validator.ts --primary us-east-1 --secondary eu-west-1 --endpoint https://api.example.com/health
5. Schedule automation: Add the validation script to your CI/CD pipeline with cron triggers (weekly dry run, monthly full failover test) and alert on status: "FAIL".