Infrastructure Monitoring: Architecting Resilient Systems for Modern Scale

re that separates collection from analysis.

Architecture Decisions and Rationale

1. OpenTelemetry as the Standard: Adopting OTel eliminates vendor lock-in and unifies metrics, logs, and traces. The OTel Collector acts as a single binary for receiving, processing, and exporting telemetry.
2. Push vs. Pull Model: Use a hybrid approach. Prometheus-style pull for stable infrastructure components (nodes, databases) to ensure scrape integrity. Push via OTel Collector for ephemeral workloads and custom application metrics to reduce load on the target services.
3. Cardinality Control: Implement processors in the Collector to drop or aggregate high-cardinality labels before they reach the backend. This protects the storage layer.
4. SLO-Driven Alerting: Alerting rules must be derived from Service Level Objectives. Alerts should trigger on error budget burn rate, not static thresholds, to account for traffic variance.

Step-by-Step Implementation

1. Deploy the OpenTelemetry Collector

The Collector should run as a DaemonSet on Kubernetes for node-level metrics and as a sidecar or gateway for application telemetry.

2. Instrument Infrastructure

Use native exporters for infrastructure components.

• Node Exporter: For host metrics.
• Database Exporters: PostgreSQL/MySQL exporters for query performance.
• Kubernetes API Server: Kube-state-metrics for resource objects.

3. Instrument Applications (TypeScript Example)

Integrate the OTel SDK into applications to emit custom business and performance metrics.

// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

// Initialize Prometheus Exporter for metrics
const promExporter = new PrometheusExporter({
  port: 9464,
  endpoint: '/metrics',
  preventServerStart: false,
});

// Initialize OTel SDK
const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'payment-service',
    [SEMRESATTRS_SERVICE_VERSION]: '1.2.0',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  metricReader: promExporter,
});

sdk.start();

// Custom Metric: Payment Processing Latency
import { MeterProvider } from '@opentelemetry/sdk-metrics';

const meterProvider = new MeterProvider();
const meter = meterProvider.getMeter('payment-meter');
const paymentLatency = meter.createHistogram('payment_processing_duration', {
  description: 'Latency of payment processing in milliseconds',
  unit: 'ms',
});

export function recordPaymentLatency(duration: number, currency: string) {
  // Currency is low cardinality; safe for metrics
  paymentLatency.record(duration, { currency });
}

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown().then(() => console.log('SDK shut down successfully'));
});

4. Configure the Collector Pipeline

The Collector configuration defines how data flows. Use processors to handle cardinality and batching.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'node-exporter'
          static_configs:
            - targets: ['localhost:9100']

processors:
  batch:
    timeout: 10s
    send_batch_max_size: 1000
  # Critical: Drop high-cardinality labels from metrics
  filter/attributes:
    metrics:
      include:
        match_type: strict
        metric_names:
          - "http.server.duration"
      exclude:
        match_type: regexp
        attributes:
          - key: "user_id"
            value: ".*"
          - key: "request_id"
            value: ".*"

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [filter/attributes, batch]
      exporters: [prometheus]

5. Implement RED/USE Methodology

Structure dashboards and alerts around proven methodologies:

• RED Method (Services): Rate, Errors, Duration. Focus on request throughput, failure rates, and latency.
• USE Method (Infrastructure): Utilization, Saturation, Errors. Focus on resource saturation and error counts for nodes and disks.

Pitfall Guide

1. High-Cardinality Metric Explosion

Mistake: Adding user_id, session_id, or request_id as metric labels. Impact: Creates millions of unique time series, causing Prometheus to run out of memory and increasing storage costs exponentially. Best Practice: Metrics must have bounded cardinality. Use traces or logs for request-specific details. Aggregate metrics by low-cardinality attributes like service, method, and status_code.

2. Static Threshold Alerting

Mistake: Setting alerts like CPU > 80% regardless of traffic patterns. Impact: Alerts fire during legitimate traffic spikes or fail during slow-bleed degradation. Best Practice: Use dynamic thresholding or error budget burn rate alerting. Alert when the error budget is being consumed faster than sustainable over a short window.

3. Monitoring Everything, Observing Nothing

Mistake: Collecting all available metrics without defining what constitutes "healthy." Impact: Dashboards become cluttered; engineers cannot distinguish critical signals from background noise. Best Practice: Define SLIs (Service Level Indicators) and SLOs first. Monitor only what impacts user experience and business goals. Prune unused dashboards quarterly.

4. Ignoring Egress and Network Boundaries

Mistake: Focusing only on internal metrics while ignoring egress traffic, DNS resolution, and third-party API latency. Impact: Incidents caused by upstream dependencies or network partitions are detected late. Best Practice: Implement synthetic monitoring and external probes. Monitor egress bandwidth and latency to critical dependencies. Include third-party health checks in the alerting workflow.

5. Lack of Runbook Integration

Mistake: Alerts fire with a link to a dashboard but no actionable steps. Impact: MTTR increases as engineers spend time diagnosing known issues or searching for documentation. Best Practice: Every alert must link to a runbook with automated remediation steps where possible. Integrate alert metadata with incident management tools (PagerDuty, Opsgenie) to provide context.

6. No Monitoring for Monitoring

Mistake: Assuming the monitoring stack is always available. Impact: If the monitoring backend crashes during an incident, the team is flying blind. Best Practice: Monitor the health of the Collector, storage backend, and alerting pipeline. Implement redundancy for critical monitoring components. Ensure the monitoring stack can operate independently of the application network if possible.

7. Testing Gaps

Mistake: Never validating that alerts actually fire or dashboards load during an incident. Impact: False confidence in monitoring coverage. Best Practice: Conduct "Fire Drill" exercises. Use chaos engineering tools to inject failures and verify that alerts trigger, dashboards update, and runbooks are effective.

Production Bundle

Action Checklist

• Define SLOs: Establish SLOs for all Tier-1 services based on user impact.
• Deploy OTel Collector: Install Collector as DaemonSet and Gateway; configure resource detection processors.
• Enforce Cardinality Limits: Apply filter processors to drop unbounded labels; audit existing metrics for high cardinality.
• Implement RED/USE Dashboards: Create dashboards focused on Rate/Error/Duration for services and USE for infrastructure.
• Configure Burn Rate Alerts: Set up multi-window burn rate alerting rules tied to error budgets.
• Integrate Runbooks: Attach actionable runbooks to every alert rule; automate remediation for common failures.
• Monitor the Stack: Add alerts for Collector health, storage capacity, and pipeline lag.
• Conduct Fire Drill: Simulate a critical failure to validate alerting, dashboards, and runbook efficacy.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Small Team / Startup	SaaS (Datadog/New Relic)	Low operational overhead; rapid setup; managed scaling.	High per-host cost; predictable OpEx.
High Scale / Cost Sensitive	Self-Hosted Prometheus + Thanos/VictoriaMetrics	Full control; no per-metric licensing fees; customizable.	High engineering effort; infrastructure costs scale with data.
Multi-Cloud / Vendor Neutral	OpenTelemetry + Backend Agnostic	Avoids lock-in; unified instrumentation across clouds.	Moderate setup cost; storage costs depend on chosen backend.
Regulatory / Data Sovereignty	On-Prem / Private Cloud Stack	Data never leaves the network; full audit control.	Highest infrastructure and maintenance cost.
Event-Driven / Serverless	OTel Push + Managed Backend	Handles ephemeral workloads; pull-based scrapers fail here.	Pay-per-use backend costs; efficient data ingestion.

Configuration Template

Copy this otel-collector-config.yaml as a baseline for a production-grade monitoring pipeline. This config includes batching, memory limits, high-cardinality filtering, and dual export for resilience.

receivers:
  otlp:
    protocols:
      grpc:
        max_recv_msg_size_mib: 32
      http:
        max_request_body_size: "32MiB"
  prometheus:
    config:
      global:
        scrape_interval: 15s
      scrape_configs:
        - job_name: 'infrastructure'
          static_configs:
            - targets: ['node-exporter:9100', 'kube-state-metrics:8080']

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 500
  batch:
    timeout: 5s
    send_batch_max_size: 2000
  filter/high_cardinality:
    metrics:
      exclude:
        match_type: regexp
        attributes:
          - key: ".*id" # Drops labels ending in id
            value: ".*"
          - key: "trace_id"
            value: ".*"
          - key: "span_id"
            value: ".*"

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      environment: "production"
  logging:
    loglevel: warn
  otlp/backup:
    endpoint: "backup-backend.internal:4317"
    tls:
      insecure: false

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, filter/high_cardinality, batch]
      exporters: [prometheus, logging]
    # Optional: Traces pipeline
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/backup]

Quick Start Guide

1. Install Collector: Run the OTel Collector container locally or deploy to your cluster.

   docker run -p 4317:4317 -p 8889:8889 \
     -v $(pwd)/otel-config.yaml:/etc/otel-collector-config.yaml \
     otel/opentelemetry-collector-contrib:latest \
     --config /etc/otel-collector-config.yaml
   

2. Instrument App: Add the OTel SDK to your TypeScript application. Set environment variables to point to the Collector.

   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
   export OTEL_SERVICE_NAME=my-service
   

3. View Metrics: Access the metrics endpoint exposed by the Collector.

   curl http://localhost:8889/metrics
   

4. Configure Grafana: Add Prometheus as a data source in Grafana pointing to http://localhost:8889. Import a standard Node.js or Kubernetes dashboard to visualize data immediately.

5. Validate Alerts: Use curl to generate load or errors on your service. Verify that metrics update in Grafana and that your alerting rules (if configured) trigger based on the defined thresholds.