LLM cost optimization strategies

to high-capability models. Implement fallback chains to prevent failures.

Step 4: Context Window Pruning & Prompt Optimization

Strip redundant history, compress system prompts, and truncate non-essential context. Use structured output schemas to bound generation length.

TypeScript Implementation

import { createClient } from '@anthropic-ai/sdk';
import { createClient as createRedisClient } from 'redis';

interface LLMRequest {
  prompt: string;
  system?: string;
  maxTokens?: number;
  temperature?: number;
}

interface LLMResponse {
  content: string;
  tokensUsed: { input: number; output: number };
  model: string;
  cost: number;
}

class LLMCostOptimizer {
  private anthropic: ReturnType<typeof createClient>;
  private cache: ReturnType<typeof createRedisClient>;
  private modelTiers = {
    lightweight: 'claude-3-haiku-20240307',
    standard: 'claude-3-sonnet-20240229',
    high: 'claude-3-opus-20240229'
  };
  private pricingPer1kTokens = {
    lightweight: { input: 0.0008, output: 0.004 },
    standard: { input: 0.003, output: 0.015 },
    high: { input: 0.015, output: 0.075 }
  };

  constructor(apiKey: string, redisUrl: string) {
    this.anthropic = createClient({ apiKey });
    this.cache = createRedisClient({ url: redisUrl });
    this.cache.connect();
  }

  async routeAndOptimize(req: LLMRequest): Promise<LLMResponse> {
    // 1. Check semantic cache
    const cacheKey = await this.generateSemanticKey(req.prompt);
    const cached = await this.cache.get(cacheKey);
    if (cached) {
      return JSON.parse(cached);
    }

    // 2. Classify complexity
    const tier = this.classifyComplexity(req.prompt);
    const model = this.modelTiers[tier];

    // 3. Prune context & enforce token limits
    const optimizedPrompt = this.pruneContext(req);
    const maxTokens = req.maxTokens || this.getDefaultMaxTokens(tier);

    // 4. Execute with fallback chain
    const response = await this.executeWithFallback(optimizedPrompt, model, maxTokens, req.temperature);

    // 5. Calculate cost & cache result
    const cost = this.calculateCost(response.tokensUsed, model);
    const result: LLMResponse = { ...response, model, cost };
    
    await this.cache.set(cacheKey, JSON.stringify(result), { EX: 3600 }); // 1h TTL
    return result;
  }

  private classifyComplexity(prompt: string): 'lightweight' | 'standard' | 'high' {
    const complexityIndicators = ['analyze', 'compare', 'reason', 'debug', 'architect'];
    const score = complexityIndicators.filter(w => prompt.toLowerCase().includes(w)).length;
    if (score >= 3) return 'high';
    if (score >= 1) return 'standard';
    return 'lightweight';
  }

  private pruneContext(req: LLMRequest): string {
    // Remove redundant prefixes, enforce max input tokens, keep system prompt concise
    const MAX_INPUT_CHARS = 8000;
    return req.prompt.slice(-MAX_INPUT_CHARS);
  }

  private async executeWithFallback(
    prompt: string,
    model: string,
    maxTokens: number,
    temperature: number = 0.7
  ): Promise<{ content: string; tokensUsed: { input: number; output: number } }> {
    try {
      const msg = await this.anthropic.messages.create({
        model,
        max_tokens: maxTokens,
        temperature,
        system: 'You are a precise assistant.',
        messages: [{ role: 'user', content: prompt }]
      });
      return {
        content: msg.content[0].type === 'text' ? msg.content[0].text : '',
        tokensUsed: { input: msg.usage.input_tokens, output: msg.usage.output_tokens }
      };
    } catch (err) {
      // Fallback to standard tier on rate limit or failure
      if (model === this.modelTiers.high) {
        return this.executeWithFallback(prompt, this.modelTiers.standard, maxTokens, temperature);
      }
      throw err;
    }
  }

  private calculateCost(tokens: { input: number; output: number }, model: string): number {
    const tier = Object.entries(this.modelTiers).find(([, m]) => m === model)?.[0] as keyof typeof this.pricingPer1kTokens;
    const rates = this.pricingPer1kTokens[tier];
    return (tokens.input / 1000) * rates.input + (tokens.output / 1000) * rates.output;
  }

  private async generateSemanticKey(prompt: string): Promise<string> {
    // In production, use an embedding model to generate a normalized vector hash
    // For demonstration, we use a simplified deterministic hash
    const crypto = await import('crypto');
    return crypto.createHash('sha256').update(prompt.toLowerCase().trim()).digest('hex').slice(0, 16);
  }

  private getDefaultMaxTokens(tier: string): number {
    const limits: Record<string, number> = { lightweight: 512, standard: 1024, high: 2048 };
    return limits[tier] || 1024;
  }
}

export default LLMCostOptimizer;

Architecture Decisions & Rationale

• Semantic over exact caching: LLM queries vary in phrasing but converge in intent. Vector-based or normalized hashing prevents cache misses on paraphrased inputs.
• Tiered routing by complexity indicators: Keyword and pattern matching provides low-latency classification without invoking a separate model. Production systems should replace this with a lightweight classifier (e.g., fasttext or small embedding model).
• Context pruning before transmission: Input token cost dominates spend. Truncating from the end preserves recent context while discarding stale history. System prompts should be static and reused, not regenerated.
• Fallback chains: Rate limits and model outages are inevitable. A deterministic fallback sequence prevents request failure while capping cost exposure.
• Cost calculation at the wrapper level: Vendor SDKs rarely expose real-time cost. Calculating at the application layer enables budget alerts, per-user billing, and capacity planning.

Pitfall Guide

1. Caching without TTL or versioning: Cached responses become stale when business logic, data sources, or model versions change. Implement explicit TTLs, cache invalidation hooks, and prompt version tags.
2. Optimizing input tokens while ignoring output costs: Teams aggressively compress prompts but leave max_tokens unbounded. Output tokens often carry higher per-unit pricing. Enforce generation limits and use structured output to constrain length.
3. Over-compressing prompts into ambiguity: Removing context to save tokens degrades model reasoning. Preserve task instructions, constraints, and output schemas. Use modular prompt templates rather than flat truncation.
4. Blind fallback to cheaper models: Fallback chains that route complex queries to lightweight models cause quality cliffs and user churn. Classify complexity first, then fallback only within capability bounds.
5. No cost-per-request observability: Without request-level cost tracking, optimization efforts are invisible. Instrument metrics for input/output tokens, cache hit rate, fallback frequency, and cost per successful response.
6. Ignoring system prompt token reuse: Sending identical system prompts on every request compounds billing. Cache system prompt tokens, use provider-specific system prompt optimization, or leverage prompt caching features where available.
7. Treating caching as a silver bullet: Cache misses still incur full model costs. If hit rates drop below 40%, the architecture is misaligned with query patterns. Analyze miss logs, adjust embedding thresholds, and refine routing rules.

Best Practices from Production

• Separate input and output cost tracking in your metrics pipeline.
• Version every prompt and model combination; roll back on quality degradation.
• Use JSON schema validation to enforce structured outputs and prevent token sprawl.
• Monitor fallback rates; a spike indicates routing misconfiguration or model instability.
• Implement budget guards: reject or queue requests when daily token thresholds are approached.
• Prefer streaming responses with early termination when the answer is complete.

Action Checklist

• Instrument token accounting: Track input, output, and retry tokens per request at the SDK wrapper level.
• Deploy semantic caching: Use embeddings or normalized hashing with TTL, versioning, and cache hit/miss metrics.
• Implement complexity routing: Classify queries by intent and map to appropriate model tiers with capability bounds.
• Enforce token boundaries: Set strict max_tokens, prune context windows, and use structured output schemas.
• Build fallback chains: Define deterministic fallback sequences that respect quality thresholds and cost caps.
• Monitor cost per request: Expose real-time cost metrics, alert on budget thresholds, and log cache/fallback statistics.
• Version prompts and models: Tag every deployment with prompt version, model ID, and routing configuration for rollback capability.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-volume repetitive queries (FAQ, lookup)	Semantic Cache + Lightweight Model	Eliminates redundant computation; cache hit rate >70%	-60% to -75%
Mixed complexity workload (SaaS app)	Tiered Routing + Context Pruning	Matches model capability to query intent; reduces context waste	-40% to -55%
Strict budget cap with quality requirements	Distilled Model + Fallback Chain	Lowest per-token cost; fallback preserves edge-case accuracy	-50% to -65%
Real-time chat with long history	Streaming + Early Termination + History Truncation	Stops generation when answer is complete; limits context window	-30% to -45%
Enterprise compliance/audit logging	Structured Output + Token Budget Guards	Enforces predictable token consumption; enables per-user billing	-20% to -35%

Configuration Template

// llm-cost-config.ts
export const LLMCostConfig = {
  cache: {
    enabled: true,
    ttlSeconds: 3600,
    embeddingThreshold: 0.85, // cosine similarity for cache match
    version: 'v2.1',
    invalidationEvents: ['prompt_update', 'model_switch', 'data_source_refresh']
  },
  routing: {
    tiers: {
      lightweight: { model: 'claude-3-haiku-20240307', maxInputTokens: 4096, maxOutputTokens: 512 },
      standard: { model: 'claude-3-sonnet-20240229', maxInputTokens: 8192, maxOutputTokens: 1024 },
      high: { model: 'claude-3-opus-20240229', maxInputTokens: 16384, maxOutputTokens: 2048 }
    },
    fallbackOrder: ['high', 'standard', 'lightweight'],
    complexityClassifier: 'keyword_pattern', // or 'embedding_model', 'fasttext'
    fallbackOn: ['rate_limit', 'timeout', 'error_5xx']
  },
  prompt: {
    maxContextChars: 8000,
    stripHistoryBefore: 3000,
    systemPrompt: 'You are a precise assistant. Output valid JSON only when requested.',
    enforceStructuredOutput: true,
    outputSchema: { type: 'object', properties: { answer: { type: 'string' }, confidence: { type: 'number' } } }
  },
  observability: {
    trackCostPerRequest: true,
    metricsPrefix: 'llm.cost',
    budgetAlertThreshold: 0.85, // 85% of daily token budget
    logCacheHits: true,
    logFallbacks: true
  }
};

Quick Start Guide

1. Initialize the wrapper: Install dependencies (@anthropic-ai/sdk, redis, crypto), import the LLMCostOptimizer class, and inject your API key and Redis URL.
2. Configure routing & caching: Copy the LLMCostConfig template, adjust embeddingThreshold, ttlSeconds, and model tiers to match your provider and workload.
3. Replace direct SDK calls: Swap raw client.messages.create() calls with optimizer.routeAndOptimize({ prompt, maxTokens }). Ensure your application handles the returned LLMResponse structure.
4. Instrument metrics: Attach a metrics emitter to the wrapper to track tokensUsed, cost, cacheHit, and fallbackTriggered. Push to your observability stack (Prometheus, Datadog, or OpenTelemetry).
5. Validate & iterate: Run a load test with mixed query types. Monitor cache hit rate, fallback frequency, and cost per 1k requests. Adjust complexity thresholds and TTLs based on telemetry.