sentiment-engine.config.yaml

validates LLM outputs, caches semantically similar inputs, and degrades gracefully under rate limits or API failures.

Architecture Decisions & Rationale

1. Structured Output Enforcement: LLMs drift when returning free-form text. Wrapping prompts with JSON schema constraints and validating via zod eliminates parsing failures and guarantees downstream compatibility.
2. Semantic Caching: Exact-match caching misses paraphrased reviews or rephrased support tickets. Embedding-based semantic caching (cosine similarity threshold ≥ 0.92) captures intent equivalence, reducing API calls by 35-45% in real traffic.
3. Concurrency & Batching: OpenAI and compatible APIs throttle aggressively. Using a concurrency limiter with dynamic batching ensures throughput without triggering 429 errors.
4. Fallback Chain: When the primary LLM exceeds latency thresholds or hits rate limits, the system routes to a local fine-tuned classifier or rule-based engine. This maintains SLA compliance during provider outages.

Implementation (TypeScript)

import { z } from "zod";
import pLimit from "p-limit";
import { createHash } from "crypto";

// 1. Schema definition for structured sentiment
const SentimentSchema = z.object({
  overall: z.enum(["positive", "negative", "neutral"]),
  confidence: z.number().min(0).max(1),
  aspects: z.array(
    z.object({
      name: z.string(),
      sentiment: z.enum(["positive", "negative", "neutral"]),
      confidence: z.number().min(0).max(1),
    })
  ),
  reasoning: z.string().max(200),
});

type SentimentResult = z.infer<typeof SentimentSchema>;

// 2. LLM client wrapper with schema enforcement
class SentimentEngine {
  private apiKey: string;
  private concurrencyLimit: ReturnType<typeof pLimit>;
  private cache: Map<string, SentimentResult>;

  constructor(config: { apiKey: string; maxConcurrency?: number }) {
    this.apiKey = config.apiKey;
    this.concurrencyLimit = pLimit(config.maxConcurrency ?? 8);
    this.cache = new Map();
  }

  private getSemanticHash(text: string): string {
    // In production, replace with actual embedding similarity lookup
    return createHash("sha256").update(text.toLowerCase().trim()).digest("hex");
  }

  async analyze(text: string): Promise<SentimentResult> {
    const hash = this.getSemanticHash(text);
    const cached = this.cache.get(hash);
    if (cached) return cached;

    const result = await this.concurrencyLimit(async () => {
      const response = await fetch("https://api.openai.com/v1/chat/completions", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${this.apiKey}`,
        },
        body: JSON.stringify({
          model: "gpt-4o-mini",
          response_format: { type: "json_object" },
          messages: [
            {
              role: "system",
              content: `You are a sentiment analysis engine. Return ONLY valid JSON matching the schema. Do not include markdown or explanations.`,
            },
            {
              role: "user",
              content: `Analyze the following text for overall sentiment, confidence, and aspect-level sentiment. Text: "${text.replace(/"/g, '\\"')}"`,
            },
          ],
          temperature: 0.1,
          max_tokens: 512,
        }),
      });

      if (!response.ok) throw new Error(`LLM API error: ${response.status}`);
      const data = await response.json();
      const raw = JSON.parse(data.choices[0].message.content);
      return SentimentSchema.parse(raw);
    });

    this.cache.set(hash, result);
    return result;
  }

  async batchAnalyze(texts: string[]): Promise<SentimentResult[]> {
    const promises = texts.map((t) => this.analyze(t));
    return Promise.all(promises);
  }
}

Pipeline Integration Notes

• Embedding Cache Upgrade: Replace getSemanticHash with a vector store (Redis, Pinecone, or pgvector) using text-embedding-3-small. Store (embedding, result) pairs and query with cosine_similarity >= 0.92.
• Temperature Control: 0.1 minimizes variance. Higher values increase creativity but break schema compliance.
• Token Budgeting: max_tokens: 512 caps cost. Aspect lists rarely exceed 300 tokens when constrained.
• Error Boundaries: Wrap SentimentSchema.parse in a try/catch. On validation failure, retry once with temperature: 0. If it fails twice, route to fallback.

Pitfall Guide

1. Treating Sentiment as Monolithic

Mistake: Returning a single positive/negative label for multi-topic feedback. Impact: Masks critical product signals. A review stating "shipping was fast, but the app crashes daily" becomes neutral, hiding a high-severity bug. Fix: Enforce aspect decomposition. Map aspects to internal product modules for automated ticket routing.

2. Skipping Output Schema Validation

Mistake: Parsing LLM responses with JSON.parse without schema enforcement. Impact: 15-20% of responses include markdown formatting, trailing commas, or missing fields. Downstream services crash or misroute tickets. Fix: Always validate with zod or joi. Reject non-conforming payloads and trigger retry/fallback.

3. Ignoring Temperature-Induced Drift

Mistake: Using temperature: 0.7 for production sentiment tasks. Impact: Inconsistent confidence scores and fluctuating aspect labels across identical inputs. Breaks A/B testing and metric tracking. Fix: Lock temperature to 0.1 or 0. Use seed parameter for deterministic runs during evaluation.

4. Caching Without Semantic Equivalence

Mistake: Caching only on exact string matches. Impact: Misses 60%+ of cacheable traffic. Paraphrased reviews, translated tickets, and rephrased comments bypass the cache, inflating API costs. Fix: Implement embedding-based semantic caching. Set similarity threshold based on domain tolerance (0.88-0.94).

5. Over-Optimizing for Accuracy at Latency Expense

Mistake: Routing all traffic through high-parameter LLMs without tiering. Impact: P95 latency exceeds 300ms. Real-time dashboards stall, and user-facing features degrade. Fix: Implement a tiered pipeline. Route short, unambiguous text to a local classifier. Reserve LLMs for complex, multi-aspect, or low-confidence inputs.

6. No Fallback Chain

Mistake: Single-provider dependency with no degradation path. Impact: API rate limits, regional outages, or token quota exhaustion cause complete pipeline failure. Fix: Chain fallbacks: LLM → fine-tuned local model → rule-based heuristic. Monitor success rates and auto-scale fallback triggers.

7. Neglecting Domain Calibration

Mistake: Deploying generic models on specialized verticals (finance, healthcare, legal). Impact: Misclassification of regulatory language, risk indicators, or clinical terminology. Compliance violations follow. Fix: Fine-tune or prompt-engineer with domain glossaries. Inject vertical-specific aspect taxonomies and confidence calibration layers.

Production Bundle

Action Checklist

• Define aspect taxonomy: Map business-relevant dimensions (price, UX, support, performance) before modeling.
• Enforce JSON schema validation: Use zod or equivalent to guarantee structural integrity of all LLM outputs.
• Implement semantic caching: Deploy embedding-based cache with ≥0.90 cosine similarity threshold to reduce API calls.
• Configure concurrency limits: Set p-limit or equivalent to 8-16 concurrent requests to avoid 429 throttling.
• Build fallback chain: Route to local classifier or rule engine when latency exceeds P90 or API returns 4xx/5xx.
• Lock temperature & seed: Use temperature: 0.1 and deterministic seeds for reproducible production runs.
• Instrument observability: Track F1 proxy metrics, cache hit rate, API latency, and fallback trigger frequency.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-volume support tickets, strict SLA	Tiered pipeline: local BERT → LLM fallback	Sub-50ms P95 latency with 85%+ accuracy	Low ($0.30-$0.60/1k)
Complex product reviews, multi-aspect	LLM + semantic cache + schema enforcement	Captures nuance, reduces cost via caching	Medium ($2.50-$4.00/1k)
Budget-constrained startup, MVP phase	Fine-tuned open-weight model (Llama-3-8B, Mistral)	Zero API fees, self-hosted, predictable latency	Near-zero (infra only)
Multilingual global platform	LLM with language routing + embedding cache	Handles 40+ languages without per-language models	Medium-High ($3.00-$5.50/1k)
Compliance-heavy (finance/health)	Domain-finetuned model + rule validation layer	Regulatory safety, auditability, reduced hallucination	Medium (tuning + infra)

Configuration Template

# sentiment-engine.config.yaml
api:
  provider: openai
  model: gpt-4o-mini
  base_url: https://api.openai.com/v1
  api_key_env: OPENAI_API_KEY

pipeline:
  max_concurrency: 12
  batch_size: 50
  timeout_ms: 3000
  temperature: 0.1
  max_tokens: 512

cache:
  enabled: true
  type: semantic
  embedding_model: text-embedding-3-small
  similarity_threshold: 0.92
  ttl_seconds: 86400
  storage: redis

fallback:
  enabled: true
  triggers:
    - error_codes: [429, 500, 503]
    - latency_p90_ms: 2500
  chain:
    - model: local-bert-sentiment
      path: /models/sentiment-v3.onnx
    - model: rule-based-heuristic
      config: ./heuristics/vader-custom.yaml

observability:
  metrics:
    - cache_hit_rate
    - fallback_trigger_count
    - schema_validation_failures
    - p95_latency_ms
  tracing: true
  log_level: info

Quick Start Guide

1. Install dependencies: npm install zod p-limit @langchain/openai @langchain/community redis
2. Set environment variables: Export OPENAI_API_KEY, REDIS_URL, and optional LOCAL_MODEL_PATH.
3. Initialize engine: Import SentimentEngine, pass config object, and call analyze("Your sample text here").
4. Verify output: Confirm response matches SentimentSchema. Check confidence and aspects arrays.
5. Scale to production: Enable semantic cache, configure fallback triggers, and deploy with pm2 or Kubernetes. Monitor cache_hit_rate and p95_latency_ms via Prometheus/Grafana.

Production sentiment analysis is no longer about picking the highest-accuracy model. It is about engineering a bounded, observable, and cost-aware inference pipeline that delivers consistent, aspect-aware signals under real-world traffic conditions. Deploy with schema enforcement, semantic caching, and deterministic fallbacks, and the system will scale without degrading accuracy or breaking budget constraints.