RAG architecture patterns

re LLM ingestion. Reduces token spend and attention dilution. 4. Pipeline Composition: Each stage returns a typed envelope with metadata for observability. Failures are captured, not swallowed.

TypeScript Implementation

import { OpenAIEmbeddings } from "@langchain/openai";
import { BM25Retriever } from "@langchain/community/retrievers/bm25";
import { Chroma } from "@langchain/community/vectorstores/chroma";
import { ChatOpenAI } from "@langchain/openai";
import { StringOutputParser } from "@langchain/core/output_parsers";
import { RunnableSequence, RunnablePassthrough } from "@langchain/core/runnables";
import type { Document } from "@langchain/core/documents";

interface RAGConfig {
  embeddingsModel: string;
  llmModel: string;
  topK: number;
  reRankThreshold: number;
  maxContextTokens: number;
}

interface PipelineResult {
  query: string;
  retrieved: Document[];
  reRanked: Document[];
  compressedContext: string;
  generation: string;
  metadata: Record<string, unknown>;
}

export class ProductionRAG {
  private config: RAGConfig;
  private embeddings: OpenAIEmbeddings;
  private vectorStore: Chroma;
  private bm25Retriever: BM25Retriever;
  private llm: ChatOpenAI;
  private outputParser: StringOutputParser;

  constructor(config: RAGConfig, vectorStore: Chroma, documents: Document[]) {
    this.config = config;
    this.embeddings = new OpenAIEmbeddings({ model: config.embeddingsModel });
    this.vectorStore = vectorStore;
    this.bm25Retriever = new BM25Retriever.fromDocuments(documents);
    this.llm = new ChatOpenAI({ model: config.llmModel, temperature: 0.1 });
    this.outputParser = new StringOutputParser();
  }

  private async hybridSearch(query: string): Promise<Document[]> {
    const vectorResults = await this.vectorStore.similaritySearch(query, this.config.topK * 2);
    const bm25Results = await this.bm25Retriever.invoke(query);
    
    // Merge and deduplicate by page content hash
    const seen = new Set<string>();
    const merged: Document[] = [];
    for (const doc of [...vectorResults, ...bm25Results]) {
      const hash = doc.pageContent.slice(0, 100);
      if (!seen.has(hash)) {
        seen.add(hash);
        merged.push(doc);
      }
    }
    return merged.slice(0, this.config.topK * 2);
  }

  private async reRank(docs: Document[], query: string): Promise<Document[]> {
    // In production, replace with a hosted cross-encoder or local Cohere/BGE-reranker
    const scored = docs.map(doc => ({
      doc,
      score: this.heuristicRelevance(query, doc.pageContent)
    }));

    const filtered = scored
      .filter(item => item.score >= this.config.reRankThreshold)
      .sort((a, b) => b.score - a.score)
      .map(item => item.doc);

    return filtered.slice(0, this.config.topK);
  }

  private heuristicRelevance(query: string, content: string): number {
    const qWords = new Set(query.toLowerCase().split(/\W+/));
    const cWords = content.toLowerCase().split(/\W+/);
    let matches = 0;
    for (const w of cWords) if (qWords.has(w)) matches++;
    return matches / Math.max(qWords.size, 1);
  }

  private compressContext(docs: Document[]): string {
    // Production: replace with LLM-based summarization or extractive compression
    return docs.map(d => d.pageContent).join("\n\n").slice(0, this.config.maxContextTokens * 4);
  }

  private buildPrompt(context: string, query: string): string {
    return `You are a precise technical assistant. Answer using ONLY the provided context.
Context:
${context}

Question: ${query}
Answer:`;
  }

  async run(query: string): Promise<PipelineResult> {
    const rawRetrieved = await this.hybridSearch(query);
    const reRanked = await this.reRank(rawRetrieved, query);
    const compressed = this.compressContext(reRanked);
    const prompt = this.buildPrompt(compressed, query);
    
    const chain = RunnableSequence.from([
      new RunnablePassthrough(),
      this.llm,
      this.outputParser
    ]);

    const generation = await chain.invoke(prompt);

    return {
      query,
      retrieved: rawRetrieved,
      reRanked,
      compressedContext: compressed,
      generation,
      metadata: {
        retrievalCount: rawRetrieved.length,
        postReRankCount: reRanked.length,
        contextTokens: Math.ceil(compressed.length / 4),
        timestamp: new Date().toISOString()
      }
    };
  }
}

Architectural Rationale

• Separation of Retrieval and Ranking: Hybrid search casts a wide net; re-ranking narrows it. This decoupling allows independent tuning of recall vs precision.
• Explicit Context Boundaries: maxContextTokens prevents unbounded expansion. Production systems must enforce hard limits before LLM ingestion.
• Observable Envelopes: Every stage returns metadata. This enables downstream evaluation pipelines (RAGAS, Arize Phoenix) to trace degradation to specific stages.
• Replaceable Components: The re-ranker and compressor are abstracted. Swapping heuristic scoring for a cross-encoder or LLM-based compressor requires zero pipeline restructuring.

Pitfall Guide

1. Fixed-Size Chunking Without Overlap or Semantic Awareness Splitting documents at exact byte/word boundaries fractures tables, code blocks, and logical arguments. Production impact: retrieval precision drops 15–20% because chunks lose contextual anchors. Mitigation: Use recursive character splitting with 10–15% overlap, or semantic chunking via sentence boundary detection + embedding similarity thresholds.

2. Vector-Only Search for Technical/Domain Content Embeddings struggle with exact identifiers, version numbers, API endpoints, and numerical constraints. Relying solely on cosine similarity guarantees lexical misses. Mitigation: Always pair dense retrieval with BM25 or SPLADE. Weight lexical matches higher for query types containing alphanumeric patterns.

3. Skipping Re-Ranking Top-K vector results contain high-recall but low-precision candidates. Feeding noise into the LLM wastes tokens and degrades attention allocation. Mitigation: Insert a cross-encoder re-ranker (BGE-reranker, Cohere, or ColBERT) between retrieval and generation. Expect 200–400ms overhead for 30–40% precision gains.

4. Unbounded Context Windows Developers assume "more context = better answers". LLMs exhibit lost-in-the-middle degradation and attention dilution beyond 8k–12k tokens. Mitigation: Enforce strict token budgets. Compress, summarize, or extract before generation. Route long-context queries to specialized pipelines.

5. Static Embeddings Without Incremental Updates Knowledge drift is inevitable. Re-embedding entire corpora daily is cost-prohibitive and causes index fragmentation. Mitigation: Implement append-only ingestion with document versioning. Use hybrid incremental updates: only re-embed changed sections. Maintain tombstone markers for deleted content.

6. Missing Evaluation Loop Accuracy is assumed, not measured. Without RAG-specific metrics, teams cannot distinguish retrieval failure from generation failure. Mitigation: Deploy automated evaluation using RAGAS or proprietary rubrics. Track context precision, answer relevance, and faithfulness per pipeline version. Gate deployments on metric thresholds.

7. Ignoring Query Type Distribution Not all queries require the same pipeline. Factual lookups, multi-hop reasoning, and comparative analysis demand different retrieval strategies. Mitigation: Implement query classification routing. Direct simple lookups to lightweight pipelines, complex reasoning to modular/graph patterns. Log distribution shifts to adjust routing weights.

Production Bundle

Action Checklist

• Define query type taxonomy: Classify expected queries into factual, comparative, procedural, and multi-hop categories to drive routing decisions.
• Implement hybrid retrieval baseline: Deploy BM25 + dense vector search with weighted scoring before adding re-rankers or graph layers.
• Enforce context token budgets: Set hard limits on pre-generation context and implement compression or extraction fallbacks.
• Insert cross-encoder re-ranking: Replace heuristic scoring with a production re-ranker and validate precision gains on holdout sets.
• Instrument pipeline observability: Log retrieval counts, re-rank scores, context token usage, and generation latency per request.
• Deploy automated evaluation: Integrate RAGAS or custom rubrics to track context precision, faithfulness, and answer relevance on every deployment.
• Establish incremental embedding workflow: Version documents, track change deltas, and re-embed only modified segments to control cost and drift.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Internal FAQ / SaaS documentation	Advanced RAG (Hybrid + Re-rank)	High lexical precision needed; re-ranker filters noise efficiently	+15% vs naive, -40% support tickets
Multi-step troubleshooting / policy reasoning	Modular RAG (Self-Correction + Multi-hop)	Requires iterative retrieval and verification steps	+35% compute, -60% hallucination
Cross-document synthesis / regulatory compliance	Graph RAG + Vector	Entity relationships and citation tracing prevent fabricated links	+55% infrastructure, +90% auditability
Low-latency customer chat (<500ms)	Lightweight Hybrid + Prompt Compression	Re-rankers exceed latency budget; compression preserves signal	+8% cost, meets SLA thresholds

Configuration Template

// rag.config.ts
export interface RAGPipelineConfig {
  mode: 'advanced' | 'modular' | 'graph';
  retrieval: {
    topK: number;
    hybridWeights: { vector: number; lexical: number };
    reRank: {
      enabled: boolean;
      model: string;
      threshold: number;
      maxCandidates: number;
    };
  };
  context: {
    maxTokens: number;
    compression: 'extract' | 'summarize' | 'none';
    chunkOverlapPercent: number;
  };
  generation: {
    model: string;
    temperature: number;
    maxTokens: number;
    fallbackModel?: string;
  };
  observability: {
    logRawRetrieval: boolean;
    metricEndpoint: string;
    evaluationThresholds: {
      contextPrecision: number;
      faithfulness: number;
    };
  };
}

export const defaultConfig: RAGPipelineConfig = {
  mode: 'advanced',
  retrieval: {
    topK: 8,
    hybridWeights: { vector: 0.6, lexical: 0.4 },
    reRank: {
      enabled: true,
      model: 'BAAI/bge-reranker-v2-m3',
      threshold: 0.72,
      maxCandidates: 32
    }
  },
  context: {
    maxTokens: 4096,
    compression: 'extract',
    chunkOverlapPercent: 12
  },
  generation: {
    model: 'gpt-4o-mini',
    temperature: 0.1,
    maxTokens: 1024,
    fallbackModel: 'claude-3-haiku'
  },
  observability: {
    logRawRetrieval: false,
    metricEndpoint: '/api/v1/metrics/rag',
    evaluationThresholds: {
      contextPrecision: 0.75,
      faithfulness: 0.80
    }
  }
};

Quick Start Guide

1. Initialize vector store and ingest sample documents: Run the embedding pipeline on a 500-document subset. Configure recursive chunking with 12% overlap and load into Chroma/Pinecone/Weaviate.
2. Deploy hybrid retrieval layer: Instantiate BM25 and vector retrievers. Apply the defaultConfig hybrid weights. Validate recall on 50 held-out queries.
3. Attach re-ranker and context compressor: Enable the cross-encoder re-ranker. Set maxTokens: 4096 and compression: extract. Run latency benchmark; ensure P95 stays under 800ms.
4. Connect LLM and observability: Wire the generation stage to your model provider. Enable metric logging to /api/v1/metrics/rag. Execute evaluation suite; gate deployment if contextPrecision < 0.75.