Enterprise RAG Architecture Guide

Retrieval-Augmented Generation (RAG) is the dominant pattern for grounding LLM outputs in real documents. But most tutorials show you the happy path. Production RAG in regulated environments - Life Sciences and Healthcare especially - has a dozen additional layers of complexity. This guide covers the full stack.

Architecture Overview

A production RAG system has six discrete layers that all need to be designed independently:

  1. Document Ingestion Pipeline
  2. Embedding and Indexing
  3. Vector Database
  4. Retrieval Strategy
  5. Generation Layer
  6. Evaluation and Monitoring

Think of it as a funnel. Documents go in at the top. Answers with citations come out at the bottom. Every layer in between is a potential failure mode.

Document Ingestion Pipeline

Garbage in, garbage out. Most RAG failures trace back to poor ingestion, not bad models.

Chunking strategies: You have three real options.

Fixed chunking splits by character count (e.g., 512 tokens, 64-token overlap). Simple and fast. Works well for uniform documents like lab reports. Fails badly on structured documents where a table row split across two chunks becomes meaningless.

Semantic chunking uses embedding similarity to find natural break points - sentences where the topic shifts. More expensive (requires an embedding call per sentence) but produces chunks that preserve meaning. Good for clinical narratives and long-form regulatory text.

Recursive chunking tries document-level splits first (sections, paragraphs), then falls back to smaller splits if chunks are still too large. This is the default in LangChain's RecursiveCharacterTextSplitter and works well for most medical documents.

For healthcare specifically: a 400-600 token chunk with 10-15% overlap performs well for clinical notes. Protocol documents do better with section-level chunking (each section as one chunk) because mixing Phase I PK data with inclusion criteria creates noisy retrieval.

Metadata extraction: Every chunk should carry: document ID, source filename, section header, page number, creation date, document type (SOP vs protocol vs label), and any relevant entity tags (drug name, indication, trial ID). This metadata is your filtering layer - without it you can't restrict retrieval to relevant document types.

Embedding Models

The embedding model converts text into a vector. The quality of that vector determines retrieval quality.

OpenAI text-embedding-3-large: 3072 dimensions, strong general performance, $0.13 per million tokens. Best default choice. The 3-small variant (1536 dims) is 5x cheaper and often within 2-3% accuracy.

Cohere embed-v3: Built-in input type separation (search_document vs search_query) which helps retrieval accuracy. Strong on multilingual content - relevant if you handle EU regulatory submissions in multiple languages.

Open-source (BGE-large, E5-mistral-7B): For environments where data cannot leave your infrastructure (most clinical trials, all government health programs), you need on-prem embeddings. BGE-large-en-v1.5 (1024 dims) is competitive with commercial models and runs on a single A10G GPU.

Dimension trade-offs: More dimensions = better semantic precision, more storage, slower similarity search. For a 10M chunk index: 1536 dims at float32 = ~58GB in memory. 768 dims = ~29GB. Use product quantization (PQ) or binary quantization in your vector DB to reduce this by 8-32x with minimal accuracy loss.

Vector Databases

Pinecone, Weaviate, Qdrant, and pgvector each fit a different scenario.

Pinecone: Fully managed, zero ops overhead. Best for teams that want to move fast and have a budget. Starts at $0.096/hr per pod. Not suitable if data residency requirements prohibit SaaS. Latency is typically 20-50ms for 1M vectors.

Weaviate: Open-source with a managed cloud option. Strong multi-tenancy story - each client or study can get their own isolated namespace. Has built-in hybrid search (BM25 + vector). Good choice for multi-tenant healthcare platforms.

Qdrant: Written in Rust. Fastest raw query performance of the four, especially under high concurrency. Excellent filtering on metadata (payload filters run before ANN search, not after). Best choice when you have complex metadata filters and high QPS.

pgvector: Postgres extension. Best choice when your existing stack is Postgres and you don't want another service to operate. Handles up to ~1M vectors well. Beyond that, query times degrade. Use it for internal tools and smaller corpora (e.g., a single drug's dossier).

Rule of thumb: start with pgvector for speed, migrate to Qdrant or Weaviate when you hit 500K+ chunks or need multi-tenant isolation.

Retrieval Strategies

Vanilla cosine similarity top-k retrieval has a ceiling. These four techniques push past it.

Hybrid search: Combine dense (vector) retrieval with sparse (BM25/TF-IDF) retrieval. Dense search finds conceptually similar chunks. Sparse search finds exact keyword matches. Clinical queries often include exact drug names, NDC codes, or ICD codes where sparse retrieval wins. Reciprocal Rank Fusion (RRF) is the standard way to merge the two ranked lists.

Reranking: After retrieving top-20 chunks, pass them through a cross-encoder reranker (Cohere Rerank, or open-source BGE-reranker) to re-score against the exact query. Cross-encoders are 2-3x more accurate than bi-encoders for relevance but too slow to run on the full index. This two-stage approach is the standard for production systems. Budget 100-200ms for the rerank step.

Query expansion: Before embedding the user query, generate 3-5 alternative phrasings using a small LLM. Search with all phrasings, merge results. Particularly useful for clinical queries where "heart attack", "MI", "myocardial infarction", and "STEMI" all mean related things.

MMR (Maximal Marginal Relevance): Selects chunks that are both relevant to the query and diverse from each other. Prevents the context window from being filled with five nearly-identical chunks. Helpful for broad questions that should draw from multiple document sections.

Generation Layer

Prompt construction: Structure matters. Put system instructions first, then retrieved context (labeled by source), then the user question. Include explicit instruction: "Base your answer only on the provided context. If the context does not contain the answer, say so." This single instruction reduces hallucination rate by 40-60% in benchmark testing.

Citation injection: Number each retrieved chunk [1], [2], etc. in the prompt. Instruct the model to cite by number. Post-process the output to replace [1] with the actual document reference. This is not optional for clinical use - every claim needs a traceable source.

Guardrails: At minimum: PII detection before output (regex + NER), toxicity check, and a "refusal to answer" fallback for out-of-scope queries. For clinical decision support, add a confidence threshold - if retrieval score is below 0.7, return "insufficient context" rather than a low-confidence answer.

Evaluation Pipeline

You need three metrics to know if your RAG system works:

Retrieval precision/recall: For a test set of question-answer pairs where you know the correct source chunks, measure how often those chunks appear in your top-k results. Target: precision@5 above 0.75 for production systems.

Answer faithfulness (RAGAS): Does the generated answer stay within the bounds of the retrieved context? Automated with LLM-as-judge. Target: above 0.85 for clinical use cases.

Hallucination rate: What fraction of factual claims in the output are unsupported by the context? This requires human evaluation or a strong LLM judge. In healthcare, a hallucination rate above 2% is typically unacceptable.

Production Concerns

Latency budgets: For interactive clinical tools, total latency should be under 3 seconds. Embed query (50ms) + vector search (30-100ms) + rerank (150ms) + generation (1-2s) = budget is tight. Cache embeddings for common queries. Use streaming for generation to reduce perceived latency.

Caching: Two levels. Embedding cache: if the same query string is seen again, skip the embed call. Semantic cache: if a new query is within cosine distance 0.02 of a cached query, return the cached answer. GPTCache and Redis are common implementations.

Versioning: Embedding models change. When you upgrade from text-embedding-3-small to text-embedding-3-large, you must re-embed your entire corpus. Plan for this. Store the embedding model name and version as metadata on every chunk. Never mix chunks embedded with different models in the same index.

A/B testing: Shadow-mode A/B testing is the safe way to evaluate retrieval strategy changes. Route 5% of queries to the new strategy, compare faithfulness and user satisfaction scores against the baseline before full rollout.

Keep reading

Further Reading