RAG Architecture Decision Guide: When to Use What

The Wrong Architecture Cost Us Six Months

We were building a clinical decision support tool for a major health system. Doctors needed fast answers: given a patient's labs, imaging, and history, what's the most likely diagnosis and what should we test next? Simple enough on paper.

Our first instinct was straightforward RAG. We chunked 50GB of clinical literature, threw it at a vector database, and pointed an LLM at it. The system worked. Answers came back in under 2 seconds. Clinicians hated it.

The problem wasn't speed or accuracy on obvious cases. It was edge cases. A 52-year-old woman with elevated liver enzymes and fatigue would retrieve textbook hepatitis content, miss the fact that she was on a specific medication that caused autoimmune hepatitis, and confidently suggest workups for viral hepatitis. The retrieval was "correct" but clinically dangerous because it lacked context about drug interactions and patient-specific nuances.

We rebuilt it. Twice. First as a hybrid system mixing RAG with fine-tuned embeddings. Then as a multi-stage system combining retrieval with entity extraction and structured reasoning. The second rebuild took three months of engineering and validation that we could have avoided with better upfront architecture thinking.

This is the real problem RAG architecture decisions hide: there's no one-size-fits-all answer. You need a framework to match the right architecture to your actual use case, data characteristics, and operational constraints. That's what I'm sharing here.

Understanding Your Actual Constraint

Before you pick an architecture, you need to name the real bottleneck. Most teams skip this and start building. Don't.

RAG architectures sit on a spectrum. On one end: simple retrieval augmented generation. You embed queries, fetch similar documents, pass them to an LLM. It's stateless, fast, and good for factual Q&A. On the other end: dense retrieval chains with ranking, re-ranking, entity extraction, knowledge graphs, and multi-hop reasoning. It's stateful, slower, and handles complex reasoning better.

Your constraint determines where on that spectrum you actually need to be. I've seen teams build toward the complex end because it felt like the "real" AI approach. I've also seen teams stay at the simple end and suffer from cascading failures because they needed reasoning they couldn't express through retrieval.

Here's how to diagnose your actual constraint:

  • Recall constraint: Are queries failing because the system can't find relevant information? This shows up as "the answer is in the docs but the system gave you something else." Clinical decision support, legal research, and compliance checking hit this hard. You need better retrieval.
  • Precision constraint: Are you getting relevant results but noisy ones? The system retrieves three documents, two are useful, one contradicts the question. This is common in customer support and product documentation. You need better ranking or filtering.
  • Reasoning constraint: Are you getting relevant information but the LLM can't synthesize it correctly? The docs say "Patient has condition A and is on drug B" but the model doesn't infer "A+B is contraindicated." You need structured reasoning, not just better retrieval.
  • Latency constraint: Does your use case demand sub-second responses? Clinical workflows, real-time monitoring, and interactive applications have hard latency budgets. Simple RAG wins here. Multi-stage systems may not fit.
  • Freshness constraint: How often does your source data change? Constantly-updating information (news, stock prices, real-time monitoring) needs different architecture than static knowledge (medical guidelines, legal precedent). Real-time indexing vs. batch processing is a fundamental split.
  • Hallucination constraint: How bad is wrong information? In research synthesis, slightly inaccurate is maybe acceptable. In clinical medicine or legal advice, hallucination is unacceptable. This drives toward retrieval-grounded approaches and away from pure generation.

Most projects actually have 2-3 primary constraints, not one. Name them explicitly. Write them down. This is the most important step and I've never seen a team regret spending 2 hours getting this right.

A Decision Framework That Actually Works

Once you name your constraints, here's the framework I use to pick an architecture:

Start with a constraint matrix. Make three columns: constraint, impact (high/medium/low), and current capability. For the clinical decision support tool, it looked like this:

  • Recall: High impact, low capability (missing edge cases)
  • Reasoning: High impact, low capability (drug interactions not surfaced)
  • Latency: Medium impact, medium capability (2 seconds acceptable, not 10)
  • Hallucination: High impact, medium capability (sometimes wrong)
  • Freshness: Medium impact, medium capability (guidelines change monthly)

This immediately told us: don't optimize for latency, optimize for recall and reasoning. Simple RAG was the wrong choice.

Map constraints to architectural patterns. High recall + high reasoning = you need retrieval that goes beyond keyword/semantic similarity. High precision + latency = you need fast ranking. High hallucination risk = you need grounding and verification.

Here's the mapping I've found reliable:

  • Simple RAG (embed -> retrieve -> generate): High precision, low hallucination risk, good latency, weak reasoning. Use for: FAQs, customer support, product documentation, straightforward fact lookup.
  • Hierarchical RAG (chunk at multiple levels): Better recall on complex documents, moderate latency. Use for: technical documentation, legal contracts, long clinical notes where context matters.
  • Multi-stage retrieval (retrieve -> rank -> re-rank): High precision, acceptable latency, medium recall. Use for: semantic search, information retrieval where you want the best results in top-5.
  • Retrieval + reasoning (retrieve facts, structure them, reason explicitly): High recall, high reasoning, slower, grounded outputs. Use for: medical diagnosis, legal analysis, technical troubleshooting, any domain where you need to infer beyond what's explicitly stated.
  • Knowledge graphs + RAG: Excellent for multi-hop reasoning and relationship discovery. Slower to build and maintain. Use for: entity-rich domains (pharma drug interactions, organizational hierarchies, supply chain), when relationships matter as much as facts.
  • Agentic RAG (retrieval as a tool within reasoning loop): Best reasoning, slowest, highest complexity. Use for: research synthesis, root cause analysis, domains where you don't know what you need to retrieve until you start reasoning.

For our clinical tool: retrieval + reasoning (with knowledge graph for drug interactions) was the right call. Not the flashiest architecture, but it matched our constraints.

Building It Step by Step

Let's say you've decided on multi-stage retrieval. Here's how you actually build it without wasting three months:

Phase 1: Establish a baseline (1-2 weeks)

Build simple RAG first. Seriously. Chunk your documents, embed them, set up a basic vector store, wire it to an LLM. It takes a week. You'll learn more from a working baseline than from architecture debates. Run it against your test set. Measure recall, precision, latency. This becomes your north star.

For the clinical tool, simple RAG gave us 62% recall on test cases. Good enough to see it fail. That 62% number was crucial - we knew exactly what we were trying to improve.

Phase 2: Diagnose failures (1-2 weeks)

Take the cases where simple RAG failed. Actually read them. Categorize failures:

  • Retrieval failures: the right document exists but wasn't retrieved
  • Ranking failures: the right document was retrieved but ranked low
  • Generation failures: the right documents were provided but the LLM misinterpreted them

For us: 40% of failures were retrieval (drug interaction information was in the database but semantic search didn't find it), 35% were reasoning (docs were retrieved but the model didn't synthesize across multiple documents), 25% were ranking (too many results, good answer buried).

This distribution tells you exactly what to build next. We needed better retrieval for entities (drugs) and better ranking. This is not obvious without doing the work.

Phase 3: Targeted improvement (2-4 weeks)

Don't rebuild everything. Fix the top failure category first. For us: retrieval. We added entity recognition before embedding queries. Instead of searching for "patient on lisinopril with elevated liver enzymes", we extracted "drug: lisinopril, finding: hepatotoxicity" and used that to bias retrieval toward drug-safety content. Recall jumped to 78%.

Then we tackled ranking. We added a learned ranker on top of semantic search. Not a heavy model - just a small classifier that scored retrieved documents. This pushed precision up and got good answers into the top-3. Latency stayed under 3 seconds.

Phase 4: Structured reasoning (3-6 weeks)

Only after retrieval and ranking are working do you add reasoning. We added a structured extraction step: given retrieved documents, extract entities and relationships. Then explicit reasoning: if drug X causes adverse effect Y and patient is on drug X, flag it. This is the phase where you might add a knowledge graph, but only if your failures specifically require it.

We didn't need a full knowledge graph. We needed to connect drug names to known hepatotoxic drugs. A small structured reference table plus explicit reasoning rules was enough.

Phase 5: Verification and grounding (ongoing)

At each stage, measure against your original constraint list. Did latency creep up past acceptable? Are hallucinations increasing? Is recall actually better on real cases or just your test set?

We validated everything against a holdout test set of 200 real patient cases reviewed by clinicians. That's the only score that mattered. Test set metrics are useful signals, but clinical relevance is the truth.

Mistakes I've Seen (And Made)

Building for scale before proving utility. Teams optimize for throughput, latency, and cost before validating that the system actually works for real use cases. You end up with a fast system that gives wrong answers. I've seen this destroy projects. Build for correctness first. Scale the correct solution.

Assuming vector similarity is retrieval. It's not. Vector databases are great for semantic search but terrible for exact matching, numeric ranges, and categorical filtering. For clinical data: a vector search might find "elevated liver enzymes" but miss "patient is on acetaminophen" (crucial context). Use hybrid search. Combine vector similarity with keyword/attribute matching. This alone improved our clinical tool's recall by 12 points.

Fine-tuning for the wrong problem. Teams fine-tune embeddings or language models thinking it'll fix retrieval. Sometimes it helps. Usually it doesn't. The problem is often architectural (you need structured reasoning, not better embeddings). I've watched teams spend 4 weeks fine-tuning embeddings for 2 point gains when a better chunking strategy would've given 15 points. Diagnosis before solution.

Ignoring chunking strategies. Everyone talks about retrieval and generation. Nobody talks about chunking. It's where a huge amount of failure lives. Chunks too small: you lose context. Chunks too large: retrieval becomes noisy. For clinical documents, we shifted from fixed 512-token chunks to semantic boundaries (by clinical concept). Retrieval precision jumped 18 points. Chunking is not boring - it's where half your wins live.

Not measuring what you care about. Teams measure NDCG and MRR because they're standard IR metrics. They don't measure whether the system actually helps users. We measured both: retrieval metrics plus clinician satisfaction and decision quality. Turns out you can have high retrieval metrics and still have clinicians distrust the system if explanations don't match their mental model. Measure what matters for your actual user.

Treating RAG as retrieval-only. RAG is "Retrieval Augmented Generation" but teams focus 80% on retrieval and 20% on the generation side. The LLM side matters enormously. Prompt engineering, model selection, output grounding - these can be bigger levers than retrieval optimization. We spent weeks improving retrieval by 5 points. Switching from GPT-3.5 to GPT-4 and using better prompts gave us 12 points. The models matter.

Real Examples: Where Each Architecture Won

Simple RAG for a medical device company's support system

They had 2,000 customer support tickets a week, mostly "how do I..." questions. Simple RAG: embed the knowledge base, retrieve relevant docs, generate answers. Fast, cheap, worked. 88% of tickets resolved without escalation. Why it worked: questions had clear answers in the knowledge base, latency mattered (support reps needed answers in seconds), hallucination was low-risk (worst case: customer gets sent to human support). Simple RAG was perfect. Over-architecture would've wasted resources.

Hierarchical RAG for a biotech contract analysis tool

Analyzing licensing agreements, clinical trial protocols, manufacturing contracts. Documents were 50-200 pages, dense with technical detail. Simple retrieval failed because relevant information was scattered across sections. We chunked hierarchically: document -> section -> paragraph. Retrieval first found the right document, then the right section, then the relevant paragraph. Precision and recall both improved. The hierarchy matched the actual structure of how humans read contracts.

Multi-stage retrieval for a MedTech search application

Sales engineers needed to find clinical evidence quickly. "Show me studies on this device's efficacy in this patient population." Simple semantic search gave too many results, good ones buried. We added a second stage: retrieve 50 candidates semantically, then rank by relevance to the specific population, study quality, recency. Top-5 results were now 85% relevant vs. 40% with simple retrieval. Still fast (sub-1 second), better results.

Retrieval + reasoning for a payer's prior authorization system

Pre-auth needed to check: is this medication indicated for this diagnosis? Is the patient on conflicting medications? Does the clinical picture match guideline recommendations? You can't answer these by retrieving treatment guidelines and hoping the LLM reasons correctly. We built explicit retrieval (fetch relevant guidelines), structured extraction (extract indication, contraindications, preferred agents), and rule-based reasoning (check if patient data matches guideline criteria). Accuracy jumped from 78% to 94%. Reasoning wasn't implicit in RAG, it was explicit in the architecture.

Agentic RAG for a pharma research synthesis system

Researchers needed to synthesize safety signals across clinical trials, case reports, pharmacovigilance databases. You don't know what to retrieve until you start investigating. We built an agent: given a safety question, the system decides what to retrieve (specific drug interactions? adverse event profiles? trial populations?), retrieves it, reasons about what it found, and decides if it needs more information. Slower (30-60 seconds) but researchers got synthesized answers they couldn't have generated manually. High-complexity architecture justified by complexity of the problem.

Your Next Steps

This week: Write down your constraints. Spend 90 minutes with your team listing the actual hard requirements. Recall? Latency? Reasoning? Hallucination risk? Freshness? Name them. Estimate their impact and your current capability. Don't skip this.

Next week: Build simple RAG. Actually build it, don't design it. Run your test cases. Measure recall, precision, latency. Document what fails and why. This is your baseline.

Two weeks out: Diagnose your top failures. Are they retrieval? Ranking? Generation? Reasoning? Don't try to fix everything - fix the top category. That's your Phase 2

Related posts

Further Reading