GenAI Build vs Buy: A Decision Matrix for Enterprise LLM Decisions

The build vs buy question for LLMs is not the same as the classic build vs buy question for enterprise software. The dynamics are different: the technology is moving too fast for long build cycles to make sense, but vendor lock-in with LLMs carries real risk because model deprecations happen faster than software EOLs. This framework helps you make a defensible decision with a clear audit trail.

Decision Criteria

Ten criteria determine whether to build or buy. Not all criteria are equally important - weight them by your organization's context.

1. Data Sensitivity - Can your data leave your environment? Clinical trial data, PHI, and MNPI (material non-public information) often legally cannot. If the answer is "no", a SaaS LLM vendor is immediately out of scope for that use case. Build or use on-prem/private cloud deployment.

2. Customization Depth - How far does the base model need to be customized for your use case? General-purpose tasks (summarization, Q&A over internal docs) rarely need fine-tuning. Specialized tasks (medical coding from clinical notes, regulatory writing with specific house style) often do. Fine-tuning requires building.

3. Latency Requirements - What is the maximum acceptable response time? Under 500ms real-time latency is hard to hit with API-based vendors (network round-trip + queuing). If you need sub-500ms, on-prem or self-hosted models on dedicated compute are likely required.

4. Team Capability - Do you have the ML engineering talent to build and maintain a custom LLM stack? A production RAG system with fine-tuned models, a vector database, evaluation pipeline, and monitoring requires 2-4 senior ML engineers to build and 1-2 to maintain. If that capability doesn't exist in-house, buying is more realistic than hiring.

5. Regulatory Constraints - Does your use case touch FDA-regulated activities (drug development, clinical decision support as a medical device), EU AI Act high-risk categories, or SOX-covered financial processes? Each adds validation requirements. SaaS vendors increasingly offer compliance packages (HIPAA BAAs, SOC2 Type II, FedRAMP) but may not meet 21 CFR Part 11 requirements for clinical trial data without additional controls.

6. Total Cost Over 3 Years - Build cost and buy cost are rarely compared honestly. See the cost model below.

Scoring Matrix

Score each criterion 1-5 where 5 = strong signal to build, 1 = strong signal to buy. Multiply by the weight. Total score above 35 = lean build, below 20 = lean buy, 20-35 = hybrid or case-by-case.

| Criterion | Weight | Buy (1-2) | Neutral (3) | Build (4-5) | | --- | --- | --- | --- | --- | | Data Sensitivity | 3x | Public/internal data OK for SaaS | Some sensitive data | PHI/PII/MNPI that cannot leave org | | Customization Depth | 2x | Standard task, prompt engineering sufficient | RAG + some fine-tuning | Deep domain adaptation required | | Latency Requirements | 2x | 3+ seconds acceptable | 1-3 seconds | Under 500ms required | | Team Capability | 2x | No ML team | Small ML team | Strong ML engineering team | | Regulatory Constraints | 3x | No regulated activities | Some compliance needs | FDA/21 CFR/EU AI Act high-risk | | Volume | 1x | Under 100K API calls/month | 100K-5M/month | Over 5M/month | | Time to Market | 2x | Need live in under 3 months | 3-6 months | Over 6 months acceptable | | Vendor Availability | 1x | Multiple mature vendors | 1-2 vendors | No vendor exists for use case | | Strategic Differentiator | 2x | Commodity capability | Important but not unique | Core IP and competitive moat | | Maintenance Tolerance | 1x | Want vendor to handle updates | Some maintenance OK | Full control required |

Cost Model

Most build vs buy analyses undercount build costs by 2-3x and undercount buy costs by 1.5x. Here is a more honest accounting.

Build costs over 3 years:

  • ML engineering (2 engineers x $250K fully loaded x 3 years) = $1.5M
  • Initial build time (6 months at 2 FTE) = $250K
  • Compute (fine-tuning: $5-50K one-time; inference: depends on volume - a 13B parameter model on 2 A100s costs about $2K/month)
  • Data annotation for fine-tuning ($50-200K depending on complexity and volume)
  • Evaluation and testing infrastructure ($20-50K)
  • Total range: $1.8M to $3M over 3 years for a mid-complexity use case

Buy costs over 3 years:

  • Licensing/API: $0.01-$0.06 per 1K tokens for GPT-4o class models. At 10M tokens/month: $1,200-$7,200/month or $43K-$259K over 3 years.
  • Integration engineering (3-6 months at 1-2 engineers) = $100-200K
  • Vendor management and contract overhead = $30-50K/year
  • Migration risk: if the vendor changes pricing, deprecates a model, or gets acquired, migration costs $100-300K
  • Total range: $350K to $1.2M over 3 years for mid-volume use case

The break-even point is typically around 50M tokens/month for high-complexity tasks where a strong commercial model is needed. Below that, buy almost always wins on pure cost. Above that, build becomes competitive.

Risk Assessment

Vendor dependency: OpenAI has deprecated GPT-3.5-turbo-0301, GPT-4-0314, and multiple embeddings models, typically with 6-12 months notice. If your application is tightly coupled to a specific model version, deprecation forces a rushed migration. Mitigation: build an abstraction layer over any LLM vendor so swapping providers is a configuration change, not a rewrite. LiteLLM and LangChain's model abstraction provide this.

Data residency: EU data protection law (GDPR Article 46), HIPAA, and clinical trial regulations (21 CFR Part 312) impose requirements on where data can be processed. Even if a vendor is HIPAA-compliant, their data centers may be in the wrong jurisdiction for EU clinical trial data. Verify data residency per use case, not per vendor relationship.

Model deprecation: Unlike software with 5-10 year support cycles, LLM models have typical support windows of 12-24 months. Plan for model migration every 18 months as a maintenance item, not a project.

Compliance liability: When a SaaS LLM makes an error in a regulated context (wrong ICD code, incorrect safety signal assessment), who is liable? Review vendor terms of service carefully. Most LLM vendors explicitly disclaim liability for use in medical, legal, or financial contexts. This shifts all liability to the deploying organization regardless of where the error originated.

Decision Tree

Work through these questions in order:

  1. Can the data leave your environment? If no - build or private deployment. Stop here.
  1. Is the use case core IP or competitive differentiator? If yes - build or private deployment. Stop here.
  1. Do you need this live in under 3 months? If yes - buy. Stop here (you can migrate later).
  1. Does your team have 2+ ML engineers available? If no - buy. Stop here.
  1. Does your volume exceed 50M tokens/month for a task requiring GPT-4 class quality? If yes - evaluate build economics. Otherwise - buy.
  1. Is this a regulated activity requiring validated software? If yes - build with full validation, or buy from a vendor offering validated deployment. If buying, confirm vendor can support 21 CFR Part 11 or equivalent.

Default: buy and iterate. The cost of getting started with a vendor is low. The cost of getting started with a custom build is high. You can always migrate later with more information.

Three Worked Examples

Example 1: Build wins - Adverse Event Narrative Generation (Pharma)

A top-10 pharma company needs to automate ICSR narrative drafting. Volume: 50K narratives/year. Data: patient PHI and clinical trial data under FDA IND. Customization: narratives must follow company house style and ICH E2B(R3) exactly. Regulatory: FDA submission artifact, requires 21 CFR Part 11 audit trail. Team: 4 ML engineers in place.

Decision: Build. Data cannot leave the environment (patient PHI, IND data). The task is core to regulatory compliance. The team exists. Over 3 years, a custom fine-tuned model on company data will outperform a generic LLM on this task by a meaningful margin. Build score: 42/50.

Example 2: Buy wins - Internal Knowledge Assistant (Health System)

A regional health system wants an internal Q&A bot over their policy and procedure documents. Volume: 5K queries/day. Data: internal policy docs, no PHI. Customization: minimal, standard RAG. Regulatory: none for the documents themselves. Team: 1 ML engineer, heavy workload.

Decision: Buy. Microsoft Copilot for Healthcare or a similar product handles this use case. Data is not sensitive (no PHI). The task is commodity. Time-to-market matters (leadership wants a demo in 60 days). Build score: 17/50.

Example 3: Hybrid wins - Clinical Trial Protocol Authoring (CRO)

A CRO wants to accelerate protocol synopsis writing. Volume: 200 protocols/year. Data: sponsor protocols contain MNPI during pre-submission but become public post-approval. Customization: needs to follow ICH E6(R3) and specific therapeutic area conventions. Regulatory: not an FDA submission artifact directly, but inputs often are. Team: 2 ML engineers.

Decision: Hybrid. Use a commercial model API (Azure OpenAI with HIPAA BAA and private endpoint) for generation - this keeps compute managed while satisfying data residency for most sponsors. Build a custom RAG layer over internal protocol templates and ICH guidelines. Fine-tune a smaller model for protocol structure classification. Total cost: ~$400K over 3 years vs $1.8M for full build. The hybrid approach captures 80% of the quality benefit at 25% of the cost.

More on this

Further Reading