Latency, Accuracy, Cost: Pick Two

There's a version of the CAP theorem for AI systems: latency, accuracy, cost - pick two. You can have low-latency and high-accuracy inference, but it will be expensive. You can have high-accuracy and low-cost inference, but it will be slow. You can have low-latency and low-cost inference, but you'll sacrifice accuracy.

Like most pithy frameworks, this is a simplification. The actual tradeoff space is a three-dimensional surface, not three discrete points. But the underlying insight is real: optimizing one dimension puts pressure on the other two. And the organizations that build good AI systems are the ones that choose their constraints deliberately before they start building.

Why This Triangle Exists

The triangle exists because of how ML inference works at a systems level. Accuracy correlates with model size and complexity - larger models, more parameters, more compute per inference. Latency is inversely related to inference compute - more compute means slower responses unless you add more hardware, which increases cost. Cost is a function of compute and hardware - cutting cost means reducing one or both.

You can partially escape the triangle through engineering optimizations: model quantization, knowledge distillation, speculative decoding, caching, batching. These shift the tradeoff frontier without eliminating it. A quantized model is faster and cheaper with modest accuracy loss. Batching amortizes inference cost across requests at the cost of latency per request. These are real tools. They reduce the tension; they don't eliminate it.

The triangle also manifests differently at different scales. At low request volumes, cost is negligible and the tension is mainly between latency and accuracy. At high request volumes, cost becomes the binding constraint and everything else flows from cost management decisions. Understanding which regime you're in changes what the optimal tradeoff looks like.

Healthcare: Accuracy First, Latency Second, Cost Managed

Clinical decision support is the case where accuracy requirements dominate everything else. When an AI system is advising on sepsis risk, medication dosing, or clinical trial eligibility, the cost of a wrong answer can include patient harm. Accuracy is not a preference - it's a regulatory and ethical floor.

This doesn't mean latency is irrelevant. In emergency settings - code alerts, rapid response systems, deterioration prediction - latency matters because the intervention window is narrow. A system that gives a 96% accurate prediction in 45 seconds might still be better than a system that gives a 94% accurate prediction in 5 seconds, but the calculus depends on how long the intervention window is.

What this looks like in practice: clinical AI systems often use larger, more expensive models than you'd expect for the task, because the accuracy requirement justifies the cost. Epic's Sepsis prediction model runs inference on every patient in the hospital at regular intervals - this is expensive at scale. They pay the cost because the alternative is using a cheaper model that's less accurate on the edge cases that matter most.

Cost in Life Sciences & Life Sciences & Healthcare AI is managed through batching - run inference on all patients every 15 minutes rather than every patient in real-time - tiered inference, using a fast cheap model for triage and a slow expensive model for confirmed candidates, and scope limitation, running the expensive model only on the patient population where it's clinically actionable.

Fintech: Latency First, Accuracy Second, Cost Managed

Payment fraud detection is the case where latency is the binding constraint. A fraud detection system that takes 2 seconds to return a decision is unusable in a checkout flow - you've already lost the customer. The requirement is sub-100ms p99 latency.

At that latency requirement, your model architecture choices are severely constrained. You're not running a large language model for fraud detection. You're running gradient boosted trees, shallow neural networks, or highly optimized deep learning models with aggressive caching. The accuracy you can achieve at sub-100ms is limited by how much compute you can pack into that window.

Stripe's fraud detection architecture is a good example of how the best systems work through this. Their approach is layered: a very fast rule-based system runs first (sub-10ms) and handles the obvious cases. A medium-speed ML model runs second (50-80ms) and handles the moderate-risk cases. A slower, more accurate model runs asynchronously for post-transaction review and model improvement. The latency constraint is met by the first two layers; the accuracy requirement is maintained by the third layer's continuous feedback into model training.

This tiered approach is the standard playbook for latency-constrained AI: fast models for real-time decisions, accurate models for asynchronous review and learning.

Retail: Cost First, Accuracy Second, Latency Managed

Product recommendations at retail scale - billions of recommendation calls per day across a catalog of millions of products - is the case where cost is the primary constraint. The accuracy requirements are real but tolerant of some degradation. The latency requirements are real but achievable with standard caching architectures. But the cost of inference at this scale is enormous if you don't manage it aggressively.

Amazon's recommendation system is famous for its engineering sophistication, but a lot of that sophistication is cost management: aggressive caching of pre-computed recommendations, offline batch inference for the long tail of products, real-time inference only for the high-traffic surface areas where personalization value is highest.

In CPG, I worked on a promotion optimization system where the inference cost for real-time personalized promotion selection was prohibitive at the scale of a major retail chain. The solution was a hybrid: batch inference to generate a personalized promotion portfolio for each customer segment (run nightly), with a lightweight real-time model that selected from the portfolio based on current cart state. The batch model could be arbitrarily complex because it ran offline. The real-time model was tiny and cheap because it was selecting from a pre-filtered set.

This separation of batch and real-time computation is one of the most powerful tools for managing the cost-accuracy-latency triangle in high-scale retail scenarios.

How to Choose Your Constraints

The decision about which constraint to optimize first should be made explicitly, before architecture, before model selection, and ideally before the first sprint planning session.

The questions that drive this decision:

  • What is the cost of a wrong answer? High cost (patient harm, financial fraud) means the accuracy constraint is primary. Low cost (suboptimal recommendation) means accuracy is secondary.
  • What is the intervention window? Narrow (seconds to act) means latency is primary. Wide (hours, days) means latency is secondary.
  • What is the volume? High volume (millions of calls per day) makes cost primary. Low volume (hundreds per day) makes cost secondary.
  • What are the regulatory requirements? Some domains have explicit accuracy floors (clinical AI, credit scoring) that cannot be traded away regardless of cost or latency.

Once you've identified your primary constraint, you can make clear architectural decisions. The latency-primary system looks different from the accuracy-primary system looks different from the cost-primary system. These differences cascade through every layer of the stack.

The Mistakes That Come From Not Choosing

The most common failure mode is building an accuracy-optimized system and then discovering that it's too expensive to run at production scale, or too slow for the user-facing feature. This is the most expensive form of architectural rework - you've built the wrong system and now need to redesign it under time pressure.

Second most common: building a latency-optimized system and discovering that accuracy is insufficient for the use case. This happens when the team assumes that good enough accuracy is good enough without actually validating what good enough means for the specific use case and user population.

Both of these failures are preventable with explicit constraint prioritization before you start building. The conversation is: if we had to sacrifice accuracy for cost, at what accuracy floor do we stop? If we had to sacrifice latency for accuracy, what's the maximum latency the user can tolerate? These questions have answers that you can discover with user research. Avoiding them doesn't make them go away - it just defers them to the worst possible moment.

You might also like

Further Reading