RICE for AI: Modified Prioritization Framework

Last quarter, I watched a team spend six weeks building a sophisticated RAG retrieval optimization for their clinical decision support system. The engineering effort was solid - they reduced latency by 200ms and improved relevance scores by 8 points. When we measured actual adoption, the feature moved the needle for maybe 15% of users. Meanwhile, three simpler requests sat in backlog: better error messages when the AI couldn't answer, clearer confidence scores, and a way to surface when the underlying data was stale.

The team had defaulted to optimizing what was technically interesting rather than what would actually move adoption and outcomes. This happens constantly in AI product work, and it's not because people are bad at prioritization - it's because the standard RICE framework (Reach, Impact, Confidence, Effort) doesn't account for the specific dynamics of AI products.

RICE works reasonably well for feature prioritization in traditional software. But AI products operate under different constraints. Your model's hallucination rate might torpedo user trust faster than a delayed feature. Your inference costs could spike unexpectedly when usage patterns change. Your data quality issues might be invisible until they cascade through an entire user workflow. And your ability to measure impact gets genuinely complicated when you're dealing with probabilistic outputs.

I've been refining a modified RICE approach specifically for AI products. It's not a replacement for the original framework - it's an adaptation that accounts for what actually kills or saves AI products.

Where Standard RICE Breaks Down for AI

The four components of RICE are straightforward: Reach (how many people will this affect), Impact (how much will it improve their experience or outcomes), Confidence (how sure are you about those numbers), and Effort (how much work is required). You multiply Reach times Impact, apply a Confidence factor, then divide by Effort. Higher scores rise in your backlog.

On paper, this makes sense. In AI product reality, it creates some specific failure modes:

Impact gets measured without considering model behavior. You estimate that a feature will improve user satisfaction by 2 points on a 10-point scale. But you haven't accounted for what happens when your model fails in specific scenarios. You might get that 2-point improvement for 70% of users and a 5-point decrease for the other 30% who hit edge cases. Standard RICE treats this as a single impact number and misses the distribution entirely.
Reach doesn't account for trust decay. A feature reaches 500 users. But if those 500 users experience the system hallucinating twice, your effective reach drops to 50 users who still engage regularly. The actual Reach number becomes stale within weeks.
Effort estimation ignores inference cost velocity. You estimate a feature will take 3 sprints to build. You don't estimate the compounding cost of adding another 200ms to your inference pipeline across millions of requests per day. By month 6, you're spending 40% more on infrastructure than you budgeted.
Confidence values don't reflect model drift. You're 85% confident that your NLP model will extract clinical entities with 92% accuracy. That confidence was based on your validation set from three months ago. Your actual production accuracy is now 81% because your user population shifted. Your Confidence estimate becomes meaningless.
The framework doesn't separate outcome risk from delivery risk. Building a new AI capability carries two distinct risks: the risk that you won't ship it (delivery risk) and the risk that you ship it but it doesn't produce the outcomes you expected (outcome risk). RICE conflates these into a single Confidence number.

I realized this after shipping a conversational search feature for a payer platform. RICE predicted it would reach 300 users, deliver medium impact (reduce support tickets by 8%), take 4 sprints, and warranted 85% confidence. The math looked good. We built it, shipped it, and adoption flatlined at 60 users. Not because we failed at delivery - the system worked. But the model kept suggesting appeals for denied claims without understanding the specific payer rules, creating liability concerns that users feared. The outcome risk wasn't captured in our prioritization at all.

The Modified RICE Framework for AI Products

Here's what I've been using instead. It keeps the structure of RICE but adds five specific modifications that matter for AI:

Modified RICE = (Reach x Impact x Outcome Confidence) / (Delivery Effort + Inference Cost Burden + Hallucination Risk Tax)

Let me break down what changed:

Split Confidence into Outcome Confidence and Delivery Risk. Keep delivery risk implicit in Effort estimation (because good estimation already accounts for uncertainty). But make Outcome Confidence explicit - this is "how confident are we that the AI will actually behave as we expect in production?" This is separate from "can we ship this?"
Add Inference Cost Burden to the denominator. Calculate not just engineering effort, but the estimated monthly inference cost increase. If a feature adds 50ms to latency across 10M daily requests, that's real cost that should be weighted against benefits. Express it in the same unit as engineering effort (e.g., "equivalent to 2 sprints of AWS spend per quarter").
Add Hallucination Risk Tax to the denominator. For any feature relying on model outputs, estimate the probability-weighted cost of model failures. A hallucination that reaches 5% of users and creates compliance risk might be worth 3 sprint-equivalents of risk tax. A hallucination that affects 0.1% of users and is merely confusing might be worth 0.5.
Adjust Reach for trust decay. Don't just estimate how many people can access the feature. Estimate how many will continue to use it after month 1, month 3, and month 6 based on similar features' retention curves. Use the month-3 number as your Reach, not the theoretical maximum.
Calculate Impact across the outcome distribution, not a single point estimate. Instead of "Impact = 2 points improvement," map it as: "70% of users see 2-point improvement, 25% see neutral, 5% see 2-point degradation." Outcome Confidence is lower when you have a fatter tail of negative outcomes.

This sounds more complex, but it's actually forcing you to think about things you should be thinking about anyway. You're just making the invisible costs visible.

How to Apply Modified RICE: Step by Step

I'll walk through a real example from a recent medtech project where we were prioritizing between three features for an AI-assisted radiology platform.

Feature A: Improved lung nodule detection sensitivity

The radiologists wanted higher sensitivity in detecting small nodules (less than 3mm). Currently the system detects 87% of nodules down to 2mm. Getting to 94% would require fine-tuning on a proprietary dataset and integrating a second ensemble model.

Step 1: Estimate Reach using month-3 retention.

The system currently has 420 active radiologists. We estimate 100% of them would benefit from better detection. But would they all use the feature at month 3? We looked at adoption curves for previous model improvements. Sensitivity improvements typically see 85% sustained engagement at month 3 (people trust better detection and rely on it). So Reach = 357 radiologists.

Step 2: Map Impact across the outcome distribution.

We talked to 15 radiologists. Outcomes ranged widely:

60% said this would eliminate 1-2 cases per month they currently miss, materially improving their workflow (Impact = 3 points on our 5-point scale, measuring workflow improvement + clinical confidence)
25% said they'd catch these through multi-read practice anyway, so marginal benefit (Impact = 0.5)
15% expressed concern that higher sensitivity might increase false positives and require additional callbacks (Impact = -1, net negative because of downstream work)

We weighted this: (0.6 x 3) + (0.25 x 0.5) + (0.15 x -1) = 1.58. Round to 1.6 for our calculation.

Step 3: Establish Outcome Confidence.

We had test data showing 94% sensitivity was achievable. But in production, model performance often drifts. Our historical drift rate for detection models is 2-3 percentage points per year. We were also adding a second model ensemble, which introduces integration risk. We set Outcome Confidence at 0.70 (70% confident we'll achieve the expected outcomes in production).

Step 4: Estimate Delivery Effort.

Fine-tuning: 1.5 sprints. Ensemble integration: 2 sprints. Validation and safety testing: 2 sprints. Total: 5.5 sprints.

Step 5: Quantify Inference Cost Burden.

Adding a second detection model would run on every image. Average cost per inference: $0.003. With 420 radiologists analyzing an average 8 cases per day, that's 3,360 inferences daily, or $10/day, or $300/month. Over a year, that's $3,600, which we estimated as equivalent to 0.7 sprints of AWS spend.

Step 6: Quantify Hallucination Risk Tax.

For detection models, "hallucination" means false positives and false negatives beyond the model's expected rate. We'd be pushing sensitivity higher, which typically increases false positive rate. Our data showed that at 94% sensitivity, false positive rate would be ~6% (vs current 2%). On 3,360 daily inferences, that's 200 additional false positives per day. Not all are costly - some are already-suspected findings. But maybe 10% create unnecessary callbacks or follow-up imaging. We estimated this as a moderate clinical risk. Risk Tax = 2 sprints-equivalent.

Step 7: Calculate the score.

Numerator: 357 (Reach) x 1.6 (Impact) x 0.70 (Outcome Confidence) = 399.8

Denominator: 5.5 (Delivery Effort) + 0.7 (Inference Cost) + 2 (Risk Tax) = 8.2

Score: 399.8 / 8.2 = 48.8

Feature B: Better error messages and confidence score display

When the model was uncertain or encountered an image it couldn't analyze, the system showed a generic error. Radiologists found this frustrating - they wanted to know whether to trust a "uncertain" result or assume the model was broken. We wanted to surface model confidence scores and explain failure modes.

Step 1: Reach.

All 420 active radiologists encounter errors and uncertain cases. Maybe 70% engage with the uncertainty information regularly (the other 30% work mostly on straightforward cases). Month-3 engagement for UI improvements is typically 75%. Reach = 220.

Step 2: Impact distribution.

We asked radiologists about this separately. This was less about clinical improvement and more about reducing friction:

70% said this would make them more confident in their workflow, reducing hesitation (Impact = 1.5 points)
20% said it would help occasionally (Impact = 0.5)
10% said they'd ignore it and just use their judgment anyway (Impact = 0)

Weighted: (0.7 x 1.5) + (0.2 x 0.5) + (0.1 x 0) = 1.15

Step 3: Outcome Confidence.

This wasn't about model behavior - it was about UX. The only outcome risk was "will radiologists actually look at these scores?" Our confidence was higher here: 0.85.

Step 4: Delivery Effort.

This was straightforward feature work. Design: 0.5 sprints. Backend to expose confidence scores: 1 sprint. Frontend: 1 sprint. Testing: 0.5 sprints. Total: 3 sprints.

Step 5: Inference Cost Burden.

Zero. We're just displaying information the model already calculates. Cost Burden = 0.

Step 6: Hallucination Risk Tax.

Near zero. We're not changing model behavior, just displaying it differently. Risk Tax = 0.2 (minimal, just in case the display itself creates confusion in edge cases).

Step 7: Calculate.

Numerator: 220 x 1.15 x 0.85 = 215.3

Denominator: 3 + 0 + 0.2 = 3.2

Score: 215.3 / 3.2 = 67.3

Feature C: Automated quality monitoring for input data

We'd had several incidents where images arriving from certain hospitals had preprocessing issues - DICOM headers were malformed, or compression artifacts corrupted the image. The model would silently process these degraded images and produce lower-quality predictions. We wanted to add automated checks upstream and flag suspicious images.

Step 1: Reach.

All radiologists benefit from this indirectly. But it's not a user-facing feature - it's infrastructure. Reach is measured by "radiologists whose experience improves because of fewer model failures" = 420. Month-3 engagement for reliability improvements: 95% (people love things that just work). Reach = 399.

Step 2: Impact distribution.

This prevents bad outcomes rather than enabling good ones. For most radiologists (80%), this is invisible - they'll never hit the issue. For 15% who work with those problematic hospitals, it prevents 1-2 cases per month where they'd have caught the poor output quality anyway through manual review. For 5%, it prevents cases where they would have missed the quality issue and potentially acted on poor data. Impact: (0.80 x 0.1) + (0.15 x 1.5) + (0.05 x 2.5) = 0.45 (lower overall impact, but concentrated in high-risk scenarios)

Step 3: Outcome Confidence.

High confidence here - we can validate data quality checks against known bad examples. Outcome Confidence = 0.88.

Step 4: Delivery Effort.

Building DICOM validation: 1.5 sprints. Implementing quality checks: 1.5 sprints. Testing against production images: 1 sprint. Monitoring and alerting: 0.5 sprints. Total: 4.5 sprints.

Step 5: Inference Cost Burden.

Quality checks run before inference, so they don't add to inference costs. Burden = 0.

Step 6: Hallucination Risk Tax.

Risk of false positives (flagging good images as bad) and false negatives (missing corrupted ones). We estimated 2% false positive rate would create unnecessary friction. Risk Tax = 1.5 sprints-equivalent.

Step 7: Calculate.

Numerator: 399 x 0.45 x 0.88 = 158.2

Denominator: 4.5 + 0 + 1.5 = 6

Score: 158.2 / 6 = 26.4

The ranking:

Feature B (Better error messages): 67.3
Feature A (Improved detection sensitivity): 48.8
Feature C (Data quality monitoring): 26.4

This ran counter to our initial instinct. Feature A felt more clinically significant. But the modified RICE framework surfaced what we should have noticed earlier: Feature B had lower risk, lower cost, wider value distribution, and simpler outcome validation. We'd ship it, learn fast, and build trust

RICE for AI: Modified Prioritization Framework

RICE for AI: Modified Prioritization Framework