Scaling Gen-AI Clinical Trial Matching Across 10+ Therapeutic Areas

The Problem Nobody Wanted to Own

When I stepped into the clinical trial matching initiative, the program had already been running for 18 months and had almost nothing to show for it. The research operations team was frustrated. The sponsors were threatening to reallocate budget. And the clinical research coordinators - the people who actually did the work - were still manually reviewing patient charts one by one, spending upward of 20 minutes per chart, across a patient population spanning six therapeutic areas.

The core problem: 80% of clinical trials fail to meet enrollment targets on time. That is not a new statistic. Tufts Center for the Study of Drug Development has been publishing variations of it for years. What organizations consistently underestimate is how much of that failure is not a patient availability problem - it is a patient identification problem. The patients exist. They are in the system. Nobody can find them fast enough, or accurately enough, to make a difference.

Our site network had over 2.4 million patient records. We were running protocols across oncology, rare disease, neurology, cardiology, metabolic disease, immunology, and four additional therapeutic areas. Each trial had eligibility criteria that could run to 40 or 50 criteria items, many of them written in language like: "Patient must have received at least one prior line of platinum-based chemotherapy but no more than three, and must not have received an immune checkpoint inhibitor for this indication within the preceding 12 months."

No structured database query handles that. Not cleanly. Not at scale.

Why the First Approach Failed

The team before me had tried to solve this with rule-based queries against structured EHR fields - ICD-10 codes, CPT codes, medication tables. The logic was sound in theory. In practice, it missed approximately 60% of potentially eligible patients because clinically relevant information - ECOG performance status, biomarker expression levels, prior treatment response, comorbidities - lives almost exclusively in physician notes, pathology reports, and discharge summaries. Unstructured text.

The structured query approach also produced high false-positive rates. Coordinators were being routed to charts where the patient had an ICD code for a condition that was historical, resolved, or explicitly contraindicated something in the trial criteria. Chart review time went up, not down, because coordinators had to audit every result to filter noise.

I brought in a red team of three experienced CRCs to audit the legacy output over two weeks. Their conclusion: the system was generating more work than it was saving. This is the classic failure mode for clinical trial matching AI - optimizing the wrong metric (query coverage) instead of the right one (coordinator time saved per confirmed-eligible patient).

Technical Approach: What We Actually Built

We rebuilt the platform on four interconnected components. I will describe each in terms of the problem it solves, not just the technology.

1. Clinical NLP and Named Entity Recognition on Unstructured Notes

The foundation was a clinical NLP pipeline purpose-built for Life Sciences & Healthcare AI applications. We evaluated three approaches: a general-purpose LLM applied zero-shot, a fine-tuned ClinicalBERT variant, and a hybrid using spaCy with a domain-specific NER model trained on de-identified oncology notes from our own network.

The hybrid won. Zero-shot LLMs handled ambiguous language well but were inconsistent on dense clinical text with abbreviations. ClinicalBERT required significant fine-tuning per therapeutic area and was slow to update. The spaCy hybrid gave us consistent extraction across the entity types that actually mattered for eligibility matching: disease stage, biomarker status (PD-L1, EGFR, HER2, KRAS), performance status scores (ECOG, Karnofsky), prior treatment lines, response categories, and documented comorbidities.

We indexed every unstructured note across our patient population into a searchable knowledge graph. That graph became the substrate for everything downstream.

2. LLM-Based Eligibility Criteria Parsing

Eligibility criteria documents are some of the most syntactically complex documents in medicine. They contain nested Boolean logic, temporal constraints, unit-based thresholds, and conditions that reference prior conditions. A human reads them fine. A rule engine chokes on them.

We used a fine-tuned LLM to decompose each eligibility criterion into a structured representation: entity type, operator, value, temporal window, and exclusion flag. The model parsed criteria like the platinum-chemotherapy example above into machine-executable logic that could be run against the NLP-extracted patient knowledge graph.

This was not perfect out of the box. The model struggled initially with negation-heavy criteria and with criteria that referenced lab values with non-standard units. We ran three iterations of human-in-the-loop correction with our clinical informatics team before deploying to production. The final parser achieved 91% agreement with expert manual interpretation across a 200-criteria test set spanning eight therapeutic areas.

3. RAG-Based Patient-Trial Matching

The matching layer used a Retrieval-Augmented Generation architecture. For each open trial, the system retrieved the parsed eligibility criteria and, for each candidate patient, retrieved the relevant sections of that patient's clinical record. Both were passed to an LLM with a structured prompt asking it to: assess preliminary eligibility, cite the specific evidence from the record supporting or contradicting each criterion, and flag criteria where the record was silent or ambiguous.

This approach - covered in more depth in the RAG vs fine-tuning breakdown - was critical for two reasons. First, it grounded the model's output in actual patient documentation, which reduced hallucination risk and made the output auditable. Second, the explicit evidence citations gave coordinators something actionable: they could go directly to the relevant note section rather than reviewing the full chart. Review time dropped from 20+ minutes per chart to under 4 minutes for a coordinator validation pass.

4. Scoring, Ranking, and Coordinator Workflow

Raw eligibility assessment is not enough. With 2.4 million patients and dozens of open trials, the coordinator queue still needed prioritization. We built a scoring model that weighted matches by: confidence of eligibility assessment, recency of supporting clinical evidence, and proximity to enrollment timeline pressure (trials at risk of missing quarterly targets were surfaced first).

The coordinator interface showed a ranked list of patients per trial, each with a match confidence score, a summary of evidence, and direct links to the source note sections. The interface was designed with one constraint: the AI is a screening support tool, not a decision maker. No patient was flagged as eligible without a coordinator sign-off. The system supported the human. It did not replace the human.

For evaluation methodology on this kind of LLM-assisted workflow, the LLM evaluation framework I use across products covers how we measured precision, recall, and coordinator acceptance rates as the primary production metrics - not model-side accuracy alone.

Architecture and Infrastructure

A few architecture decisions that were non-obvious and worth flagging for anyone building in this space.

Data ingestion was harder than modeling. We had site networks running Epic, Cerner, and two legacy EHR systems with no FHIR-compliant APIs. The HL7 FHIR integration path worked cleanly for Epic. For everything else, we built a data normalization layer that consumed HL7v2 ADT feeds, applied a mapping schema to a common data model (we used OMOP CDM), and ran the NLP pipeline on the normalized output. This normalization infrastructure took four months and represented roughly 40% of total engineering investment. Any product manager scoping a clinical NLP project who underestimates this phase is going to miss their milestones.

21 CFR Part 11 compliance was an architecture constraint, not a post-launch checklist item. Every query, every match assessment, every coordinator action was logged to an immutable audit trail with timestamps, user identity, system version, and the specific model checkpoint used to generate each output. When the NLP model was updated - which happened four times in 18 months - every prior output remained traceable to the model version that generated it. Sponsors asked for these logs. Two sponsors made audit trail access a contractual requirement. Build the audit infrastructure first, not after.

Multi-site coordination required explicit data governance. Across an 11-site network, different IRB configurations meant different patient data access permissions. We built a permissions layer that enforced site-level data boundaries at query time, so coordinators at site A could not inadvertently surface patients whose records were governed by site B's IRB protocol. This sounds obvious. It was not obvious to anyone until we built the cross-site matching capability and realized the governance gap.

Results

Across 18 months of production operation spanning 10+ therapeutic areas and 34 open protocols:

Matching time from protocol activation to first eligible cohort: reduced from 14.2 days to 3.8 days. This was the metric sponsors cared most about. Faster identification meant faster site activation and faster enrollment start dates.
Trial enrollment increased by 34% across the active protocol portfolio. This was measured against the 12-month pre-implementation baseline for the same sites and therapeutic areas.
Screen failure rate dropped from 41% to 23%. By surfacing patients with richer multi-criteria pre-screening rather than structured-query broadness, the candidates reaching coordinator review were already more tightly qualified.
Coordinator chart review time per confirmed-eligible patient: reduced from 20+ minutes to 3.8 minutes. This freed coordinator capacity to manage more trials simultaneously - average coordinator trial load increased from 4.2 to 6.8 active protocols.
Patient identification NLP recall: 88% against expert manual review on a sampled audit. We missed roughly 12% of eligible patients - primarily cases where the relevant clinical information existed only in scanned PDFs that were not yet part of our OCR pipeline. That pipeline was added in month 14.

Three sponsors moved additional trial volume to our site network directly citing the matching platform's performance data. One large biopharma partner ran a formal feasibility analysis - "how quickly can you identify eligible patients for this protocol in your network?" - and used the result to inform their site selection for a Phase III program. That is a different category of value than operational efficiency: it becomes a competitive differentiator in sponsor relationships.

What Broke and What I Would Do Differently

This section is the one I would have wanted to read before starting.

The rare disease therapeutic area was almost a separate project. Matching algorithms tuned on oncology data transfer poorly to rare disease. Patient populations are smaller by definition, terminology is less standardized, and the eligibility criteria tend to rely on specialist-level nuance that our NLP pipeline was not trained to handle. We ended up building a separate fine-tuned model for each of the four rare disease protocols we ran. The oncology-trained NLP produced a 58% recall rate on rare disease charts. The condition-specific models got to 84%. Lesson: NLP clinical trial matching AI does not generalize across therapeutic areas as well as vendors claim.

The coordinator acceptance problem was cultural, not technical. Early versions of the platform were technically sound but coordinator adoption was below 40% at month three. The interface surfaced too much information and the confidence scores were misinterpreted as hard eligibility determinations. We ran structured interviews with 12 coordinators across four sites, redesigned the interface around the coordinator's actual workflow rather than our model's output structure, and ran a three-week training and feedback cycle. Adoption went from 38% to 79% over the following two months. The best AI screening workflow is worthless if the humans it is designed to support do not trust it or use it.

Explainability was a sponsor requirement I did not anticipate. Two sponsors asked for documentation of how the matching system worked, specifically requesting that we demonstrate the AI's output could be independently verified against the source clinical record. They were preparing their own regulatory packages and needed to characterize any AI tool that touched their trial data. We had the audit trails, but we did not have a clear documentation package. We built one retroactively. It should have been part of the initial compliance deliverable.

Model versioning and rollback should have been designed from day one. When we updated the NLP model in month 8, retrospective validation showed that precision had improved but recall had dropped slightly for a specific biomarker class. We needed to roll back for three protocols while the issue was corrected. Without an explicit model registry and rollback protocol, that would have been a production incident. We were lucky we had the infrastructure. It was not planned - it was built reactively after a near-miss in month 5.

The Broader Pattern for NLP Patient Screening

I have seen this class of problem across multiple programs now - clinical trial matching AI, prior authorization NLP, AI enrollment optimization for observational registries. A few patterns hold across all of them.

The data pipeline is always the bottleneck, not the model. A production-grade clinical NLP system spends more engineering time on ingestion, normalization, and data quality than on model development. Organizations that scope these projects as "an AI project" rather than "a data infrastructure project that also involves AI" consistently underestimate effort and timeline.

Human-in-the-loop is not a risk mitigation strategy - it is a quality input. Coordinators and clinical experts who review AI outputs provide feedback that is essential for continuous model improvement. Systems that treat human review as a compliance checkbox rather than a signal source degrade over time instead of improving.

The regulatory surface is larger than it looks at first glance. 21 CFR Part 11 is the visible requirement. Underneath it are IRB governance, site data access contracts, de-identification standards, model validation documentation requirements, and sponsor audit rights that collectively define the compliance perimeter. Engage your regulatory affairs team in the architecture phase, not the launch phase.

And the metric that actually matters for the business is not model accuracy. It is time-to-first-eligible-cohort and per-trial enrollment rate. Those are the numbers sponsors track. Those are the numbers that drive network value. Everything else - recall rates, precision, NER F1 scores - is infrastructure that supports those outcomes. Keep that hierarchy clear when you are reporting to leadership and when you are making trade-offs about where to invest engineering cycles.

Clinical trial matching is a solved problem in principle. The hard part is building the operational system around the model that makes it work reliably, compliantly, and at scale across a heterogeneous site network where almost nothing is standardized. That is a product problem more than an AI problem. And it is one worth getting right.