Clinical decision support trust: what the evidence base for CDS alerts requires
Between 49% and 96% of clinical decision support alerts are overridden by physicians, and the primary reason is not alert fatigue. It is data trust. CDS systems built on unscored, unstandardized health data generate alerts that clinicians cannot verify, cannot contextualize, and cannot act on safely.
Physicians override between 49% and 96% of clinical decision support alerts, depending on the institution and alert type. The standard explanation is alert fatigue. The deeper problem is that most CDS alerts are generated from data that has never been scored for trustworthiness, verified for recency, or validated against external sources.
Clinical decision support systems are only as reliable as the data feeding them. When a drug-drug interaction alert fires based on a medication list that has not been reconciled in nine months, or when a sepsis risk flag triggers from lab values with unknown provenance, the alert is not clinically actionable. It is noise. And clinicians treat it accordingly.
What do we mean by clinical?
Before examining what CDS alerts require, it is worth clarifying the word itself. "Clinical" has at least four distinct meanings that matter here.
First, the core medical definition: clinical refers to the direct observation, examination, and treatment of patients. A clinical setting is a hospital, clinic, or any environment where patient care occurs. Clinical data is data generated during or about patient encounters.
Second, when someone is described as "being clinical," it means they approach situations with detached, analytical precision. A surgeon who remains calm during a crisis is clinical. This meaning carries connotations of emotional restraint and systematic thinking.
Third, "very clinical" in common usage describes something coldly efficient. A sports commentator might call a goal "very clinical" to mean it was executed with ruthless precision and no wasted movement. In healthcare, this adjective sometimes describes facilities that feel sterile or impersonal.
Fourth, clinical has an older meaning related to the bedside. The Greek root "klinike" refers to practice at the sickbed. This etymology matters because clinical decision support was designed to bring evidence to the point of care, literally to the bedside. When that evidence is built on untrustworthy data, it fails at the exact moment it was designed to help.
All four meanings converge in CDS: the system must deliver bedside-relevant, analytically precise, evidence-based information during direct patient care. That demands a data foundation most health systems have never built.
Key statistics
The evidence base for CDS alert failure is extensive, and the numbers point consistently to data quality as a root cause.
Why CDS alerts fail: the data trust deficit
Most analyses of CDS alert failure focus on the alert logic, the rules engine, or the user interface. These matter. But they are downstream of the fundamental problem: the data inputs are not trustworthy.
A CDS system that checks for drug-drug interactions relies on an accurate, current medication list. If the list was last updated during a hospitalization eight months ago, any alert it generates is suspect. The clinician who overrides that alert is not being reckless. They are making a rational judgment that the underlying data is unreliable.
Similarly, a CDS alert for renal dosing adjustment depends on a recent creatinine value. If the most recent lab result in the record is from a different institution, transmitted via an HL7 v2 feed with no provenance metadata, the clinician has no way to assess whether that value is current, correctly mapped, or even belongs to the right patient.
This is the clinical decision support data trust problem: alerts generated from unscored data carry no verifiable evidence weight.
The evidence base for CDS data quality requirements
The literature on CDS effectiveness reveals several non-negotiable data quality requirements that most implementations fail to meet.
Recency: CDS alerts must be generated from data that reflects the patient's current clinical state. A 2021 study in JAMIA found that 22% of overridden drug alerts were overridden specifically because the clinician knew the triggering data was outdated. This aligns with what SuperTruth measures through the Recency dimension of the Data Trust Index, which accounts for 15% of the DTI score.
Provenance: Clinicians need to know where a data element came from. A lab value from a CLIA-certified laboratory carries different evidentiary weight than a patient-reported value entered through a portal. Without provenance tracking, the CDS system treats all inputs as equivalent. They are not. Provenance carries the highest weight in the DTI framework at 25%.
Concordance: When data elements from multiple sources contradict each other, the CDS system must either resolve the conflict or flag it. A patient with an active penicillin allergy in one system and no allergy documentation in another creates a CDS environment where any drug alert involving penicillin derivatives is fundamentally ambiguous.
Validation: Has the data element been verified against an external source? A diagnosis code entered during billing may not reflect a confirmed clinical finding. CDS systems that fire alerts based on unvalidated ICD-10 codes are building clinical logic on billing artifacts.
What alert fatigue actually measures
Alert fatigue is real. But it is a symptom, not a cause. What alert fatigue actually measures is the cumulative erosion of clinician trust in a system that cannot distinguish between high-confidence and low-confidence data inputs.
When every alert carries the same visual urgency regardless of the quality of data behind it, clinicians develop override habits. Research from Brigham and Women's Hospital found that physicians who overrode a CDS alert once were 2.5 times more likely to override subsequent alerts of the same type, even when the subsequent alerts were clinically appropriate.
This cascading distrust is a direct consequence of missing data trust infrastructure. If a CDS system could display the confidence level of its underlying data, clinicians could triage alerts differently. An alert backed by a verified, recent, concordant medication list from a primary care provider deserves different attention than an alert triggered by a medication entry from an urgent care visit three years ago.
Recent coverage from MedPage Today highlights how institutional trust failures cascade through clinical systems. When the CDC restructured its vaccine advisory liaison organizations, the stated concern was bias from "special interest groups." Whatever the merits of that decision, it illustrates a broader pattern: when the provenance and independence of advisory inputs are questioned, the entire decision support structure weakens. CDS alerts face the same dynamic at the individual patient level.
The terminology layer: when codes lie to the CDS engine
CDS systems depend on standardized clinical terminologies to match patient data against clinical rules. SNOMED CT codes, LOINC codes, RxNorm identifiers, and ICD-10 classifications form the vocabulary through which CDS engines interpret patient records.
When these codes are mapped incorrectly, the CDS alert evidence base collapses. A common example: a patient's medication is documented using a free-text entry that an NLP engine maps to the wrong RxNorm concept. The CDS system then either fails to detect a real drug interaction or fires a false alert for an interaction that does not exist.
The scale of this problem is significant. Studies of RxNorm mapping accuracy in commercial EHR systems find error rates between 5% and 12% for non-formulary medications. For SNOMED CT, the mapping challenge is worse in specialties where clinical concepts do not map cleanly to available codes.
MedPage Today's 2022 reporting on outdated medical language illustrates a related problem: the words clinicians use shape how data is recorded, which shapes how CDS engines interpret it. Phrases that are imprecise or stigmatizing do not just affect patients. They create data quality problems that propagate through every downstream system.
What a trust-scored CDS pipeline looks like
A CDS system that incorporates data trust scoring would work fundamentally differently from current implementations.
First, every data element feeding a CDS rule would carry a trust score. A medication list entry with a DTI score of 92 (recent, validated, concordant across sources, clear provenance) would generate alerts that display differently from an entry with a DTI score of 34 (old, unvalidated, single-source, no provenance metadata).
Second, alert thresholds would adjust based on aggregate data trust. If the patient record backing a clinical rule has an overall DTI score below a configurable floor, the system would flag the data gap rather than generating a potentially misleading clinical alert. This is similar to how SuperTruth's DTI Engine enforces minimum trust thresholds before data reaches AI models.
Third, the CDS system would surface provenance information alongside the alert. Instead of "Potential interaction between Drug A and Drug B," the alert would read "Potential interaction between Drug A (prescribed by Dr. Smith, confirmed in pharmacy claims 3 days ago) and Drug B (documented in ER visit, March 2023, not confirmed in recent fill data)."
This provenance transparency transforms the alert from a binary interruption into a clinical communication that the physician can evaluate.
The regulatory trajectory
FDA guidance on clinical decision support is tightening. The 2022 final guidance on Clinical Decision Support Software narrowed the exemptions under Section 520(o) of the FD&C Act, making more CDS systems subject to regulatory oversight. The key question the FDA asks: does the clinician have the ability to independently review the basis for the recommendation?
If the basis for the recommendation is data with unknown provenance, unverified accuracy, and uncertain recency, the clinician cannot meaningfully perform independent review. This means unscored data in CDS systems creates not just a clinical risk but a regulatory risk.
CMS quality programs are moving in the same direction. The Promoting Interoperability program requires CDS use for eligible clinicians, but the quality measures increasingly demand documentation that CDS alerts are clinically appropriate. Systems with high override rates face scrutiny. The path forward requires demonstrating that CDS alerts are backed by trustworthy data, not just that alerts were displayed.
What claims data lag means for CDS
Many CDS systems incorporate claims data to build longitudinal medication histories, identify care gaps, and flag potential duplicate therapies. But claims data arrives 30 to 90 days after the clinical event it describes. A CDS alert based on a claim for a medication fill that happened two months ago may be firing against a medication the patient has already discontinued.
This temporal disconnect is one of the least discussed contributors to CDS alert failure. The alert is not wrong in terms of logic. The interaction rule is valid. But the data driving it is stale, and the clinician standing in front of the patient knows this even when the system does not.
Trust scoring for recency solves this by flagging data elements that fall below a configurable freshness threshold. An alert generated from a claims-derived medication entry older than 60 days could be tagged accordingly, changing how the clinician weighs it.
The patient-generated data challenge
Patient-reported data adds another layer of complexity to CDS trust. Medication lists that patients enter through portals, symptoms logged in apps, and home blood pressure readings all feed CDS logic in some systems. But patient-generated health data has different trust characteristics than clinician-documented data.
This is not a reason to exclude PGHD from CDS. It is a reason to score it differently. A blood pressure reading from a validated, FDA-cleared home monitor with consistent readings over two weeks carries different evidentiary weight than a single patient-reported value. The DTI framework's Quality and Validation dimensions capture exactly this distinction.
Building the evidence base forward
The CDS alert evidence base needs to expand beyond alert override rates as a primary outcome measure. Three additional metrics would better capture CDS effectiveness:
These metrics require infrastructure that most health systems do not have. Building that infrastructure is not optional if CDS is going to fulfill its original promise: bringing trustworthy evidence to the bedside.
The DTI Engine scores every health data record 0-100 across 8 trust dimensions before your AI model sees it. CDS systems built on unscored data will continue to generate alerts that clinicians override, regulators question, and patients never benefit from. If your team is building or evaluating clinical decision support infrastructure and needs to quantify the trustworthiness of the data feeding it, contact Louis Simeonidis at louis@supertruth.ai or (215) 918-4140.
Further reading:
Jason Alan Snyder
Co-founder of SuperTruth and Artists & Robots, and an inventor on the Data Trust Index patents. Twenty-plus years building technology inside Interpublic Group. He writes here nearly every day on data trust, provenance, and what AI should be allowed to act on, and publishes essays on his Substack.
About SuperTruth · LinkedIn · Substack · jasonalansnyder.com
See it in practice
DTI scores the record, not the patient.
8 dimensions. 0–100. Travels with every record permanently.