Explainable data provenance: why health AI has to show where every record came from

A sepsis prediction model flags a patient for early intervention. The clinician asks why. The model vendor points to feature importance scores, SHAP values, a saliency map. The clinician nods. Nobody asks where the lab value that triggered the alert actually came from, whether it was transcribed correctly, whether the patient consented to its use in predictive analytics, or whether it arrived 72 hours late from a reference lab that batches results weekly.

This is the state of explainable AI in healthcare. We explain the model. We never explain the data.

What explainability actually means in health AI

Most explainability frameworks focus on model behavior. They answer: given these inputs, why did the model produce this output? Tools like LIME, SHAP, and attention visualization have become standard in machine learning research. Regulatory bodies including the FDA have signaled that algorithmic transparency will be part of future device approvals.

But model explainability is only half the question. The other half, the one that matters more in healthcare, is data explainability. Can you trace every record that entered the model back to its source system, its original author, its timestamp of creation, its method of capture, and its chain of custody through every transformation?

If the answer is no, then your explainability is performative. You are explaining how a model processed data you cannot verify. As we wrote previously, trusting a model output means nothing if the training data was never verified.

The provenance gap in clinical AI

Provenance in health data means knowing where a record was created, by whom, under what circumstances, and what happened to it between creation and use. This sounds straightforward. It is not.

A single patient record in a health system may include data from an EHR, a claims adjudication system, a pharmacy benefit manager, a remote monitoring device, a patient portal entry, and a faxed referral letter that was scanned and OCR-processed. Each of these sources has a different reliability profile. Each introduces different error types. And in most health AI pipelines, these sources are flattened into a single feature vector with no trace of origin.

The result: a model trained on data from six different source systems treats all inputs as equally trustworthy. A blood pressure reading captured by a calibrated clinical device during a nurse visit gets the same weight as one self-reported through a patient portal three weeks later. A diagnosis code entered by a board-certified specialist carries the same signal as one upcoded by a billing optimization tool.

This is not a theoretical problem. Research published in JAMIA has documented that up to 30% of diagnosis codes in claims data do not match clinical documentation. The chain of custody problem in health data is the hardest dimension to solve because it requires tracking data through systems that were never designed to be tracked.

Key statistics

DTI dimension weights: provenance leads at 25%

Provenance carries a 25% weight in the DTI framework, the single largest dimension of the eight scored. This reflects both its difficulty and its importance.

Up to 30% of diagnosis codes in claims data diverge from clinical documentation, according to studies of ICD-10 coding accuracy in administrative datasets.

SuperTruth's work with imaware standardized 105,000 diagnostic records, reducing processing time from 3 weeks to 2 hours, a 95% reduction. Provenance tracking was a prerequisite for that standardization.

The FDA's 2023 draft guidance on AI/ML-based software as a medical device explicitly names data provenance as a requirement for premarket submissions.

Health data fragmentation costs the U.S. system an estimated $3.5 trillion annually, and the inability to trace records across systems is a primary driver of that waste.

Why model-level explainability is insufficient

Consider a readmission prediction model deployed at a health system. The model flags a 68-year-old patient as high risk for 30-day readmission. The explainability layer reports that the top contributing features were: three or more ED visits in the prior year, HbA1c above 9.0, and a social determinant flag for housing instability.

A clinician reviewing this output might trust it. But here is what model-level explainability does not reveal.

The ED visit count of three includes one visit at a different health system that was imported via a CCD document with known quality issues. The CCD document quality problem means that imported visit data frequently contains duplications, misattributions, and date errors. Was that third ED visit real?

The HbA1c value was captured 11 months ago. The model treats it as current because no newer value exists in the record. But an 11-month-old HbA1c in a diabetic patient may not reflect current glycemic control at all. Recency is the most underrated dimension in health AI data scoring.

The housing instability flag was inferred from an address-level geocoding match to a census tract with high eviction rates. The patient actually owns their home. The census tract mismatch problem created a false positive that the model amplified.

None of these issues are visible in a SHAP plot. They are only visible when you can trace each record back to its source.

What explainable data provenance looks like

imaware diagnostic record processing: before and after DTI

Explainable data provenance requires four capabilities that most health AI pipelines lack.

First, source attribution. Every data element must carry metadata identifying the originating system, the method of capture (clinician entry, device upload, patient self-report, NLP extraction, claims adjudication), and the timestamp of original creation. This is not the same as the timestamp of ingestion into the analytics platform.

Second, transformation logging. Every step that modifies a record, whether normalization, deduplication, imputation, or mapping to a standard terminology, must be logged with the logic applied and the before-and-after states preserved. When an ICD-10 code is mapped from one version to another, or a LOINC code is inferred from a free-text lab name, that mapping must be auditable.

Third, trust scoring at the record level. Not all records are equal. A record with verified provenance, recent capture, validated coding, and explicit consent should carry a different weight in model training and inference than a record with unknown origin, stale data, inferred codes, and no consent documentation. The DTI framework scores records 0 to 100 across eight dimensions: Provenance (25%), Consent (20%), Recency (15%), Quality (10%), Concordance (10%), Validation (10%), Breadth (5%), and Stability (5%).

Fourth, query-time lineage. When a model produces an output, it should be possible to trace backward from the prediction to the specific records that influenced it, and from each record to its source system and chain of custody. This is not a reporting feature. It is a clinical safety requirement.

The regulatory case for provenance

Regulatory pressure is building from multiple directions.

The FDA's draft guidance on AI/ML-based Software as a Medical Device (SaMD) includes data management practices as a core element of the predetermined change control plan. The agency expects manufacturers to document the provenance of training data and to demonstrate that data quality was assessed before use. We covered the specifics in how the FDA will audit your health AI's training data.

CMS requires data integrity documentation for value-based care programs including ACO REACH, MSSP, and the new ACCESS program. Health systems participating in these programs must demonstrate that the data underlying quality measures and risk adjustment is accurate, complete, and traceable. The $420 per beneficiary payment in ACCESS comes with data requirements that most organizations are not prepared to meet.

NCQA's HEDIS measure specifications include data validation requirements that implicitly demand provenance tracking. If you cannot demonstrate where a quality measure numerator event originated, the measure is not valid.

The EU AI Act, which will affect any health AI vendor operating in European markets, classifies medical AI as high-risk and requires documentation of training data provenance, quality criteria, and bias assessment. This is not aspirational. It takes effect in 2025-2026.

What happens when provenance is missing

The consequences of missing provenance are not abstract. They show up in clinical workflows, regulatory audits, and patient outcomes.

In 2023, UnitedHealth Group faced public scrutiny over AI-driven claim denials. The controversy centered on whether the model's training data accurately represented the clinical scenarios it was adjudicating. Without provenance documentation, it was impossible to determine whether the training data included the types of cases the model was denying.

In clinical decision support, a model that recommends a medication based on a diagnosis code it cannot trace to a verified clinical encounter is making a recommendation on unverified ground. If that diagnosis code was entered for billing optimization rather than clinical documentation, the recommendation may be clinically inappropriate.

In research, real-world evidence submissions to the FDA require documentation of data sources, collection methods, and quality controls. RWE studies built on claims data with no provenance tracking face higher rejection rates and longer review cycles.

The difference between provenance and audit trails

Audit trails record who accessed a record and when. Provenance records where a record came from and what happened to it. These are related but distinct concepts.

An EHR audit trail will show that Dr. Smith opened a patient chart at 2:14 PM on March 3. It will not show that the allergy list in that chart was imported from a CCD document sent by an outside facility, that the CCD was generated by an HL7 v2 to FHIR conversion pipeline, and that the original allergy entry was made by a medical assistant at a community health center using a dropdown menu that mapped "sulfa" to the wrong RxNorm concept.

Provenance goes deeper than access logging. It traces the data itself, not the users who viewed it. Audit trails are the foundation of health AI accountability, but they are not sufficient without provenance.

How DTI scores provenance

The DTI framework assigns provenance the highest weight of any dimension at 25%. This is deliberate. Provenance is the dimension that all other dimensions depend on.

You cannot assess recency without knowing when a record was originally created, not just when it was ingested. You cannot assess consent without knowing who created the record and under what data use agreement. You cannot assess concordance without knowing which source systems contributed conflicting values.

DTI provenance scoring evaluates several factors. Source system identification: is the originating system known and documented? Capture method: was the data entered by a clinician, generated by a device, extracted by NLP, or inferred by an algorithm? Transformation chain: how many processing steps occurred between creation and current state, and are they logged? Authority: does the source have clinical authority for the data type in question? A pharmacy system has authority for medication dispensing data. It does not have authority for diagnosis data.

A record with verified provenance across all factors scores high on the provenance dimension. A record with unknown origin, undocumented transformations, or questionable source authority scores low. The score is not pass/fail. It is a continuous value that feeds into the overall DTI score and can be used to set minimum thresholds for different use cases.

Why glass-box data matters more than glass-box models

The industry conversation about explainable AI healthcare has been dominated by model transparency. Making neural networks interpretable. Publishing model cards. Providing feature importance rankings.

These are necessary but insufficient steps. A perfectly interpretable model trained on data with broken provenance will produce perfectly interpretable wrong answers. The model will tell you exactly why it made the wrong prediction. It just cannot tell you that the data feeding the prediction was unreliable.

The shift required is from glass-box models to glass-box data. Every record should be inspectable. Every transformation should be visible. Every source should be named. Every chain of custody should be traceable. This is what health AI transparency provenance actually means.

The path forward

Explainable data provenance is not a feature request. It is a prerequisite for trustworthy health AI. The organizations that build provenance tracking into their data pipelines now will be the ones that survive regulatory scrutiny, earn clinician trust, and produce models that actually work in production.

The ones that treat provenance as an afterthought will discover that no amount of model explainability can compensate for data they cannot trace.

The DTI Engine scores every record 0 to 100 across eight dimensions before your AI model sees it. Provenance carries the highest weight because it is the dimension everything else depends on. If your team is building clinical AI and cannot answer the question "where did this record come from?" for every input, talk to the SuperTruth commercial team. Schedule a conversation or call (215) 918-4140.

Further reading:

DTI™ Engine

Health systems solution

Glass box vs black box: why health AI needs explainable data provenance

The chain of custody problem in health data: why provenance is the hardest dimension

AI explainability solves the wrong problem: why trusting a model output means nothing if the training data was never verified