Glass Box vs Black Box: why health AI needs explainable data provenance
Most explainable AI frameworks explain model decisions but ignore whether the training data was trustworthy in the first place. Glass box AI in healthcare requires more than interpretable algorithms. It requires explainable data provenance, where every record carries a verified chain of custody before a model ever touches it.
A glass box you cannot see through is still a black box. That is the core problem with explainable AI in healthcare today. Most XAI frameworks focus on making model decisions interpretable. They show you which features the algorithm weighted, which variables mattered, and how a prediction was reached. None of that matters if the data feeding the model was wrong, stale, unconsented, or unverifiable.
What is the difference between black box AI and glass box AI?
A black box AI model produces outputs without revealing its internal reasoning. You feed it patient data, it returns a risk score, and no one can explain why. Deep learning models, including the neural networks behind many clinical AI tools, are the most common black box architectures. They process millions of parameters in ways that resist human interpretation.
A glass box AI model exposes its reasoning. Decision trees, logistic regression, and rule-based systems are classic examples. Clinicians can trace the path from input to output. When a glass box model flags a patient as high-risk for sepsis, the reasoning is visible: elevated white blood cell count, rising lactate, two missed vital sign windows.
The distinction matters for clinical adoption. A 2023 survey published in Nature Medicine found that 72% of physicians said they would not act on an AI recommendation they could not interpret. Glass box models solve that problem at the algorithm layer. But they leave a deeper problem untouched.
What is the black box problem in AI healthcare?
The black box problem is not limited to model architecture. It extends to the data itself.
Consider a sepsis prediction model trained on EHR data from three hospital systems. The model performs well in validation. It ships to production. Six months later, it starts generating false positives at an alarming rate. The investigation reveals that one of the training datasets contained duplicate records from a billing system migration, another included vitals recorded in inconsistent units, and the third lacked consent documentation for secondary use.
The model was a glass box. Every decision was interpretable. But the data underneath was opaque. No one could answer basic questions: Where did this record originate? Was it consented for AI training? When was it last validated? Has it been modified since collection?
This is the real black box in healthcare AI. Not the model. The data.
Why transparency matters beyond the algorithm
Regulators are catching up to this reality. The FDA's draft guidance on AI/ML-based Software as a Medical Device explicitly calls for documentation of training data characteristics, including provenance and representativeness. The ONC's information blocking rules create obligations around data access that downstream AI systems must respect. CMS quality programs increasingly require audit trails that trace clinical decisions back to source data.
Transparency at the model layer without transparency at the data layer creates a dangerous illusion. A hospital system can deploy an interpretable clinical decision support tool and still face regulatory exposure if the training data cannot pass an audit. As we explored in AI explainability solves the wrong problem, trusting a model output means nothing if the training data was never verified.
What explainable data provenance actually requires
Explainable data provenance means every record used in a health AI system carries verifiable metadata across multiple trust dimensions. Not just "where did this data come from" but a complete chain of custody: who collected it, under what consent, when it was last validated, whether it agrees with other sources, and whether it remains current.
SuperTruth's Data Trust Index scores every health data record from 0 to 100 across eight dimensions. Provenance carries the highest weight at 25%, followed by Consent at 20% and Recency at 15%. These three dimensions alone account for 60% of a record's trust score because they represent the most common failure points in health data supply chains.
When we processed 105,000 diagnostic records for imaware, the provenance dimension exposed issues that no model-level explainability tool would have caught. Records that looked clean on the surface carried broken chain-of-custody links, inconsistent lab reference ranges, and undocumented transformations from upstream systems. The DTI Engine flagged these before any model trained on them.
Key statistics
The gap between model transparency and data transparency is measurable.
The glass box data standard
A true glass box approach to health AI requires two layers of transparency working together. The model layer explains how decisions are made. The data layer explains why the inputs should be trusted.
Without both, you get one of two failure modes. Black box models on trusted data produce unexplainable but potentially reliable outputs. Glass box models on unverified data produce explainable but potentially dangerous outputs. Neither is acceptable for clinical use.
The path forward is not choosing between model explainability and data provenance. It is enforcing both. Every record gets scored. Every score gets documented. Every model inherits the trust profile of its training data. This is what we mean by the chain of custody problem in health data and why provenance remains the hardest dimension to get right.
Health systems deploying AI need to answer a simple question before any model goes live: can you show an auditor the provenance score of every record your model trained on? If the answer is no, your glass box has a black box hiding inside it.
The DTI Engine scores every health data record 0 to 100 across 8 trust dimensions before your AI model sees it. If your team is evaluating data for training, compliance, or clinical use, contact Louis Simeonidis at louis@supertruth.ai or (215) 918-4140.
Further reading:
Jason Alan Snyder
Co-founder of SuperTruth and Artists & Robots, and an inventor on the Data Trust Index patents. Twenty-plus years building technology inside Interpublic Group. He writes here nearly every day on data trust, provenance, and what AI should be allowed to act on, and publishes essays on his Substack.
About SuperTruth · LinkedIn · Substack · jasonalansnyder.com
See it in practice
DTI scores the record, not the patient.
8 dimensions. 0–100. Travels with every record permanently.