AI explainability solves the wrong problem: why trusting a model output means nothing if the training data was never verified
Explainability techniques tell you why a model made a decision, but they cannot tell you whether the data behind that decision was accurate, consented, or even real. The AI industry has spent billions making outputs interpretable while ignoring the fact that 91% of health data quality issues originate before a model ever trains. Trusting a model's reasoning is meaningless if nobody verified the inputs.
Explainability is the wrong finish line. The AI industry treats interpretability as the primary mechanism for building trust. Regulators ask for it. Procurement teams require it. Researchers publish papers about it. But explainability answers a question that assumes the hard part is already done: that the data feeding the model was accurate, timely, properly consented, and traceable to its source.
That assumption is wrong.
The explainability trap
Explainability tools like SHAP values, attention maps, and counterfactual explanations tell you which features influenced a prediction. They show you the model's reasoning path. They do not, and cannot, tell you whether the input features were correct.
A sepsis prediction model can explain that it weighted lactate levels and heart rate variability most heavily. That explanation looks rigorous. But if the lactate value was transcribed incorrectly during an EHR migration, or if the heart rate data came from an uncalibrated wearable device with no provenance record, the explanation is a detailed story about a wrong answer.
This is the core problem with AI explainability: it validates the logic of a system while ignoring the integrity of its foundation.
Why shouldn't you fully trust the output given by AI tools?
AI outputs are probabilistic. Every prediction carries uncertainty. But the deeper reason not to trust outputs blindly has nothing to do with probability. It has to do with the data supply chain.
A 2021 MIT study (Northcutt et al.) found that 3.4% of labels in commonly used benchmark datasets were incorrect — with errors concentrated in high-stakes categories. In healthcare, the problem is far worse. Duplicate records, stale demographics, missing consent flags, and undocumented data transformations are standard across EHR systems. When an AI model trains on data with these defects, no amount of post-hoc explanation repairs the damage.
You should not fully trust AI output because the pipeline that produced it almost certainly includes data that was never independently verified for accuracy, recency, or consent status.
What is the problem with AI explainability?
The problem is scope. Explainability operates at the model layer. It answers: given these inputs, why did the model produce this output? It does not answer: should these inputs have been used at all?
Consider a clinical decision support tool that recommends dosage adjustments. An explainability module can show that the recommendation was driven by the patient's recorded weight, renal function, and medication history. But if the weight was recorded 14 months ago, the renal function value was copied from a different patient during a system merge, or the medication history is missing a drug prescribed by an out-of-network provider, the explanation is technically correct and clinically dangerous.
Explainability without data verification is like auditing a building's architecture while ignoring the soil report.
How does approach to explainability reinforce trust in AI decision-making?
Explainability does reinforce trust, but only when it sits on top of a verified data foundation. The sequence matters. First, verify the data. Score it for provenance, consent, recency, quality, concordance, validation, breadth, and stability. Then, and only then, does explaining the model's reasoning add value.
When a model can say "I used these features, they were sourced from these systems, last validated on this date, with consent status confirmed," that is genuine trust. When a model can only say "I used these features and here is how I weighted them," that is theater.
The Data Trust Index provides this pre-model verification layer. Every record scored 0 to 100 across 8 dimensions before any model touches it.
What is the 30% rule in AI?
The 30% rule is an informal industry guideline suggesting that at least 30% of an AI system's development effort should go toward data preparation, cleaning, and validation rather than model architecture or tuning. In practice, most organizations fall far short. A 2024 Gartner survey found that data engineering teams spend the majority of their time on integration and formatting, not on trust verification.
The 30% rule acknowledges something the explainability community often ignores: the model is not the product. The data is the product. And if you do not invest in verifying the data, no amount of model sophistication compensates.
In healthcare, 30% is arguably too low. When patient outcomes depend on model accuracy, data that has never been scored for trust is a liability that compounds with every inference.
Key statistics
The order of operations matters
The AI industry has the sequence backwards. The current approach: build model, explain model, hope the data was good enough. The correct approach: verify data, score data, enforce trust thresholds, then build and explain.
This is not a theoretical distinction. The FDA's emerging guidance on AI in healthcare explicitly asks about training data provenance. Regulatory auditors will want to see not just what a model decided, but what data it trained on, where that data came from, whether patients consented to its use, and when it was last validated.
Explainability without data trust is a compliance gap waiting to surface during the worst possible audit.
Trust starts before the model
The next time a vendor tells you their model is explainable, ask them a different question: is your training data verified? Can you show me provenance scores? Consent status? Recency? If they cannot answer those questions, their explainability is decorative.
The DTI Engine scores every health data record 0 to 100 across 8 trust dimensions before your AI model sees it. If your team is evaluating data for training, compliance, or clinical use, contact Louis Simeonidis at louis@supertruth.ai or (215) 918-4140.
Further reading:
Jason Alan Snyder
Co-founder of SuperTruth and Artists & Robots, and an inventor on the Data Trust Index patents. Twenty-plus years building technology inside Interpublic Group. He writes here nearly every day on data trust, provenance, and what AI should be allowed to act on, and publishes essays on his Substack.
About SuperTruth · LinkedIn · Substack · jasonalansnyder.com
See it in practice
DTI scores the record, not the patient.
8 dimensions. 0–100. Travels with every record permanently.