How the FDA Will Audit Your Health AI's Training Data

FDA's 2021 AI/ML-Based Software as a Medical Device action plan identified training data quality as the most significant unaddressed gap in the current regulatory framework for clinical AI. Three years later, that gap has not closed. The agency is moving toward formal guidance that will require developers to document the quality of their training data before receiving regulatory clearance.

When an FDA auditor arrives at a health AI company, they will ask six questions about training data. Most health AI companies cannot answer any of them.

The six questions

1. What were the inclusion and exclusion criteria for training data selection?

Not at the dataset level — at the record level. Why is this specific patient record in the training dataset? What criteria determined it was appropriate for inclusion? What criteria would have excluded it?

Most health AI training pipelines do not have record-level inclusion criteria. They have dataset-level criteria: include records from these systems, within this date range, with these diagnosis codes. The auditor will push past the dataset criteria to the record level, and at the record level, most pipelines have no answer.

2. What is the source of each training record and what is the chain of custody?

The auditor wants to know: which institution generated this record, which system captured it, how it moved from the source to the training pipeline, and who had access to it along the way. This is the provenance chain.

For most health AI training datasets, the provenance chain is partially documented at best. The auditor will find gaps.

3. What was each patient's consent for this specific use of their data?

Not the general HIPAA notice. The specific authorization that covers AI training for this specific application. As discussed in previous work on this topic, HIPAA treatment authorization does not cover AI model training. The auditor knows this. They will ask for the specific authorization.

4. How did you verify that training data meets your stated quality thresholds?

If the developer claims the training data meets clinical-grade standards, the auditor will ask how that was verified. What is the verification methodology? What threshold was applied? What records were excluded on quality grounds, and how were they identified?

5. How does your model's performance change when trained on different subsets of your training data?

This is the bias and subgroup analysis question. The auditor wants to know whether the model performs differently on data from different institutions, demographics, or time periods. The answer requires being able to segment training data by source quality — which requires record-level scoring.

6. If a patient withdraws consent after your model is trained, what happens to the model?

This is the emerging question that most health AI developers have not yet addressed. If a patient who contributed records to the training dataset later revokes consent, does the model need to be retrained? What is the process? What is the documentation?

What documentation you need before the audit

The six questions above map to six categories of documentation:

Inclusion criteria documentation — record-level criteria logged at ingestion time, not reconstructed after the fact.

Provenance records — source attribution, chain of custody, access log for every training record.

Consent documentation — record-level consent tier at the time of training inclusion, with the authorization basis documented.

Quality assessment records — per-record quality scores with the methodology used to derive them, and the threshold applied for inclusion.

Subgroup analysis — model performance broken down by training data source quality, demographic subgroup, and time period.

Consent revocation protocol — documented process for handling post-training consent revocations, with evidence it has been tested.

SuperTruth's DTI pipeline generates the documentation for questions 1-4 and 6 automatically at ingestion time. Question 5 requires the developer to run the subgroup analysis — but the segmentation data exists in the DTI scores.

The FDA audit is coming. The developers who survive it will be the ones who built the documentation infrastructure before they were asked for it.

Further reading: Health AI data trust infrastructure and The HIPAA problem health AI companies are ignoring