Natural language processing on EHR notes: data trust requirements before NLP
Up to 80% of clinical data sits in unstructured EHR notes, yet most NLP pipelines never verify the trustworthiness of that text before processing it. Without provenance, recency, and quality scoring on the source notes themselves, clinical NLP outputs inherit every upstream data problem and amplify it at scale.
Approximately 80% of all clinical data in electronic health records exists as unstructured free text. Progress notes, discharge summaries, radiology reports, operative notes, nursing assessments, consult letters. This is the raw material that NLP models consume when they extract diagnoses, identify adverse events, flag social determinants, or populate research cohorts.
The problem is not whether NLP can parse clinical text. Modern transformer-based models handle medical language with increasing accuracy. The problem is whether the text itself deserves to be parsed at all.
Most clinical NLP pipelines treat every note as equal input. A discharge summary dictated by a rushed resident at 3 a.m., a copy-forwarded note from a system migration five years ago, a templated assessment auto-populated from a macro. The NLP model processes them all identically, extracts entities, and passes structured outputs downstream. Nobody asks whether the source note was trustworthy before the model touched it.
That is where data trust enters.
The unstructured data quality crisis nobody measures
Structured EHR fields get extensive validation. Lab values have reference ranges. ICD-10 codes have edit checks. Medication lists reconcile against RxNorm. But unstructured notes exist in a quality vacuum.
A 2023 study in the Journal of the American Medical Informatics Association found that 26% of clinical notes contained copy-and-paste content from prior encounters. Another analysis from UCSF found that up to 54% of note text in some specialties was duplicated or auto-generated. When NLP models extract clinical concepts from duplicated text, they count the same finding multiple times. A single mention of "chest pain" becomes three or four mentions across copied notes, inflating the apparent significance of a symptom that may have resolved weeks ago.
Recency compounds this problem. A note from a 2019 hospitalization may still appear in an NLP pipeline processing 2025 data if nobody filters by encounter date. The model has no concept of clinical relevance decay. It treats a six-year-old progress note with the same weight as yesterday's assessment.
Without a trust score on each note before NLP processing, every downstream extraction inherits these problems.
Why provenance is the first requirement for clinical NLP
Provenance answers the question: where did this note come from, who created it, when, and through what system?
This sounds basic. It is not. EHR notes arrive through multiple pathways. A clinician dictates a note using Dragon Medical. A nurse enters a flowsheet narrative. A specialist sends a consult letter via fax that gets scanned and OCR'd into the record. A prior system migrates notes during a platform transition. An external health information exchange delivers a care summary.
Each pathway introduces different reliability characteristics. Dictated notes carry voice recognition errors. Scanned documents carry OCR artifacts. Migrated notes may lose their original timestamps or author attribution. HIE-delivered summaries may be incomplete or delayed.
An NLP pipeline that does not track provenance cannot distinguish between a note dictated by the attending physician who examined the patient and a scanned fax from an outside facility that was OCR'd with 87% character accuracy. Both enter the pipeline as text. Both get processed identically. Both produce structured outputs that look equally authoritative.
The DTI framework assigns provenance 25% of the total trust score for exactly this reason. For unstructured clinical notes, provenance is not metadata. It is the difference between signal and noise.
Seven data trust requirements before NLP processing
Before any NLP model processes a clinical note, the note itself needs to pass through a trust evaluation. Here are the specific requirements.
1. Author attribution verification
Every note should have a verified author with credentials traceable to the encounter. Notes with missing author fields, generic system accounts ("INTERFACE_ENGINE"), or unresolved proxy documentation should be flagged. If the author cannot be verified, the NLP output should carry a reduced confidence score.
2. Timestamp integrity
The note's creation timestamp must correspond to a real clinical encounter. Notes with default timestamps (midnight on admission date), notes backdated by more than 72 hours, and notes with timestamps that precede the patient's registration should be quarantined. NLP extractions from temporally questionable notes contaminate any time-series analysis downstream.
3. Copy-paste and note bloat detection
Before NLP processing, each note should be evaluated for duplicated content. Text-similarity algorithms can detect when more than 40% of a note's content appears verbatim in a prior note for the same patient. Duplicated sections should be tagged so that NLP models can de-weight or skip repeated content rather than counting it as independent clinical evidence.
4. Document type classification
Not all notes carry equal clinical weight. A signed attending physician progress note differs from a student note, a nursing assessment, a telephone encounter, or an auto-generated medication reconciliation summary. NLP pipelines should classify document types before extraction and weight outputs accordingly. Extracting a cancer diagnosis from a discharge summary is different from extracting it from a triage nurse's intake note.
5. Source system provenance
Notes that entered the current EHR through migration, interface, or health information exchange need source system tagging. A note originally created in a Meditech system that was migrated to Epic during a platform transition may have lost formatting, special characters, section headers, or structured references. The NLP model needs to know this. Processing a migrated note as if it were a native note introduces extraction errors that compound across the dataset.
6. Recency scoring
Clinical relevance decays with time. A progress note from an active admission carries different weight than an archived note from a resolved episode five years ago. NLP pipelines should apply recency scoring to every note, with configurable thresholds based on the use case. A readmission prediction model may need notes from the past 90 days. A cancer registry extraction may need notes from the past 24 months. A research cohort identification project may need the full longitudinal record but with temporal weighting.
The DTI framework assigns recency 15% of the total trust score. For NLP applications, this dimension often determines whether an extraction is clinically actionable or historically interesting but operationally irrelevant.
7. Consent verification for NLP use
Processing clinical notes with NLP for research, quality improvement, or AI model training requires consent verification. Not all notes carry the same consent status. Behavioral health notes, substance use disorder notes, and psychotherapy notes have special protections under 42 CFR Part 2 and state privacy laws. An NLP pipeline that processes all notes without consent stratification creates regulatory exposure.
The DTI framework assigns consent 20% of the total trust score. For unstructured notes, consent is not a binary check. It is a multi-tier classification that determines which notes can be processed, for what purpose, and by whom.
Key statistics
What happens when NLP runs on unscored notes
The consequences of skipping trust evaluation are not theoretical. They show up in production.
A health system deploying an NLP-based sepsis surveillance tool extracts "fever" and "tachycardia" mentions from clinical notes. But 30% of those mentions come from copy-forwarded assessments documenting resolved symptoms from prior shifts. The model flags patients for sepsis workup based on stale clinical text. Alert fatigue increases. Clinicians lose trust in the system. The tool gets turned off within six months.
A pharma company building a real-world evidence dataset extracts adverse event mentions from clinical notes across 15 health systems. But three of those systems recently migrated from legacy EHRs, and their migrated notes lost section headers. The NLP model cannot distinguish between "history of" mentions and "active problem" mentions because the structural cues are missing. The adverse event rate looks 40% higher than it actually is. The FDA submission gets delayed for re-analysis.
A research team building a lung cancer cohort identifies patients with "nodule" mentions in radiology notes. But the NLP pipeline does not filter by note recency. It pulls mentions from CT reports dating back eight years. Patients who had benign nodules resolved years ago appear in the cohort alongside patients with active findings. The cohort is contaminated before enrollment begins.
Every one of these failures traces back to the same root cause: the NLP model processed text that had never been evaluated for trustworthiness.
The DTI framework applied to clinical notes
SuperTruth's Data Trust Index scores every health data record on a 0-100 scale across eight dimensions. For unstructured clinical notes, the framework maps directly to the trust requirements described above.
Provenance (25%) covers author attribution, source system identification, and the pathway through which the note entered the current record. Consent (20%) covers the legal and regulatory status of the note for the intended use. Recency (15%) scores the temporal relevance of the note relative to the clinical question being asked. Quality (10%) evaluates completeness, structural integrity, and the presence of known corruption patterns like OCR artifacts or encoding errors. Concordance (10%) checks whether the note's content aligns with other data sources for the same patient and encounter. Validation (10%) confirms that the note has been signed, authenticated, and finalized rather than existing in draft or unsigned status. Breadth (5%) evaluates whether the note provides sufficient clinical context or is a fragmentary entry. Stability (5%) measures whether the note has been amended, addended, or corrected, and whether those changes are tracked.
A note that scores below a configurable DTI floor should not enter an NLP pipeline without explicit acknowledgment of its limitations. Setting a DTI floor of 60 for NLP processing means that notes with poor provenance, outdated timestamps, or unverified consent status are excluded before the model ever sees them.
This is not quality control after extraction. This is trust evaluation before processing. The distinction matters because NLP errors compound. A bad extraction from a bad note gets normalized into a structured field, which gets aggregated into a dataset, which trains a model, which makes a clinical recommendation. By the time someone notices the error, it has been laundered through three layers of processing and looks like a verified clinical fact.
SNOMED, LOINC, and the terminology mapping problem in NLP output
Even when NLP correctly extracts a clinical concept from a trustworthy note, the mapping of that concept to a standard terminology introduces another trust challenge. NLP models extract text spans. Those spans need to be mapped to SNOMED CT concepts, ICD-10 codes, or LOINC codes for downstream use.
Mapping accuracy depends on context. The word "depression" in a psychiatric assessment maps to a mood disorder. The word "depression" in a chest X-ray report describes an anatomical finding. The word "depression" in a social work note may describe a patient's self-reported symptom without clinical validation.
Terminology mapping errors in NLP output do not get caught by the same edit checks that catch coding errors in structured data. There is no claims adjudication process for NLP-extracted SNOMED codes. The extracted concept enters a research database or a clinical decision support system with no external validation.
This is why SNOMED CT mapping trust requires the same rigor for NLP-derived codes as for manually entered codes. And why trust scoring on the source note provides an upstream safeguard that terminology validation alone cannot.
The operational case for pre-NLP trust scoring
Scoring notes before NLP processing is not just a quality measure. It is an operational decision that reduces cost.
NLP processing at scale is computationally expensive. Running transformer-based clinical NLP models on millions of notes requires significant GPU resources. If 25-30% of those notes are duplicated, outdated, or provenance-deficient, the organization is spending compute on text that will produce unreliable outputs.
Pre-NLP trust scoring acts as a filter. Score the notes first. Process only the notes that meet the trust threshold. This reduces compute cost, improves extraction accuracy, and produces outputs that can withstand audit.
SuperTruth's work with imaware demonstrated this principle with diagnostic records: scoring and standardizing 105,000 records reduced processing time from three weeks to two hours. The same principle applies to clinical notes. Score first. Process second. Trust the output because you trusted the input.
What FDA expects from NLP-derived evidence
The FDA's 2023 guidance on AI and machine learning in drug development explicitly addresses the quality of training data. For NLP-derived real-world evidence, the agency expects documentation of data provenance, quality assurance processes, and validation of extraction accuracy.
An NLP pipeline that cannot demonstrate provenance on its source notes cannot meet these requirements. If the FDA asks "where did this adverse event extraction come from?" the answer cannot be "a clinical note somewhere in the dataset." The answer needs to include the note's author, creation date, source system, document type, and consent status.
This is what FDA AI guidance and data provenance requirements are converging toward. Pre-NLP trust scoring creates the audit trail that regulators will expect.
Scoring notes before NLP is not optional
The NLP community has spent years improving model accuracy. Better tokenizers, better entity recognition, better relation extraction, better negation detection. These are real advances.
But model accuracy on untrustworthy input produces precisely calibrated garbage. A model that extracts clinical concepts with 95% accuracy from notes that are 30% duplicated, 15% outdated, and 10% provenance-unknown does not produce 95% accurate clinical intelligence. It produces confident-looking outputs built on a foundation that nobody verified.
The fix is not a better model. The fix is a better input. Score the notes. Verify provenance. Check recency. Confirm consent. Filter by document type. Then run NLP.
SuperTruth scores incoming EHR data at the point of ingestion, before it reaches a model. If your system is deploying clinical NLP and needs to answer an auditor's questions about where the training data came from and whether it was trustworthy, contact Louis Simeonidis at louis@supertruth.ai or (215) 918-4140.
Further reading:
Jason Alan Snyder
Co-founder of SuperTruth and Artists & Robots, and an inventor on the Data Trust Index patents. Twenty-plus years building technology inside Interpublic Group. He writes here nearly every day on data trust, provenance, and what AI should be allowed to act on, and publishes essays on his Substack.
About SuperTruth · LinkedIn · Substack · jasonalansnyder.com
See it in practice
EHR data scored before any AI model sees it.
DTI integrates with Epic, Oracle Health, and all major EHR systems.