Sepsis prediction algorithm data requirements: what trust score a model needs
Sepsis prediction algorithms consume six to dozens of clinical variables, but most validation studies never measure whether the underlying data was trustworthy enough to train on. A sepsis model scoring an AUROC of 0.85 on clean vital signs can collapse to near-random performance when fed late, miscoded, or unstandardized EHR data. This post maps the specific data requirements for sepsis AI and defines the trust score thresholds each input stream needs before any model should act on it.
Sepsis kills approximately 350,000 adults in U.S. hospitals every year. Early detection by even four to six hours can reduce mortality by 20% or more. That single statistic has fueled a wave of sepsis prediction algorithms, from Epic's deterioration index to open-source models like InSight and custom gradient-boosted classifiers trained on MIMIC-III data. But the conversation about these models almost always centers on model architecture and AUROC scores. It almost never centers on whether the data feeding the model was accurate, timely, or trustworthy enough to justify a clinical alert.
That gap is where sepsis prediction models fail in production.
What is the sepsis prediction model?
A sepsis prediction model is a machine learning system trained to identify patients at risk of developing sepsis before clinical criteria are fully met. Most models use a combination of vital signs, laboratory values, medication administration records, and demographic data to generate a risk probability. The goal is early intervention: if a model flags a patient six hours before organ dysfunction becomes apparent, clinicians can initiate fluid resuscitation and antibiotics sooner.
The most widely cited models fall into three categories. First, rule-based scoring systems like SIRS, qSOFA, and SOFA that use fixed thresholds. Second, machine learning models trained on retrospective EHR data, such as InSight (which uses only six vital signs) or the model described in Nemati et al. that uses 65 features. Third, proprietary commercial systems embedded in EHR platforms like Epic's sepsis early warning model.
Each category has different data requirements. But all of them share a single vulnerability: they assume the input data is correct.
What is the scoring system for sepsis?
Clinicians use several scoring systems to assess sepsis severity, and each one maps to specific data fields that must be present and accurate.
SIRS (Systemic Inflammatory Response Syndrome) requires four inputs: temperature, heart rate, respiratory rate, and white blood cell count. Meeting two or more criteria in the presence of suspected infection triggers a SIRS-positive classification.
qSOFA (Quick Sequential Organ Failure Assessment) uses three bedside variables: respiratory rate ≥22, altered mentation (Glasgow Coma Scale <15), and systolic blood pressure ≤100 mmHg. A score of 2 or more suggests high risk.
SOFA (Sequential Organ Failure Assessment) is more comprehensive, scoring six organ systems on a 0-4 scale each, for a maximum of 24. It requires PaO2/FiO2 ratio, platelet count, bilirubin, mean arterial pressure (plus vasopressor data), Glasgow Coma Scale, creatinine, and urine output.
Machine learning models typically ingest all of these variables plus dozens more. The InSight algorithm, validated across multiple centers, deliberately limits itself to six vital signs (systolic BP, diastolic BP, heart rate, respiratory rate, SpO2, and temperature) to reduce dependency on lab data that may arrive late. Other models incorporate lactate levels, procalcitonin, C-reactive protein, blood culture results, medication timing, and nursing assessments.
The critical point: every additional feature a model consumes is another data stream that can be wrong, late, or missing.
What is the score for the sepsis model?
When researchers evaluate a sepsis prediction model, they report discrimination metrics. The most common is the AUROC (Area Under the Receiver Operating Characteristic curve), which measures the model's ability to distinguish patients who will develop sepsis from those who will not.
High-performing sepsis models report AUROCs between 0.83 and 0.92 in retrospective validation. InSight reported 0.88 in its multicenter study. The Nemati et al. model achieved an AUROC of 0.85 with a prediction window of four hours before onset. Epic's sepsis model initially claimed strong performance but was later shown in external validation by Wong et al. (2021) to have an AUROC of only 0.63 in real-world deployment at a large academic medical center, with an alert-to-true-positive ratio that frustrated clinicians.
That gap between retrospective and prospective performance is not primarily a model problem. It is a data problem.
Retrospective studies use curated datasets where missing values are imputed, timestamps are clean, and lab results are already available. Real-time deployment faces a different reality: vital signs documented 45 minutes late, lab values pending for hours, medication administration records entered in batch at shift change, and inconsistent coding of clinical observations.
How to interpret sepsis score
A sepsis model's output is typically a probability between 0 and 1, or a risk tier (low, moderate, high). A patient with a model score of 0.72 has, according to the model, a 72% likelihood of meeting sepsis criteria within the prediction window. Clinical teams usually set an alerting threshold; for example, firing a best practice advisory (BPA) when the score exceeds 0.50.
But interpretation requires context that most implementations ignore.
A score of 0.72 generated from six vital signs that were all documented within the last 15 minutes and validated by the bedside nurse is a fundamentally different signal than a score of 0.72 generated from vital signs where two were carried forward from two hours ago, one was manually entered from a paper flowsheet, and the SpO2 reading came from a finger probe on a patient with poor perfusion.
The model does not know the difference. The model treats both inputs identically. And that is the core problem that data trust scoring solves.
Key statistics
Sepsis prediction data trust requirements can be quantified. These numbers frame what is actually at stake.
The six data streams that sepsis models consume and where trust breaks
Every sepsis prediction algorithm depends on a combination of these input categories. Each one has specific trust failure modes.
1. Vital signs. Temperature, heart rate, respiratory rate, blood pressure, SpO2. These are the backbone of minimal-feature models like InSight. Trust failures include carried-forward values (the EHR displays the last recorded value, which may be hours old), transcription errors from manual entry, and device artifact from motion or poor sensor contact. A recency score below 80 on any vital sign used in a sepsis model should trigger a confidence discount on the output.
2. Laboratory values. CBC with differential, lactate, creatinine, bilirubin, blood gases, procalcitonin. Trust failures include specimen collection-to-result lag (often 45 to 90 minutes for stat labs, longer for cultures), LOINC code mismatches that cause the wrong analyte to populate a model field, and reference range variation across analyzers. Without LOINC code standardization, a lactate of 2.1 mmol/L from one lab system may not be directly comparable to a lactate from another.
3. Medication administration records. Vasopressor initiation and dose, antibiotic timing, fluid bolus volume. SOFA scoring depends on vasopressor data. Trust failures include batch documentation at shift change (a norepinephrine drip started at 14:00 but documented at 19:00), free-text entries that do not map cleanly to RxNorm codes, and discrepancies between order time and actual administration time. RxNorm drug data integrity is directly relevant here.
4. Clinical assessments. Glasgow Coma Scale, nursing flowsheet observations, physician documentation of altered mental status. These are often the most subjective and least standardized inputs. A GCS score documented as "15" by default unless actively changed creates systematic underreporting of neurological decline.
5. Demographic and comorbidity data. Age, immunosuppression status, chronic conditions. These are typically pulled from problem lists or claims history. ICD-10 coding accuracy directly affects whether a patient's diabetes or chronic kidney disease is captured, which in turn affects baseline risk stratification.
6. Temporal metadata. The timestamps on every observation above. This is arguably the most critical trust dimension for sepsis models because the entire value proposition is early detection. A model that consumes a vital sign timestamped at 10:00 but actually measured at 08:30 has already lost 90 minutes of its prediction window. Recency is the most underrated dimension in health AI data scoring.
What trust score a sepsis model's training data needs
The Data Trust Index (DTI) scores every health data record from 0 to 100 across eight dimensions: Provenance (25%), Consent (20%), Recency (15%), Quality (10%), Concordance (10%), Validation (10%), Breadth (5%), and Stability (5%). For sepsis prediction specifically, certain dimensions carry disproportionate weight in determining whether a model's inputs are reliable enough to act on.
Provenance (25% of DTI). For sepsis data, provenance means knowing exactly which device generated a vital sign, which analyzer produced a lab result, and which clinician documented an assessment. A heart rate value with clear provenance (bedside monitor, Patient X, Room 412, 09:14:32 UTC) is categorically different from a heart rate imported from an external facility's summary document. Sepsis training data should require a provenance sub-score of 85 or higher.
Recency (15% of DTI). This is where sepsis models are most vulnerable. A model predicting onset four hours in the future cannot tolerate input data that is already two hours stale. Vital signs used in real-time sepsis scoring should have a recency sub-score above 90, meaning the time between measurement and model ingestion is under 15 minutes. Training datasets should document and enforce recency thresholds per feature.
Quality (10% of DTI). Completeness, plausibility range checks, and format standardization. A temperature of 106.2°F should trigger a plausibility flag, not silent ingestion. Missing lactate values should be explicitly flagged, not silently imputed with population means.
Concordance (10% of DTI). Does the vital sign trend agree with other concurrent observations? A heart rate of 55 in a patient whose vasopressor dose just doubled and whose MAP is 58 may indicate a documentation error rather than actual bradycardia. Cross-variable concordance checks are essential for sepsis data integrity.
Validation (10% of DTI). Has the data been independently verified? For vital signs, validation might mean cross-referencing the monitor's discrete data feed against the nurse's flowsheet entry. For lab values, it means confirming the result was released (not preliminary) and the specimen was not hemolyzed.
A composite DTI score of 80 or above (Gold tier) should be the minimum for any record used in sepsis model training. Records scoring below 60 (Bronze tier) should be excluded entirely. Records between 60 and 80 (Silver tier) can be used with appropriate weighting or uncertainty quantification.
For real-time inference, the threshold should be higher. A clinical alert that fires based on data with a composite DTI below 75 should carry a visible confidence modifier for the clinician reviewing it.
Why retrospective validation overstates sepsis model performance
The AUROC gap between validation and deployment is well documented. Wong et al. showed Epic's model dropping from reported performance to an AUROC of 0.63. Researchers at the University of Michigan found that the same model generated alerts with a positive predictive value of just 12%, meaning 88% of alerts were false positives.
These failures are routinely attributed to "distribution shift" or "population differences." Those explanations are incomplete. A significant portion of the degradation comes from data quality differences between the curated retrospective dataset and the live EHR feed.
Retrospective datasets like MIMIC-III have been cleaned, de-identified, and structured by research teams. Timestamps are internally consistent. Lab values are finalized. Missing data is explicitly marked. The live EHR feed has none of these properties. It has carried-forward values, pending results, documentation lag, and inconsistent coding.
Training a model on MIMIC-III and deploying it on a community hospital's live Epic feed is like testing a self-driving car on a closed track and then deploying it on a dirt road. The model did not change. The road did.
The alert fatigue problem is a data trust problem
Sepsis alert fatigue is the most commonly cited barrier to clinical adoption. Clinicians ignore alerts because too many are false positives. The standard response is to adjust the alerting threshold, raising it from 0.50 to 0.65 or 0.70 to reduce noise.
But threshold adjustment treats the symptom. The cause is often that the model is generating high scores from low-quality data. A carried-forward respiratory rate of 24 (documented two hours ago when the patient was anxious, now resolved) combined with a pending lactate that defaults to the last known value of 2.3 can push a model past its alerting threshold even though the patient's actual current state does not warrant concern.
If the model could see that the respiratory rate has a recency score of 40 and the lactate has a validation score of 30 (pending, not finalized), it could appropriately discount its own confidence. This is what data trust scoring enables: not better models, but models that know when their inputs are unreliable.
What this means for early warning system data quality
Health systems deploying sepsis early warning systems should evaluate their data infrastructure before evaluating their model. The questions are specific:
These are not theoretical questions. They are the operational prerequisites for a sepsis model that performs in production the way it performed in validation.
Building the trust layer before building the model
The current approach to sepsis AI development works backward. Teams build models on curated data, validate them on curated data, publish impressive AUROC numbers, and then struggle when deployment performance collapses. The data trust layer is an afterthought, if it exists at all.
The correct sequence is to score the data first. Before any model trains on a vital sign, that vital sign should have a provenance record, a recency timestamp verified against the source device, a quality check for plausibility, and a concordance check against concurrent observations. Before any model fires a clinical alert, the composite trust score of its current inputs should be visible to the clinician alongside the risk probability.
This is not a hypothetical architecture. SuperTruth's DTI Engine already scores health data records across these dimensions. When imaware needed to standardize 105,000 diagnostic records for clinical use, the DTI pipeline reduced processing time from three weeks to two hours while applying trust scoring at every step. The same framework applies to the vital signs, lab values, and medication records that sepsis models consume.
Sepsis prediction is a high-stakes, time-critical application where data trust is not optional. A model that cannot distinguish between a fresh, validated vital sign and a stale, carried-forward artifact is a model that will generate false alerts, erode clinician trust, and eventually get silenced. The trust score is not metadata. It is a clinical safety requirement.
The DTI Engine scores every health data record 0 to 100 across 8 trust dimensions before your AI model sees it. If your team is building or deploying sepsis prediction systems and needs to verify that the training and inference data meets clinical-grade trust thresholds, contact Louis Simeonidis at louis@supertruth.ai or (215) 918-4140.
Further reading:
Jason Alan Snyder
Co-founder of SuperTruth and Artists & Robots, and an inventor on the Data Trust Index patents. Twenty-plus years building technology inside Interpublic Group. He writes here nearly every day on data trust, provenance, and what AI should be allowed to act on, and publishes essays on his Substack.
About SuperTruth · LinkedIn · Substack · jasonalansnyder.com
See it in practice
The FICO score for health data.
8 dimensions. 0–100. Travels with every record permanently.