Surgical outcomes data trust: what the NSQIP registry requires for quality benchmarking

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is the closest thing surgery has to a single source of outcomes truth. Over 700 participating hospitals submit risk-adjusted, 30-day postoperative outcomes data on millions of cases each year. It is the gold standard for surgical quality benchmarking.

But gold standards corrode when the data underneath them is not verified. And NSQIP data, for all its rigor, carries structural vulnerabilities that most surgical AI developers never examine before training models on it.

What NSQIP actually collects

NSQIP is not a billing registry. Unlike claims-based quality programs, NSQIP relies on trained Surgical Clinical Reviewers (SCRs) who abstract data directly from medical records. Each participating hospital employs at least one SCR who captures preoperative risk factors, intraoperative details, and 30-day postoperative outcomes including mortality, morbidity, readmissions, and returns to the operating room.

The registry tracks over 150 variables per case. These include patient demographics, comorbidities (diabetes, smoking status, functional health status, ASA class), procedure-specific CPT codes, wound classification, operative time, and a standardized set of 22 postoperative complications.

NSQIP then generates risk-adjusted, observed-to-expected (O/E) ratios that allow hospitals to benchmark their performance against national averages. An O/E ratio below 1.0 means a hospital is performing better than predicted. Above 1.0 means worse.

This structure makes NSQIP data enormously valuable for AI. Risk-adjusted outcomes data, linked to granular clinical variables, is exactly what predictive models need. But the gap between what NSQIP is designed to collect and what actually makes it into the database is where surgical outcomes data trust breaks down.

The case capture problem

NSQIP does not require hospitals to submit every surgical case. The standard sampling methodology uses an 8-day cycle to select cases for abstraction. This means roughly 35-40% of eligible cases at a given hospital are captured in any given year.

For high-volume procedures like cholecystectomy or total knee arthroplasty, this sampling produces statistically reliable benchmarks. For lower-volume, higher-acuity procedures like pancreaticoduodenectomy or esophagectomy, the sample can become dangerously thin. A hospital performing 25 Whipple procedures per year might have only 9 or 10 abstracted into NSQIP.

When AI models train on NSQIP data to predict surgical complications, they inherit this sampling bias. Rare procedures and rare complications are underrepresented. The model learns the distribution of what was captured, not the distribution of what happened.

NSQIP's Procedure Targeted programs (colectomy, pancreatectomy, hepatectomy, and others) address part of this problem by requiring 100% case capture for specific operations. But most hospitals participate in the general NSQIP program, not the targeted modules. And the targeted modules cover only a fraction of surgical specialties.

Abstraction quality varies more than hospitals admit

SCR abstraction is the backbone of NSQIP data quality. The ACS requires SCRs to pass a certification exam and complete ongoing education. Inter-rater reliability audits occur, and the ACS reports disagreement rates across participating sites.

But the reality on the ground is more complicated. SCR turnover at community hospitals averages 15-20% annually, according to estimates from quality improvement officers at mid-size health systems. New SCRs, even after training, take 6-12 months to reach proficiency in complex variable definitions.

Consider wound classification. NSQIP uses a 4-tier system: clean, clean-contaminated, contaminated, and dirty/infected. Misclassification of wound class directly affects risk adjustment. A case coded as "clean" when it was "clean-contaminated" will appear to have a higher-than-expected complication rate, distorting the O/E ratio.

Or consider the definition of "superficial surgical site infection." NSQIP requires documentation of specific clinical criteria within 30 days. If the SCR does not find explicit documentation in the chart, the complication is not captured, even if the patient was treated for one. Documentation gaps become data gaps.

These abstraction inconsistencies compound across hundreds of hospitals. The resulting dataset looks clean in aggregate but carries site-level noise that any downstream AI model will absorb.

The 30-day window creates a ceiling on what outcomes data can tell you

NSQIP tracks outcomes at 30 days postoperatively. This window captures most acute complications: surgical site infections, pneumonia, urinary tract infections, VTE, cardiac events, sepsis, and death. For acute care surgery and many elective procedures, 30 days is a reasonable horizon.

But for surgical AI benchmarking, 30 days is often not enough. Readmissions after joint replacement often cluster between days 30 and 90. Hernia recurrence rates peak at 1-2 years. Oncologic outcomes like disease-free survival require years of follow-up.

When AI models trained on NSQIP data are deployed to predict "surgical outcomes," the implicit assumption is that 30-day outcomes are a reliable proxy for longer-term results. Sometimes they are. Often they are not. And the model cannot know the difference because the data stops at day 30.

This is a recency and breadth problem in data trust terms. The data is recent enough to capture acute events but not broad enough to capture the full outcome trajectory. Any trust scoring framework applied to NSQIP data needs to flag this limitation explicitly, not just for individual records but for the entire dataset's temporal architecture.

CPT coding integrity determines whether cases are even comparable

NSQIP uses CPT codes to categorize procedures and assign risk models. A case coded under CPT 44204 (laparoscopic partial colectomy) receives a different expected complication rate than one coded under CPT 44140 (open partial colectomy). If the CPT code is wrong, the risk adjustment is wrong, and the O/E ratio becomes meaningless.

CPT miscoding in surgical registries is not rare. A 2019 analysis of surgical coding accuracy found disagreement rates between operative reports and submitted CPT codes of 10-25% depending on specialty. In complex cases involving multiple procedures, the primary CPT code selection can shift risk adjustment by 15-30%.

This is the same CPT code integrity problem that plagues claims-based AI models, but in NSQIP the stakes are different. Claims coding errors affect reimbursement. NSQIP coding errors affect quality benchmarks that determine whether a hospital is flagged as an outlier, whether a surgeon's outcomes are questioned, and whether an AI model learns the right risk profile for a given procedure.

Key statistics

NSQIP data quality vulnerabilities by source

NSQIP participation spans over 700 hospitals contributing risk-adjusted outcomes data, making it the largest clinical surgical outcomes registry in the United States.

Standard NSQIP sampling captures approximately 35-40% of eligible surgical cases per hospital per year, leaving the majority of cases outside the registry's benchmarking dataset.

SCR turnover at community hospitals is estimated at 15-20% annually, creating rolling gaps in abstraction expertise that directly affect variable accuracy.

CPT code disagreement between operative reports and submitted codes ranges from 10-25% across surgical specialties, with risk adjustment shifts of 15-30% in complex multi-procedure cases.

SuperTruth's DTI Engine processed 105,000 diagnostic records for imaware in 2 hours, a task that previously required 3 weeks, demonstrating that trust scoring at registry scale is operationally feasible.

What surgical AI benchmarking actually requires from the data

DTI trust dimensions applied to NSQIP surgical outcomes data

Building a surgical outcomes prediction model on NSQIP data is not a simple supervised learning problem. The data demands scrutiny across multiple trust dimensions before it can serve as reliable ground truth.

Provenance. Every NSQIP record originates from a specific hospital, abstracted by a specific SCR, using a specific version of the NSQIP data dictionary. Changes in variable definitions across program years (and there have been many) mean that a "pneumonia" complication in 2015 may not mean the same thing as "pneumonia" in 2023. Models trained across multiple program years without accounting for definitional drift will conflate different clinical entities.

Concordance. NSQIP data should agree with the hospital's own EHR data, operative reports, and discharge summaries. But concordance audits are expensive, and most hospitals perform them only when required by ACS site visits (typically every 3 years). Between audits, drift accumulates. A concordance score applied at the record level would surface discrepancies before they propagate into training sets.

Recency. NSQIP data files are released semi-annually, with a reporting lag of approximately 6-9 months. A model trained on the most recent NSQIP release is learning from cases that are already a year old. For rapidly evolving surgical practices (robotic surgery adoption, enhanced recovery protocols), this lag means the model's learned patterns may already be outdated. As we have explored in how temporal drift destroys AI model accuracy, even modest staleness in training data can degrade predictive performance.

Validation. NSQIP has built-in validation through ACS site audits, inter-rater reliability testing, and automated data checks. But these validation mechanisms operate at the program level, not the record level. A hospital can pass its site audit while individual records within its submission contain errors. Record-level validation scoring would catch what program-level audits miss.

The risk adjustment trap

NSQIP's risk adjustment models are its most valuable feature and its most dangerous assumption for AI developers. The models use logistic regression to estimate expected complication rates based on preoperative risk factors. When the risk factor data is accurate, risk adjustment works well. When it is not, the adjustment creates false confidence.

Consider ASA classification, a subjective assessment of a patient's preoperative physical status scored 1-5 by the anesthesiologist. ASA class is one of the strongest predictors in NSQIP risk models. But inter-rater agreement on ASA classification has been shown to be only 56-67% in published studies. Two anesthesiologists evaluating the same patient may assign different ASA scores.

This subjectivity flows directly into NSQIP's risk adjustment. A hospital where anesthesiologists tend to assign higher ASA scores will appear to have sicker patients and will receive more "credit" in risk adjustment. Its O/E ratio will look better than a hospital whose anesthesiologists assign lower ASA scores for equivalent patients.

AI models that use NSQIP O/E ratios as labels or features absorb this ASA classification noise without any mechanism to detect it. A trust score that evaluates the consistency and provenance of subjective clinical variables like ASA class would flag this vulnerability before the data reaches a model.

How NSQIP data compares to other surgical data sources

NSQIP is not the only source of surgical outcomes data. The Society of Thoracic Surgeons (STS) database, the Vascular Quality Initiative (VQI), state-level all-payer claims databases, and institutional EHR extracts all serve as alternative or supplementary sources.

Each carries different trust profiles. STS data has higher case capture rates for cardiac surgery but narrower specialty coverage. Claims data covers all payers but lacks clinical granularity and suffers from coding accuracy problems that compound at scale. EHR extracts offer the deepest clinical detail but vary wildly in structure and completeness across institutions.

The responsible approach for surgical AI benchmarking is not to pick one source and assume it is trustworthy. It is to score each source, each record, and each variable against a consistent trust framework before any model training begins. That is what the DTI was designed to do.

Why surgical AI cannot skip the trust layer

Surgical AI is moving fast. Models now predict postoperative complications, estimate length of stay, identify high-risk patients for enhanced recovery protocols, and flag cases for quality review. Several FDA-cleared surgical AI products are already on the market.

But the underlying assumption in most of these products is that the training data, much of it derived from or benchmarked against NSQIP, is trustworthy. That assumption is wrong, or at least unverified.

NSQIP data is better than most clinical data. It is abstracted by trained reviewers, validated by site audits, and governed by a well-resourced national program. But "better than most" is not a trust standard. It is a relative claim that provides no quantitative assurance about any individual record's fitness for AI use.

AI explainability does not solve this problem. Knowing why a model made a prediction is useless if the data the model learned from was never scored for integrity. Surgical outcomes data trust requires a step that comes before model development: systematic, record-level evaluation of every data element that will enter the training pipeline.

SuperTruth scores incoming EHR data at the point of ingestion, before it reaches a model. If your system is deploying clinical AI and needs to answer an auditor's questions about the integrity of the surgical outcomes data feeding your quality benchmarks, contact Louis Simeonidis at louis@supertruth.ai or (215) 918-4140.

Further reading:

DTI™ Engine

Health systems solution

CPT code integrity: what claims data quality means for AI model accuracy

How temporal drift destroys AI model accuracy in healthcare

Readmission prediction model bias: how training data trust affects clinical AI