Skip to content

Clinicians in the loop: which records need one, and what the record must carry

Guide · Updated November 16, 2026 · Sources read 16 November 2026 · Maintained by SuperTruth Inc.

On 10 November 2026 Mercor announced a $750 million round that priced the company at $20 billion (Mercor, 10 November 2026), and priced an idea with it: AI needs expert humans to train and test it; the next day MDisrupt's Ruby Gadelrab answered that healthcare needs its own version, built around clinicians in the loop. This guide is about the half they leave out: not every record needs a clinician, and in the DTI sandbox a record at or above the review threshold with no conflicts between its sources goes straight through, while anything else goes to clinical review, by a clinician you supply, with the reasons attached. Nothing below is quoted from memory: every figure carries its source and the day it was read, 16 November 2026.

Model scores fall when the test moves from recall to practice

Benchmark scores, not performance in care. One review of 39 benchmarks, three kinds of test. Each band is the lowest and highest score the review reports for that kind, from different benchmarks and models. The two practice bands overlap; the safety band is lower at both ends.

Score, percent. Source: landscape report for the AMA Center for Digital Health and AI, with the Digital Medicine Society, May 2026, page 4, citing Gong et al., JMIR 2025. Report · Study. Read 16 November 2026.

See the numbers
Model scores fall when the test moves from recall to practice: Score, percent, the low and high of each range and what the range is
TestLow, %High, %What the range is
Medical knowledge benchmarks8490accuracy; 21 of the review's 39 benchmarks are knowledge-based. 84 to 90 is the accuracy of leading models on USMLE-style examinations, approaching or exceeding average physician performance.
Clinical decision-making tasks4569success rate; Practice-based benchmarks, 15 of 39. The ends of the range are two named benchmarks' success rates: DiagnosisArena at 45.82 percent and MedAgentBench at 69.67 percent; HealthBench sits at 60.
Patient safety evaluations4050accuracy; The review's task-level cut: safety assessment at 40 to 50 percent accuracy, against 85 to 93 for factual retrieval, 50 to 60 for clinical reasoning and 45 to 55 for diagnostic tasks.

AMA Center for Digital Health and AI with the Digital Medicine Society, From Benchmark to Bedside: The Current State of AI in Clinical Practice, Phase 1 Landscape Report, May 2026, page 4, citing Gong EJ, Bang CS, Lee JJ and Baik GH, Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks, Journal of Medical Internet Research 2025;27:e84120. The per-band notes are from the review's abstract. Report · Study. Read 16 November 2026.

Mercor priced the judge. MDisrupt named the clinician. We work one layer down.

Mercor is the expert marketplace that supplies AI labs with people to evaluate and train models; its 10 November 2026 round is about double the $10 billion of its October 2025 round (Mercor, 10 November 2026). Mercor says the money goes to "infrastructure and research for evals and post-training" and to environments where agents learn complex workflows.

The next day MDisrupt, a health expert network, published its answer: Mercor Is Worth $20B. Healthcare Needs Its Own Version, Built Around Clinicians in the Loop (Ruby Gadelrab, Founder and CEO, 11 November 2026). Its one-line case: "AI is only as trustworthy as the people who can tell a right answer from a plausible one." Its definition, which this guide adopts with credit: a clinician in the loop is "a practicing specialty or subspecialty clinician who works inside the AI build, not after it." It names a second role, the health expert in the loop: the revenue cycle, regulatory, bioinformatics and reimbursement specialists who know how a product is trained, paid for and approved.

The trade press calls this the expert data market (Dealroom, 6 July 2026: "AI labs buy expert data"); the glossary entry defines it. We agree with both arguments, and we work one layer down. The judge needs evidence. Before a clinician can say whether an answer is right, the record the answer came from has to be proven: where it came from, whether consent covers this use, how current it is, whether its sources agree. That is what SuperTruth does. It is also how you decide which records need a clinician at all.

The drop the AMA and DiMe reported, and their warning about the loop

A May 2026 landscape report prepared as a pre-read for the AMA Center for Digital Health and AI's convening, with support from the Digital Medicine Society (DiMe) puts the gap in one sentence: "Models that score 84 to 90% on medical knowledge benchmarks fall to 45 to 69% on clinical decision-making tasks and to 40 to 50% on patient safety evaluations." (From Benchmark to Bedside, page 4, citing a systematic review of 39 benchmarks, Gong et al., JMIR, 2025). These are benchmark scores, not measured performance in care. The three bands are different benchmarks and different models: the knowledge band is leading models' accuracy on licensing-style examinations, the decision band runs between two named practice benchmarks' success rates, and the safety band is the review's task-level cut. The chart above shows them on one scale; press a band for what it is made of.

The same report: "Roughly 60% of clinical AI pilots do not reach sustained deployment" (page 4, citing Rahamtalla et al., Future Internet, 2025, whose abstract puts it as "approximately 60% of projects failing beyond pilot testing"), and the barriers it names are workflow, manual data entry and clinician trust, not model accuracy alone. A clinician in the loop is the standard answer to that gap. The report's own caution, on page 9, is the reason this guide exists: "The concept of 'human in the loop' describes a workflow position and does not guarantee meaningful oversight."

The loop has two halves: the judge and the evidence

Every loop has a judge and the evidence the judge reads. The expert layer, MDisrupt's clinicians and health experts or Mercor's evaluators, is the judge: a person qualified to say whether an output is right for the patient in front of it. The record layer is the evidence: the chart, the lab result, the fill history and the intake form the output was built from. Oversight of an output built on an unproven record is oversight of the wrong thing. The clinician reviews a plausible answer, finds nothing wrong with the reasoning, and never sees that the medication list was from 2019.

So the record has to be proven before the clinician sees the answer, and the proof has to travel with it. Data truth is the state of a record that has been proved before anything acts on it. Data provenance, where a value came from and every hand it passed through, is the heaviest part of that proof.

Which records need a clinician in the loop, and what decides

Not all of them. A clinician's hour is the scarcest and most expensive input in the loop. Routing every record to a human is how pilots die of workflow. Routing none is how a wrong record reaches a patient. The decision should be made by the record's own score, at intake, with the reasons written down.

The Data Trust Index scores every record 0 to 100 across eight weighted dimensions and combines them into one composite, the number for the whole record (DTI white paper, 2026). The composite maps to SuperTruth's tiers: below 55 a record is held; Bronze (55 to 69) is what SuperTruth calls exploratory, Silver (70 to 79) operational-grade, Gold (80 to 89) clinical-grade and Platinum (90 and above) regulatory-grade. The names are SuperTruth's; no regulator has adopted them. A clinical use sets its floor in those terms. The review threshold is the floor for routing: the score a record needs to go straight through without a human. DTI scores any record; the published weights were calibrated in health. The strip and the table below show the weights, the tiers and the default threshold.

The DTI scale, the tiers, and where the default review threshold sits

A composite of 0 to 100. Below 55 a record is held. The sandbox's default review threshold is 80, the start of Gold: a record at or above it with no conflicts goes straight through.

See the numbers
DTI tiers on the 0 to 100 scale
TierLowHighMeaning
Held054held for remediation; enters no tiered use
Bronze5569exploratory
Silver7079operational-grade
Gold8089clinical-grade
Platinum90100regulatory-grade
The eight dimensions and their published default weights
DimensionWeight, %What it measures
Provenance25source pedigree, device attestation, chain of custody
Consent20explicitness, scope, duration, revocation
Recency15age against a decay curve for the kind of value
Quality10completeness, format, missingness, noise
Concordance10agreement between independent sources
Validation10outcome linkage and evidence grade of the data type
Breadth5how much the record covers
Stability5agreement of repeated measurements
Total100the composite

Weights, tiers and the fail band: the DTI white paper, DOI 10.5281/zenodo.19601616, 16 April 2026. The tier names and descriptors are SuperTruth's own (supertruth.ai/products/dti-engine); no regulator has adopted them. Review threshold: the public OpenAPI specification, version 4.5.0, supertruth.ai/api/openapi. Read 16 November 2026.

How the routing works

In the DTI sandbox, a free endpoint that scores a set of source documents without storing them (POST /sandbox in the public OpenAPI specification, version 4.5.0, read 16 November 2026; the full sandbox documentation sits behind the request form on supertruth.ai/developers):

  1. Every source document is scored element by element (each value in it), and the record gets a composite.
  2. Three conflicts between sources are checked from fixed tables (a written list of rules, never a model): a medication with no supporting diagnosis in any source, one medication at different doses across sources, and a condition staged in one source but end-stage in another.
  3. An optional review threshold, an integer from 0 to 100 with a default of 80, sets the route. A composite at or above it with no conflicts goes straight through. Anything else goes to clinical review, with the reasons attached.
  4. The threshold routes; it never changes a score (the specification's own words: "Routing only; scores never change with it"). Each source carries its own route too, so the reviewer sees which source pulled the record below the line.

The clinician on the other end of that route can be yours, or one from a network built for the work, such as MDisrupt's. SuperTruth does not supply clinicians. It supplies the record they can trust and says why it reached them.

What the record carries into review, and where the record stays

From the sandbox response, as the specification describes it on 16 November 2026:

  • The composite score and its tier, so the reviewer knows at a glance how far the record sits from the floor.
  • Each source's score, tier, flags and route: which values came from a CLIA-certified laboratory (CLIA is the federal certification for clinical laboratories), which from a patient's own intake form, and which source pulled the record below the line.
  • The element-level scores: one per value in the record.
  • The conflicts, named: which two sources disagree, and about what.
  • The route reasons: in plain words, why this record came to a human instead of going straight through.
  • An audit trail entry for the scoring event.

In the full API, what persists is the DTI sidecar: the score, the dimension breakdown and an audit hash; the record payload is processed and discarded. The developer documentation describes two deployments (read 16 November 2026): the hosted API, where records pass through SuperTruth's infrastructure for scoring and are not kept, and a containerized engine that would run inside your own environment so the record would never leave your perimeter, listed as planned and built on request. The sandbox and the public demo take synthetic data only; never post a real patient's record to either.

Four things the FDA's draft asks for, and what a scored record answers

The FDA's draft guidance on AI-enabled device software functions, issued 7 January 2025 under docket FDA-2024-D-4488 (the FDA's file number for the comment record) and still a draft, and so non-binding, on 16 November 2026 (FDA; the line numbers below are the FDA's own, from the PDF), asks for the same things in its own words. On data: information on "data collection, development and test data independence, reference standards, and representativeness" (lines 807 to 808), and "addressing representativeness in data collection for development, testing, and monitoring throughout the product lifecycle" (lines 249 to 250). On people: the performance of the "Human-AI team, rather than just the performance of the model in isolation" when a model has a "human in the loop" (lines 1147 to 1149, quoting the FDA's Good Machine Learning Practice principles). On subgroups: "device performance across important subgroups" by patient characteristics, sites and data collection equipment (lines 614 to 616). After launch: "manufacturers should have a postmarket performance monitoring plan" (line 1292).

The draft never uses the word provenance. The asks above are what provenance, consent and recency documentation on each record speak to. A per-record score with its audit hash is one input to the data management section; the route reasons are one way to describe what the human in the loop was shown and why; continuous re-scoring as sources change, and VIGIL, SuperTruth's scoring of the agents that use the data, can feed a monitoring plan. None of it is sufficient on its own. Whether a given product is a device, and what its submission needs, are questions for your regulatory counsel.

Three figures from MDisrupt's article this guide did not carry, and why

Clinician override rates of 1.7 percent for transparent AI and over 73 percent for opaque AI. The study behind the figure (Yu et al., Diagnostics 2025;15(17):2204) is a modelled, retrospective analysis of 6,689 intensive-care cases, and the two numbers measure different things: 1.7 percent is the override rate for predictions in the 90 to 99 percent confidence bin, 73.9 percent is the rate for diagnoses with minimal transparency. They are not one pair.

Yu et al., Diagnostics 2025, full text; the World Economic Forum piece of 5 December 2025 that MDisrupt cites repeats the pairing.

A $5.7 trillion market. The figure is CMS's projection of total US national health spending for 2025, up 7.3 percent, with $9.0 trillion projected for 2034 (CMS National Health Expenditure projections, June 2026). National spending is not a market for expert judgment in AI, so the guide does not use it.

CMS Office of the Actuary projections summary, June 2026, read via cms.gov; the AHA summary of 25 June 2026 carries the same numbers.

An FDA ask for data provenance. The January 2025 draft never uses the word; it asks for data management information (how data were collected, independence of development and test data, reference standards, representativeness). The guide quotes the FDA's wording with line numbers instead.

FDA draft guidance PDF, searched for "provenance": no occurrence.

The expert layer beside the record layer: MDisrupt, Mercor, SuperTruth

Three companies, as their own sites describe them on 16 November 2026:

MDisrupt: a health expert network of practicing clinicians and non-clinical health experts for healthcare AI training, labeling, evaluation and monitoring. Its Health Expert in the Loop product launched 8 January 2026.

mdisrupt.com, the Health Expert in the Loop and Clinicians in the Loop pages; Business Wire, 8 January 2026.

Mercor: an expert marketplace for AI labs, covering evaluation, post-training data and agent environments. It recruits physicians, nurses and pharmacists for medical work.

mercor.com, the 10 November 2026 announcement; work.mercor.com listings; mercor.com/resources/apex/ai-in-medicine, 6 October 2026.

SuperTruth: the record layer. DTI scores any record 0 to 100 before a model, a clinician or an agent acts on it; in the DTI sandbox, the ones that need a human go to clinical review with route reasons. DataSpine supplies the sourced figure about a place. VIGIL scores the agent while it works.

supertruth.ai/products and supertruth.ai/developers.

MDisrupt and Mercor are named so you can place them. SuperTruth is not affiliated with either and neither has endorsed it; their names are trademarks of their owners. The descriptions above are paraphrased from their own pages; every quotation on this page is marked and credited where it appears.

Run a synthetic record through the pipeline and watch the score move

Three ways in, by what you want to see. To see the route and its reasons, post two or three source documents to the sandbox endpoint and read the response; the endpoint is in the public OpenAPI specification, and the key is issued by request on the developers page. To see a score computed step by step without a key, run the pipeline on the DTI Engine page: three synthetic records, none describing a real person. To see the score move as sources change, remix a record from five synthetic sources. All three take synthetic data only.

Questions people ask

Which records should go to a clinician in the loop?

The ones the score cannot clear: in the DTI sandbox, a record whose composite is below the review threshold (default 80), or whose sources conflict, goes to clinical review with the reasons attached; the rest go straight through. The definition of the role itself is on the glossary entry.

Where do health experts in the loop fit in this?

They judge the parts of a product accuracy cannot: how it is paid for, how its algorithm is validated, what evidence a regulator expects. They need the same thing a clinician does from the record, proof of where each value came from. The glossary entry has MDisrupt's four roles.

Does SuperTruth provide clinicians?

No. SuperTruth scores the record; in the DTI sandbox the ones that need a human go to clinical review with the reasons attached. The clinician is yours, or comes from a network built for the work.

How does DTI decide which records go to clinical review?

By score and by conflict, in the DTI sandbox. A record whose composite is at or above the review threshold (default 80) with no conflicts between its sources goes straight through; any other record goes to clinical review with route reasons. The threshold routes; it never changes a score.

What is the difference between human in the loop and clinician in the loop?

Human in the loop is a position in a workflow: some person reviews or approves. The May 2026 landscape report for the AMA Center for Digital Health and AI, with DiMe, cautions that it "does not guarantee meaningful oversight". Clinician in the loop names who the human is and when they work: a practicing specialist, inside the build. Neither says whether the record under the output was proven; that is the record layer's job. The glossary entry has more.

Is MDisrupt or Mercor a SuperTruth partner?

No. They are named here so you can place the expert layer beside the record layer. SuperTruth is not affiliated with either and neither has endorsed it.

Sources

All sources read on 16 November 2026. Each entry says what was read; the full verification of MDisrupt's citations is kept with the site's research notes.

Mercor, "Organizing human intelligence for the entire economy", 10 November 2026: the $750 million round and the $20 billion price. https://www.mercor.com/blog/series-d/

The post in full. The October 2025 round's $10 billion is in Mercor's own post of 27 October 2025.

MDisrupt, Ruby Gadelrab, "Mercor Is Worth $20B. Healthcare Needs Its Own Version, Built Around Clinicians in the Loop", published 11 November 2026. https://mdisrupt.com/health-experts-and-clinicians-in-the-loop/

The article in full; the definitions of clinician in the loop and health expert in the loop are quoted from its FAQ.

"From Benchmark to Bedside: The Current State of AI in Clinical Practice", Phase 1 Landscape Report, May 2026, pages 4 and 9: a landscape report that describes itself as a pre-read for the AMA Center for Digital Health and AI's convening, prepared with support from the Digital Medicine Society. https://dimesociety.org/wp-content/uploads/AMA-DiMe-Phase-1-Landscape-Report.pdf

The PDF, served publicly on dimesociety.org; its file metadata is titled "Internal only". Page numbers are the printed folios. The benchmark ranges cite Gong et al. (2025); the 60 percent figure cites Rahamtalla et al. (2025); the human-in-the-loop sentence is the report's own.

Gong EJ, Bang CS, Lee JJ, Baik GH, "Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks", Journal of Medical Internet Research 2025;27:e84120, 1 December 2025. https://doi.org/10.2196/84120

The abstract, which carries the per-band figures used on the chart.

Rahamtalla BM et al., "The AI-Powered Healthcare Ecosystem: Bridging the Chasm Between Technical Validation and Systemic Integration: A Systematic Review", Future Internet 2025;17(12):550, 29 November 2025. https://doi.org/10.3390/fi17120550

The abstract: "approximately 60% of projects failing beyond pilot testing".

FDA, "Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations", draft guidance, 7 January 2025, docket FDA-2024-D-4488. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing

The guidance page and the PDF; still listed as draft. Line numbers are the FDA's own.

Yu et al., Diagnostics 2025;15(17):2204, the study behind the override-rate figure. https://doi.org/10.3390/diagnostics15172204

The full text; see the section on figures not carried.

CMS, National Health Expenditure Projections 2025 to 2034, June 2026. https://www.cms.gov/data-research/statistics-trends-and-reports/national-health-expenditure-data/projected

The forecast summary; see the section on figures not carried.

SuperTruth, "The Data Trust Index: A Multidimensional Framework for Evaluating Health Data Integrity in AI Systems", 16 April 2026, DOI 10.5281/zenodo.19601616. https://doi.org/10.5281/zenodo.19601616 The DTI API's public OpenAPI specification, version 4.5.0, at supertruth.ai/api/openapi.

Weights, tiers and the fail band: the paper. Routing, the threshold and the response fields: the specification's /sandbox entry. The conflicts and the two deployments: the developer documentation, which sits behind the request form on supertruth.ai/developers.