A benchmark turns into marketing at the moment its definitions go missing. So this page leads with what is counted, how, and by whom, and only then gives the numbers. The full methodology is a public document in our repository and the run report it summarises is committed beside it.
What is measured
The unit is a clinical assertion: an atomic statement about the patient that could independently be true or false. A symptom, the presence or absence of a symptom, a history item, an exam finding, a measurement, a diagnosis, or a plan item. “Denies fever, chills, and nausea” is three assertions. A measurement and its value are one. Section headers and boilerplate that make no claim about the patient are excluded. Hedging is kept verbatim: “possible chest pressure” is extracted as possible chest pressure, not as chest pressure.
Every assertion gets exactly one label against the transcript:
| Label | Meaning |
|---|---|
| S · Supported | The transcript states it, or the assertion is a direct paraphrase, a translation into clinical terminology, or a faithful aggregation of stated content. A denial in the note that matches a denial in the transcript is Supported. |
| I · Inferred | Not stated, but a reasonable clinical inference with a real basis in the transcript. Tracked separately because it is the genuinely contested category: neither fabrication nor grounding. |
| U · Unsupported | No basis in the transcript. An invented detail, a value never given, a symptom never mentioned, an order never placed. |
| C · Contradicted | The transcript states the opposite. Asserting a symptom the patient denied, asserting a treatment that was declined or deferred, attributing another person’s symptom to the patient, or stating a diagnosis the clinician refused to make. |
Assertions the judge fails to label are reported as Unlabelled. They are never dropped and never defaulted to a flattering value.
The metrics
- Hallucination rate
- (Unsupported + Contradicted) divided by all assertions. Lower is better. This is the headline.
- Contradiction rate
- Contradicted divided by all assertions. Lower is better. This is the class with the highest clinical consequence.
- Key-fact coverage
- Key facts present in the note divided by key facts defined for the case. Higher is better. A note that says only "Patient was seen" scores perfect faithfulness, so faithfulness and coverage are always read together.
- Trap fire rate
- Traps fired divided by traps defined. Each trap is a specific fabrication the note must not contain, expressed as a pattern plus a rationale. Traps need no model, so they are the one measure here that cannot drift with one.
The traps are the deterministic layer. Each case defines specific fabrications the note must not contain. A trap fires only when a mention survives suppression by a negation or conditional cue in the same sentence (“Denies cough, headaches, dizziness” and “Call if fever develops” do not fire). Every mention, suppressed or not, is written to the artifact with the rule that suppressed it, so any single suppression can be challenged by a reviewer.
The corpus
Twelve transcripts, 144 key facts, 61 traps. All of it is original synthetic dialogue: no real patient data, no de-identified real data, no transcription of any real encounter. The corpus was committed to the repository before the harness existed and before any generation run, so the expectations could not be written to match the output. Seven of the twelve cases are adversarial, where the faithful answer is to say less.
| Case | Kind | What it probes |
|---|---|---|
| FB-001 | standard | Hypertension follow-up, uncomplicated |
| FB-002 | standard | Viral URI, antibiotic requested and declined |
| FB-003 | standard | Diabetes follow-up, regimen simplification, GLP-1 deferred |
| FB-004 | standard | Behavioral health intake, passive SI without plan |
| FB-005 | standard | Patellofemoral knee pain, MRI declined |
| FB-006 | adversarial | Dense pertinent negatives, 11 explicit denials |
| FB-007 | adversarial | Crosstalk, a spouse describes her own symptoms |
| FB-008 | adversarial | Pervasive hedging, the faithful answer is to stay uncertain |
| FB-009 | adversarial | Clinician reasons aloud and explicitly declines to diagnose |
| FB-010 | adversarial | Specifics absent: no vitals taken, dose unknown, lab result unavailable |
| FB-011 | adversarial | Audio dropouts remove clinically important content |
| FB-012 | adversarial | Treatments discussed then explicitly declined or deferred |
The two arms
Ungrounded is the scribe draft as generated. It is the baseline to lose to. Grounded is the same draft after the production grounding verifier, which removes sentences it cannot trace to the transcript and, since the 2026-08-09 changes, removes individual items from an enumerated denial when nobody actually denied them. One generation feeds both arms, so the comparison isolates the verifier’s effect instead of mixing it with run-to-run generation variance. That design is valid only because the verifier is subtractive: it deletes, it never adds content.
Every run also injects three unambiguous fabrications into every grounded note (a blood pressure never taken, a cough never reported, an antibiotic never started) and checks that each comes back Unsupported or Contradicted. If any planted fabrication escapes, every other number in that run is suspect and the report says so at the top.
Results, 2026-08-09 run
Generation, grounding, and primary judging all used gpt-4o. Cost floor for the run was $0.98. The negative control caught 36 of 36 planted fabrications.
| Metric | Ungrounded (baseline) | Grounded (shipped path) |
|---|---|---|
| Clinical assertions extracted | 296 | 279 |
| Hallucination rate (unsupported + contradicted) | 1.7% | 0.0% |
| Contradiction rate | 1.0% | 0.0% |
| Faithfulness precision (supported) | 94.6% | 97.8% |
| Inference rate (plausible, not stated) | 3.7% | 2.2% |
| Key-fact coverage, strict | 88.2% | 86.8% |
| Traps fired | 6 of 61 | 5 of 61 |
| Planted fabrications caught (negative control) | n/a | 36 of 36 |
Read the two coverage numbers together with the hallucination numbers. The grounded arm removed every unsupported and contradicted assertion the judge found on this corpus, and it paid for that with 1.4 percentage points of strict key-fact coverage. Five traps still fired in the grounded arm: two in the crosstalk case, where the pattern cannot tell whose symptom a sentence is about, and three in the declined-treatment case, where deferred items were listed under the plan.
Judge agreement was measured on 163 assertions across the six standard cases, holding the assertion list fixed. The same judge re-run agreed with itself on every label. A different model, gpt-4o-mini, agreed on 96.3% of the four-way labels and on every binary hallucination-or-not call. At a base rate this low, almost every assertion is in the majority class, so raw agreement is inflated. Read the disagreement counts in the artifact, not the kappa alone.
Limitations
These are copied in substance from the methodology document. They are not footnotes. Several of them are reasons the numbers above could be wrong.
Limitation 1
The judge is a model from the same family that wrote the notes
Labels are assigned by gpt-4o, not by a human, and gpt-4o also generated the notes. Self-preference bias is documented in the literature on model judges and it is uncontrolled here. No clinician has adjudicated any label. The harness author, an AI agent, also wrote the corpus, the traps, and the judge prompts. Agreement is measured every run against a re-run of the same judge and against a different model (gpt-4o-mini), but agreement between two models is not the same thing as agreement with a clinician.
Limitation 2
The corpus is synthetic and small
Twelve cases of written English dialogue, authored by the same agent that built the harness, with no real or de-identified patient data. They are not ASR output, so they carry none of the disfluency, substitution, and speaker-attribution errors of real transcripts except where one case crudely simulates dropouts. Per-case differences are anecdotes. Only large aggregate gaps are signal, and no confidence intervals are reported because at this size they would be wider than any effect.
Limitation 3
There is no competitor arm, on purpose
We cannot test other scribes fairly: no API access, no matched corpus, and no control over their configuration. Publishing a number for a competitor under those conditions would say more than the evidence supports, so we do not.
Limitation 4
The coverage loss is a real cost
Grounding removed 12 sentences across the corpus and strict key-fact coverage fell from 88.2% to 86.8%. All of that loss sits in one case, FB-008, where the judge stripped a whole sentence that bundled a fabricated denial with a true hedged finding. Whole-sentence removal is the residual defect. A faithfulness gain bought by dropping content is a regression, and the benchmark is designed to show it rather than hide it.
Limitation 5
The trap layer cannot tell whose symptom it is
In FB-007 the note correctly recorded that the spouse described her own headaches, and the traps still fired, because a pattern cannot do subject attribution. This is left unsuppressed on purpose: for wrong-person errors, the most dangerous class in a record, the instrument is biased toward false positives, because a rule good enough to silence this case would also hide a real wrong-patient attribution.
Limitation 6
Neither layer reliably scores a fabricated denial
An invented "denies X" can read as grounded to the judge when the transcript merely mentions X, and the trap detector treats a negated mention as correctly suppressed. The defect that drove the 2026-08-09 changes was found by reading notes, not by a metric moving. A number that cannot go up when this defect occurs is not evidence that the defect is absent.
Limitation 7
This measures the scribe, not Note Check
The benchmark measures the transcript-to-SOAP drafting path and its grounding verifier. It does not measure Note Check, the separate tool that reports on a pasted note from any scribe, and we publish no accuracy figure for Note Check. It also does not measure the chart-context drafting path, which has a different grounding stack.
What this benchmark permits us to say
On a small synthetic corpus judged by a model, the shipped grounding path measured a 0.0% hallucination rate against a 1.7% baseline, at a cost of 1.4 points of key-fact coverage, and caught 36 of 36 planted fabrications. That is the whole claim. It does not say the scribe is accurate in your clinic, it does not compare us to any other vendor, and it does not cover Note Check. The clinician who signs the note still reads the note.
The methodology document, including the parts of the instrument we distrust and why, is public: docs/ai/FAITHFULNESS_BENCHMARK.md on GitHub. The run report it cites is committed in the same repository.