Source, licence and claim boundary
Dialogues and notes: Omi Health medical-note-eval, MIT license, copyright (c) 2025 Omi Health B.V. All of them are synthetic. None of them is a record of a real person. No accuracy figure, recall figure or catch rate is stated for Note Check on this page, in the report behind it, or anywhere else we write.
Part of the scribe red-team series. The next installment is installment 02.
Krasyn ships a tool called Note Check. Paste a visit transcript and the note any AI scribe drafted from it, and it labels each sentence Supported, Unsupported, Contradicted, Scaffolding or Unverified against the transcript, raises three pure-code flags (a number the transcript never contained, a denial about a topic raised and never denied, content filled over an inaudible marker) and lists the facts the note left out. It never edits the note. The clinician reads the whole note and signs it.
We publish no accuracy figure for it, because it has not been measured against a clinician-adjudicated reference set. What we can do is run it on open data that someone else has already labelled, publish the counts and the disagreements, and say which side looks right on reading. This is the first installment.
The data
Omi Health publishes medical-note-eval under the MIT license: 300 synthetic primary-care dialogues, SOAP notes written for each by named frontier models, and labels from a panel of model judges. The 2025 benchmark ships per-note counts of unsupported claims from three cross-family judges. The 2026 benchmark publishes per-writer totals only.
We picked six dialogues with different presenting problems and took the notes from three 2025 writers (GPT-5, Claude Opus 4.5, Gemini 3 Pro Preview) and three 2026 writers (GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro Preview): 36 pairs. Sentence identifiers were removed from the transcripts and citation tags from the notes. Nothing else changed. Every pair went through the production Note Check service on engine notecheck-1, judge gpt-4o at temperature 0. Eight were also run in the live product with screenshots and a screen recording.
The counts
| Set | Pairs | Statements | Supported | Unsupported | Contradicted | Scaffolding | Rule flags | Omissions |
|---|---|---|---|---|---|---|---|---|
| 2025 notes (per-note labels exist) | 18 | 627 | 354 | 3 | 0 | 270 | 6 | 55 |
| 2026 notes (no per-note labels) | 18 | 774 | 422 | 0 | 0 | 352 | 10 | 30 |
For the 18 labelled notes, the corpus panel had counted a major unsupported claim (two or more judges agreeing) in 10 of them. Note Check’s judge labelled a statement Unsupported in one of those 10 and in none of the other nine. On the eight notes the panel had not flagged, Note Check also found nothing. The two tools agree on clean notes and disagree on almost every note the panel thought had a problem.
| Dialogue | Model | Corpus majors per judge | Note Check Unsupported and Contradicted | Note Check flags | Our reading |
|---|---|---|---|---|---|
| dialogue_0 | GPT-5 | 3, 3, 4 | 0 | 1 | Plan adds counselling nobody said. The flag is a rule misfire. |
| dialogue_0 | Claude Opus 4.5 | 2, 2, 2 | 0 | 0 | “Order laboratory studies” firms up “may need some labs”. |
| dialogue_2 | GPT-5 | 1, 1, 1 | 0 | 2 | An assessment in a visit that reached none. One flag right, one wrong. |
| dialogue_2 | Claude Opus 4.5 | 1, 1, 1 | 0 | 0 | The same invented assessment. |
| dialogue_24 | GPT-5 | 1, 3, 1 | 3 | 0 | Hit three “None” placeholders, not the type label. |
| dialogue_24 | Claude Opus 4.5 | 1, 1, 0 | 0 | 0 | “Type 2” is never stated in the visit. |
| dialogue_48 | Gemini 3 Pro Preview | 1, 1, 0 | 0 | 3 | All three flags wrong. The patient did deny. |
| dialogue_57 | GPT-5 | 2, 3, 4 | 0 | 0 | The word diabetes never appears in the transcript. |
| dialogue_57 | Claude Opus 4.5 | 3, 3, 1 | 0 | 0 | “Continue metformin” when the plan was to adjust medications. |
| dialogue_70 | GPT-5 | 1, 1, 0 | 0 | 0 | A differential and a symptom log nobody proposed. |
The eight rows where both sides found nothing, and the 18 notes from the 2026 writers, are in the full report.
The surprising one
In dialogue_48 the doctor asks the patient to confirm there is no history of blood clots, migraines with aura or uncontrolled blood pressure. The patient says, “Correct, none of those.” Four models (Gemini 3 Pro Preview, GPT-5.5, Claude Sonnet 4.6, Gemini 3.1 Pro Preview) each wrote some form of “Denies history of blood clots, migraines with aura, or uncontrolled blood pressure.” That is the right sentence.
Our pertinent-negative rule flagged all three items in all four notes at High severity: the topic was raised and, it claimed, nobody denied it. The rule found the question and did not recognise the answer as a denial. It is pure code, and it behaved identically every time. Twelve of the sixteen flags in this run are that one mistake. A thirteenth read “without sharp pain and intermittent foot swelling” as a denial of the swelling. Two flags were right: weight change recorded as negative in two notes when weight was asked about and never answered.
The quiet one
The larger miss is quieter. In dialogue_2 the visit is history only. It ends with the doctor saying the timeline is now clearer. Both the GPT-5 and Claude Opus 4.5 notes carry an assessment: cough “likely related to allergic etiology”. All three of the corpus judges counted that as a major unsupported claim. Note Check’s judge called it Supported.
The same pattern covers smoking cessation counselling in dialogue_0, “Type 2 diabetes mellitus” where the doctor said “your diabetes” or nothing at all, and a differential of migraine against transient ischemic attack in dialogue_70 that nobody voiced. Our judge prompt allows faithful clinical translation to count as Supported. On this corpus it let clinical conclusions through under that heading. Each is sensible, which is exactly why a signer would not catch them either.
What held
The fingerprint for a pair was the same in the command-line run and in the live product. Fresh judge calls were not identical: verdicts matched on seven of eight pairs run both ways, and the omission list changed on four of eight. This is why the product stores a report under its fingerprint instead of recomputing it, and why the omission list is a prompt to look rather than a count.
Limitations
- The dialogues are synthetic, short and clean. They are not the output of speech recognition, so they carry none of the disfluency and speaker-attribution damage of a real recording.
- Both sides of the comparison are model judges. No clinician adjudicated any label, on either side. The reading column is ours.
- The corpus labels are counts without statement text, so overlap between their finding and ours is inferred rather than matched statement to statement.
- The notes come from general-purpose models writing under a benchmark prompt, not from any commercial scribe product.
- Six dialogues is a small set. This is why the installment reports counts and no rate.
What happened next
The rule defect and the judge leniency were filed in our issue tracker as KRA-1916 before this installment published. Under the series contract, the next installment reran this exact set on the next engine version before touching any new data. That rerun, and what it improved and broke, is installment 02.
Send us a pair
Send a de-identified synthetic transcript and the note written from it, and we will run it in a later installment and publish the result whether or not it flatters us. Synthetic only. We do not accept real patient data, de-identified or otherwise, and we never will. Send us a pair.