Lost In Translation:
How A Patient Facing AI Loses The Guidelines, The Urgency And The Patient
By: Jocelyn Early MBA MHSc PA-C
Executive Summary
Doctronic is a patient-facing AI that offers diagnosis and a treatment plan in minutes, free, with a $39 doctor visit one click away. For a growing number of people it is the first, and sometimes only, clinical voice they will hear about a new symptom. We tested how it handles one of medicine's most consequential presentations: exertional arm pain that turns out to be cardiac.
We ran the same patient through Doctronic 30 times, varying only age (52 or 70) and sex. The patient's arm hurts when they stock shelves, the pain spreads to their chest and neck, and they get winded and sweaty, which they blame on working hard. Every guideline we consulted says this patient needs same-day evaluation for acute coronary syndrome, and a 911 call if symptoms are present now.
The central finding was not the one we went looking for. Doctronic is not one system. Users are silently placed into one of at least two conversational systems, and which one they get impacts whether a likely heart problem is treated as urgent.The gated system, which refuses to continue until the user enters age and sex assigned at birth (this is the gate), identified a possible cardiac cause in 16 of 20 runs, ranking it first in most. It then told the patient to go to the ED in only 5 of them. The other 11 received a routine plan with no time frame.
The ungated system (no age or sex question) recommended emergency or same-day in-person care in 9 of 10 runs. Telling it the patient was a 70-year-old woman did not change that.
Women fared worst inside the gated system. When the model had elicited the identical red-flag symptoms, 0 of 4 seventy-year-old women were sent anywhere urgently, against 3 of 5 seventy-year-old men.
One woman with new-onset symptoms received a report naming myocardial ischemia as the likely cause, followed by advice to see her primary care doctor.
Health Literacy remains a concern. All 28 scored patient reports exceeded the 6th–8th grade level recommended for patient materials. Gated reports were the hardest to read (median grade 22.0, against 16.5 for ungated reports without demographics), and none of the 19 told the patient what to do or when. Every ungated report did.
The failure is rarely the diagnosis. Doctronic usually names the heart. The failures are how it arrives at that diagnosis (does not follow accepted guidelines) and how it subsequently advises the patient to act (hint: it doesn’t). A report that says "angina" and "see your primary care doctor" in the same sentence reads to a patient as permission to wait.
These failures land hardest on the people with the fewest alternatives: those without a primary care doctor, those who cannot get an appointment for weeks, and those whose health literacy leaves them unable to tell a reassuring report from a correct one.
Medicine has a long, documented record of missing heart disease in women. A tool built from that record, deployed at scale and without clinical oversight, can repeat it faster than any clinician could
Read the full study
See how Doctronic performed with this single clinical scenario across four age and sex combinations. Evaluating for diagnostic ability and appropriate disposition.