Reliability of LLMs as medical assistants for the general public: a randomized preregistered study
Andrew M. Bean; Rebecca Payne; Guy Parsons; Hannah Rose Kirk; Juan Ciro; Rafael Mosquera-Gómez; Sara Hincapié M; Aruna S. Ekanayaka · 2026 · Nature Medicine
WASTE classifies this as Negative / Null Result Report · AI classification, approximate
The study found no significant effect — useful as a negative control or null benchmark for your own design.
Abstract
Global healthcare providers are exploring the use of large language models (LLMs) to provide medical advice to the public. LLMs now achieve nearly perfect scores on medical licensing exams, but this does not necessarily translate to accurate performance in real-world settings. We tested whether LLMs can assist members of the public in identifying underlying conditions and choosing a course of action (disposition) in ten medical scenarios in a controlled study with 1,298 participants. Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+) or a source
Abstract by Andrew M. Bean; Rebecca Payne; Guy Parsons; Hannah Rose Kirk; Juan Ciro; Rafael Mosquera-Gómez; Sara Hincapié M; Aruna S. Ekanayaka, Nature Medicine (2026) — licensed CC BY 4.0.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
A Randomized Trial of Intraarterial Treatment for Acute Ischemic Stroke
Negative / Null Result ReportDuodenal Infusion of Donor Feces for Recurrent Clostridium difficile
Negative / Null Result ReportStenting versus Endarterectomy for Treatment of Carotid-Artery Stenosis
Negative / Null Result ReportEffects of Combination Lipid Therapy in Type 2 Diabetes Mellitus
Replication FailureA Randomized Trial of Bevacizumab for Newly Diagnosed Glioblastoma
Negative / Null Result ReportSpironolactone for Heart Failure with Preserved Ejection Fraction
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: OpenAlex · DOI 10.1038/s41591-025-04074-y