A comparative analysis of DeepSeek R1, DeepSeek-R1-Lite, OpenAi o1 Pro, and Grok 3 performance on ophthalmology board-style questions
Ryan Shean; Tathya Shah; Aditya Pandiarajan; Alan Tang; Kyle Bolo; Văn Thành Nguyễn; Benjamin Y. Xu · 2025 · Scientific Reports
WASTE classifies this as Negative / Null Result Report · AI classification, approximate
The study found no significant effect — useful as a negative control or null benchmark for your own design.
Abstract
The ability of large language models (LLMs) to accurately answer medical board-style questions reflects their potential to benefit medical education and real-time clinical decision-making. With the recent advance to reasoning models, the latest LLMs excel at addressing complex problems in benchmark math and science tests. This study assessed the performance of first-generation reasoning models-DeepSeek's R1 and R1-Lite, OpenAI's o1 Pro, and Grok 3-on 493 ophthalmology questions sourced from the StatPearls and EyeQuiz question banks. o1 Pro achieved the highest overall accuracy (83.4%), signifi
Abstract by Ryan Shean; Tathya Shah; Aditya Pandiarajan; Alan Tang; Kyle Bolo; Văn Thành Nguyễn; Benjamin Y. Xu, Scientific Reports (2025) — licensed CC BY 4.0.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
A Randomized Trial of Intraarterial Treatment for Acute Ischemic Stroke
Negative / Null Result ReportDuodenal Infusion of Donor Feces for Recurrent Clostridium difficile
Negative / Null Result ReportStenting versus Endarterectomy for Treatment of Carotid-Artery Stenosis
Negative / Null Result ReportEffects of Combination Lipid Therapy in Type 2 Diabetes Mellitus
Replication FailureA Randomized Trial of Bevacizumab for Newly Diagnosed Glioblastoma
Negative / Null Result ReportSpironolactone for Heart Failure with Preserved Ejection Fraction
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: OpenAlex · DOI 10.1038/s41598-025-08601-2