Computerized diagnostic decision support systems – a comparative performance study of Isabel Pro vs. ChatGPT4
Joe M. Bridges · 2024 · Diagnosis
WASTE classifies this as Negative / Null Result Report · AI classification, approximate
The study found no significant effect — useful as a negative control or null benchmark for your own design.
Abstract
OBJECTIVES: Validate the diagnostic accuracy of the Artificial Intelligence Large Language Model ChatGPT4 by comparing diagnosis lists produced by ChatGPT4 to Isabel Pro. METHODS: This study used 201 cases, comparing ChatGPT4 to Isabel Pro. Systems inputs were identical. Mean Reciprocal Rank (MRR) compares the correct diagnosis's rank between systems. Isabel Pro ranks by the frequency with which the symptoms appear in the reference dataset. The mechanism ChatGPT4 uses to rank the diagnoses is unknown. A Wilcoxon Signed Rank Sum test failed to reject the null hypothesis. RESULTS: Both systems p
Abstract by Joe M. Bridges, Diagnosis (2024) — licensed CC BY 4.0.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
A Randomized Trial of Intraarterial Treatment for Acute Ischemic Stroke
Negative / Null Result ReportDuodenal Infusion of Donor Feces for Recurrent Clostridium difficile
Negative / Null Result ReportStenting versus Endarterectomy for Treatment of Carotid-Artery Stenosis
Negative / Null Result ReportEffects of Combination Lipid Therapy in Type 2 Diabetes Mellitus
Replication FailureA Randomized Trial of Bevacizumab for Newly Diagnosed Glioblastoma
Negative / Null Result ReportSpironolactone for Heart Failure with Preserved Ejection Fraction
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: OpenAlex · DOI 10.1515/dx-2024-0033