Automatically Finding and Categorizing Replication Studies
Bob de Ruiter · 2023 · arXiv
WASTE classifies this as Negative / Null Result Report · AI classification, approximate
The study found no significant effect — useful as a negative control or null benchmark for your own design.
The finding, in one line
“In many fields of experimental science, papers that failed to replicate continue to be cited as a result of the poor discoverability of replication studies.”
Abstract (excerpt)
In many fields of experimental science, papers that failed to replicate continue to be cited as a result of the poor discoverability of replication studies. As a first step to creating a system that automatically finds replication studies for a given paper, 334 replication studies and 344 replicated studies were collected. Replication studies could be identified in the dataset based on text content at a higher rate than chance (AUROC = 0.886). Additionally, successful replication studies could be distinguished from failed replication studies at a higher rate than chance (AUROC = 0.664).
Excerpt shown for reference under fair use — read the full paper at the publisher.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
Leakage and the reproducibility crisis in machine-learning-based science
Negative / Null Result ReportDefining and detecting quantum speedup
Negative / Null Result ReportService robots in hotels: understanding the service quality perceptions of human-robot interaction
Negative / Null Result ReportBoosting methods for multi-class imbalanced data classification: an experimental review
Negative / Null Result ReportFINANCIAL DEVELOPMENT AND ECONOMIC GROWTH: A META‐ANALYSIS
Negative / Null Result ReportThe impact of site-specific digital histology signatures on deep learning model accuracy and bias
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: arXiv