Time to adjust: Improving replicability in experimental psychology by adjustment for evident selective inference
Yoav Zeevi; Sofi Astashenko; Liad Mudrik; Yoav Benjamini · 2020 · arXiv
WASTE classifies this as Negative / Null Result Report · AI classification, approximate
The study found no significant effect — useful as a negative control or null benchmark for your own design.
Abstract (excerpt)
The field of psychological sciences has been grappling with the replicability crisis. Various issues have been identified as potential sources of this problem. We bring to light a potential source that has largely been overlooked and demonstrate its significant contribution to the problem: the practice of multiple comparisons. We analyzed 88 papers from the Reproducibility Project in Psychology and found that multiple results are commonly reported in a single paper, ranging from 4 to 730 (M=77.7), without multiple comparison adjustments. We retroactively applied such an adjustment using a hier
Excerpt shown for reference under fair use — read the full paper at the publisher.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results*
Negative / Null Result ReportThe harmonic mean p -value for combining dependent tests
Negative / Null Result ReportGeneralizability of heterogeneous treatment effect estimates across samples
Negative / Null Result ReportNumerical predictors of arithmetic success in grades 1–6
Negative / Null Result ReportMethods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics
Negative / Null Result ReportShould multiple imputation be the method of choice for handling missing data in randomized trials?
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: arXiv