Chains That See, Answers That Don't: A Multi-Aspect Evaluation Recipe for Forced Chain-of-Thought on Video-MME
Zhichao Fan; Yanhang Li; Zexin Zhuang · 2026 · arXiv
WASTE classifies this as Replication Failure · AI classification, approximate
A previously reported effect did not replicate here — verify it holds before you build on it.
Abstract (excerpt)
Forced chain-of-thought (CoT) is widely assumed to make vision-language models more reliable on video question answering. We propose a small three-probe evaluation recipe to test that assumption: paired accuracy across direct, CoT, answer-first, and no-video conditions; a counterfactual video-swap diagnostic over the CoT chains; and a four-rung visual-degradation ladder. Each probe is reported under both a strict and a permissive regex scorer, with multiplicity correction over a manuscript-declared primary family. Applied to Qwen2.5-VL on Video-MME subsets, the recipe returns a two-part findin
Excerpt shown for reference under fair use — read the full paper at the publisher.
About to run something similar?
Run an AI Precheck on your own design to catch failure modes like this one before you spend the time. Your first desk check is free.
Related failures
Leakage and the reproducibility crisis in machine-learning-based science
Negative / Null Result ReportDefining and detecting quantum speedup
Negative / Null Result ReportService robots in hotels: understanding the service quality perceptions of human-robot interaction
Negative / Null Result ReportBoosting methods for multi-class imbalanced data classification: an experimental review
Negative / Null Result ReportFINANCIAL DEVELOPMENT AND ECONOMIC GROWTH: A META‐ANALYSIS
Negative / Null Result ReportThe impact of site-specific digital histology signatures on deep learning model accuracy and bias
WASTE indexes this work — it does not host or republish it. Failure-type classification is automated and approximate.
Metadata source: arXiv