Showing Your Work Doesn’t Always Work

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2766–2772 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2766 Showing Your Work Doesn’t Always Work Raphael Tang,1,2 Jaejun Lee,1 Ji Xin,1,2 Xinyu Liu,1 Yaoliang Yu,1,2 and Jimmy Lin1,2 1David R. Cheriton School of Computer Science, University of Waterloo 2Vector Institute for Artificial Intelligence firstname.lastname@uwaterloo.ca Abstract In natural language processing, a recently pop- ular line of work explores how to best report the experimental results of neural networks. One exemplar publication, titled “Show Your Work: Improved Reporting of Experimental Results” (Dodge et al., 2019), advocates for reporting the expected validation effectiveness of the best-tuned model, with respect to the computational budget. In the present work, we critically examine this paper. As far as statistical generalizability is concerned, we find unspoken pitfalls and caveats with this approach. We analytically show that their estimator is biased and uses error-prone as- sumptions. We find that the estimator favors negative errors and yields poor bootstrapped confidence intervals. We derive an unbiased alternative and bolster our claims with em- pirical evidence from statistical simulation. Our codebase is at https://github.com/ castorini/meanmax. 1 Introduction Questionable answers and irreproducible results represent a formidable beast in natural language processing research. Worryingly, countless exper- imental papers lack empirical rigor, disregarding necessities such as the reporting of statistical signif- icance tests (Dror et al., 2018) and computational environments (Crane, 2018). As Forde and Pa- ganini (2019) concisely lament, explorimentation, the act of tinkering with metaparameters and pray- ing for success, while helpful in brainstorming, does not constitute a rigorous scientific effort. Against the crashing wave of explorimentation, though, a few brave souls have resisted the urge to feed the beast. Reimers and Gurevych (2017) argue for the reporting of neural network score distribu- tions. Gorman and Bedrick (2019) demonstrate that deterministic dataset splits yield less robust re- sults than random ones for neural networks. Dodge et al. (2019) advocate for reporting the expected validation quality as a function of the computation budget used for hyperparameter tuning, which is paramount to robust conclusions. But carefully tread we must. Papers that advo- cate for scientific rigor must be held to the very same standards that they espouse, lest they birth a new beast altogether. In this work, we critically examine one such paper from Dodge et al. (2019). We acknowledge the validity of their technical con- tribution, but we find several notable caveats, as far as statistical generalizability is concerned. An- alytically, we show that their estimator is nega- tively biased and uses assumptions that are subject to large errors. Based on our theoretical results, we hypothesize that this estimator strongly prefers underestimates to overestimates and yields poor confidence intervals with the common bootstrap method (Efron, 1982). Our main contributions are as follows: First, we prove that their estimator is biased under weak con- ditions and provide an unbiased solution. Second, we show that one of their core approximations often contains large errors, leading to poorly controlled bootstrapped …

Content truncated to 500 words