With Little Power Comes Great Responsibility

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9263–9274, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9263 With Little Power Comes Great Responsibility Dallas Card1 Peter Henderson1 Urvashi Khandelwal1 Robin Jia1 Kyle Mahowald2 Dan Jurafsky1 1Stanford University, Stanford, CA 2University of California Santa Barbara, Santa Barbara, CA dcard@stanford.edu, phend@stanford.edu, urvashik@stanford.edu, robinjia@stanford.edu, mahowald@ucsb.edu, jurafsky@stanford.edu Abstract Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experi- ments make it more difficult to discern the difference between statistical noise and mean- ingful model improvements, and increase the chances of exaggerated findings. By meta- analyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that under- powered experiments are common in the NLP literature. In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately pow- ered. Similarly, based on reasonable assump- tions, we find that the most typical experimen- tal design for human rating studies will be un- derpowered to detect small model differences, of the sort that are frequently studied. For ma- chine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point. To improve the situation going forward, we give an overview of best practices for power analysis in NLP and release a series of note- books to assist with future power analyses.1 1 Introduction Despite its importance to empirical evaluation, rel- atively little attention has been paid to statistical power in NLP. In particular, if it is the case that typical experiments in NLP are underpowered, not only would we expect many meaningful improve- ments to go undetected, we would also expect many apparently significant differences to be exagger- ated (Gelman and Carlin, 2014). In this paper, we build on past work calling for greater rigor 1https://github.com/dallascard/NLP-power-analysis Figure 1: Cartoon example of statistical power in com- paring two models: 65% of all people in the population always prefer system B (left). A comparison using a sample of 100 people would be well-powered (middle): over 80% of such samples will show a significant dif- ference (plotted in red) from the null hypothesis that the models are equally good (dashed line). In samples of 25 people (right), far fewer tests will be significant (power ⇡ 30%). Note that the observed mean of sig- nificant findings (dotted line) slightly overestimates the true proportion that prefer system B when n = 100 and more severely overestimates it when n = 25. in evaluation (McCoy et al., 2019; Azer et al., 2020), including the need for careful hypothesis testing (Koehn, 2004; Berg-Kirkpatrick et al., 2012; Søgaard et al., 2014; Dror et al., 2018), and show why and how power matters to NLP, addressing challenges unique to this domain. Roughly …

Content truncated to 500 words