Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5715–5725 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5715 Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses Erfan Sadeqi Azer1 Daniel Khashabi2∗ Ashish Sabharwal2 Dan Roth3 1Indiana University 2Allen Institute for Artificial Intelligence 3University of Pennsylvania esadeqia@indiana.edu {danielk,ashishs}@allenai.org danroth@cis.upenn.edu Abstract Empirical research in Natural Language Pro- cessing (NLP) has adopted a narrow set of prin- ciples for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues. While alternative pro- posals have been well-debated and adopted in other fields, they remain rarely discussed or used within the NLP community. We address this gap by contrasting various hypothesis as- sessment techniques, especially those not com- monly used in the field (such as evaluations based on Bayesian inference). Since these statistical techniques differ in the hypotheses they can support, we argue that practitioners should first decide their target hypothesis be- fore choosing an assessment method. This is crucial because common fallacies, misconcep- tions, and misinterpretation surrounding hy- pothesis assessment methods often stem from a discrepancy between what one would like to claim versus what the method used actu- ally assesses. Our survey reveals that these is- sues are omnipresent in the NLP research com- munity. As a step forward, we provide best practices and guidelines tailored towards NLP research, as well as an easy-to-use package called HyBayes for Bayesian assessment of hypotheses,1 complementing existing tools. 1 Introduction Empirical fields, such as Natural Language Pro- cessing (NLP), must follow scientific principles for assessing hypotheses and drawing conclusions from experiments. For instance, suppose we come across the results in Table 1, summarizing the accu- racy of two question-answering (QA) systems S1 and S2 on some datasets. What is the correct way to interpret this empirical observation in terms of ∗Work done while the second author was affiliated with the University of Pennsylvania. 1https://github.com/allenai/HyBayes System Description ARC-easy ARC-challenge ID #Correct Acc. #Correct Acc. S1 BERT 1721 72.4 566 48.3 S2 Reading Strategies 1637 68.9 496 42.3 Table 1: Performance of two systems (Devlin et al., 2019; Sun et al., 2018) on the ARC question-answering dataset (Clark et al., 2018). ARC-easy & ARC- challenge have 2376 & 1172 instances, respectively. Acc.: accuracy as a percentage. the superiority of one system over another? While S1 has higher accuracy than S2 in both cases, the gap is moderate and the datasets are of limited size. Can this apparent difference in performance be explained simply by random chance, or do we have sufficient evidence to conclude that S1 is in fact inherently different (in particular, inherently stronger) than S2 on these datasets? If the latter, can we quantify this gap in inherent strength while accounting for random fluctuation? Such fundamental questions arise in one form or another in every empirical NLP effort. Researchers often wish to draw conclusions such as: (Ca) I’m 95% confident that S1 and S2 are inherently different, in the sense …

Content truncated to 500 words