An Empirical Investigation of Statistical Significance in NLP

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics An Empirical Investigation of Statistical Significance in NLP Taylor Berg-Kirkpatrick David Burkett Dan Klein Computer Science Division University of California at Berkeley {tberg, dburkett, klein}@cs.berkeley.edu Abstract We investigate two aspects of the empirical behavior of paired significance tests for NLP systems. First, when one system appears to outperform another, how does significance level relate in practice to the magnitude of the gain, to the size of the test set, to the similar- ity of the systems, and so on? Is it true that for each task there is a gain which roughly implies significance? We explore these issues across a range of NLP tasks using both large collec- tions of past systems’ outputs and variants of single systems. Next, once significance lev- els are computed, how well does the standard i.i.d. notion of significance hold up in practical settings where future distributions are neither independent nor identically distributed, such as across domains? We explore this question using a range of test set variations for con- stituency parsing. 1 Introduction It is, or at least should be, nearly universal that NLP evaluations include statistical significance tests to validate metric gains. As important as significance testing is, relatively few papers have empirically in- vestigated its practical properties. Those that do focus on single tasks (Koehn, 2004; Zhang et al., 2004) or on the comparison of alternative hypothe- sis tests (Gillick and Cox, 1989; Yeh, 2000; Bisani and Ney, 2004; Riezler and Maxwell, 2005). In this paper, we investigate two aspects of the empirical behavior of paired significance tests for NLP systems. For example, all else equal, larger metric gains will tend to be more significant. How- ever, what does this relationship look like and how reliable is it? What should be made of the conven- tional wisdom that often springs up that a certain metric gain is roughly the point of significance for a given task (e.g. 0.4 F1 in parsing or 0.5 BLEU in machine translation)? We show that, with heavy caveats, there are such thresholds, though we also discuss the hazards in their use. In particular, many other factors contribute to the significance level, and we investigate several of them. For example, what is the effect of the similarity between the two sys- tems? Here, we show that more similar systems tend to achieve significance with smaller metric gains, re- flecting the fact that their outputs are more corre- lated. What about the size of the test set? For ex- ample, in designing a shared task it is important to know how large the test set must be in order for sig- nificance tests to be sensitive to small gains in the performance metric. Here, we show that test size plays the largest role in determining discrimination ability, but that we get diminishing returns. For ex- ample, doubling the …

Content truncated to 500 words