What’s in a p-value in NLP?

Proceedings of the Eighteenth Conference on Computational Language Learning, pages 1–10, Baltimore, Maryland USA, June 26-27 2014. c⃝2014 Association for Computational Linguistics What’s in a p-value in NLP? Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy and Hector Martinez Center for Language Technology University of Copenhagen soegaard@hum.ku.dk Abstract In NLP, we need to document that our pro- posed methods perform significantly bet- ter with respect to standard metrics than previous approaches, typically by re- porting p-values obtained by rank- or randomization-based tests. We show that significance results following current re- search standards are unreliable and, in ad- dition, very sensitive to sample size, co- variates such as sentence length, as well as to the existence of multiple metrics. We estimate that under the assumption of per- fect metrics and unbiased data, we need a significance cut-off at ⇠0.0025 to reduce the risk of false positive results to <5%. Since in practice we often have consider- able selection bias and poor metrics, this, however, will not do alone. 1 Introduction In NLP, we try to improve upon state of the art language technologies, guided by experience and intuition, as well as error analysis from previous experiments, and research findings often consist in system comparisons showing that System A is bet- ter than System B. Effect size, i.e., one system’s improvements over another, can be seen as a random variable. If the random variable follows a known distribu- tion, e.g., a normal distribution, we can use para- metric tests to estimate whether System A is bet- ter than System B. If it follows a normal dis- tribution, we can use Student’s t-test, for exam- ple. Effect sizes in NLP are generally not nor- mally distributed or follow any of the other well- studied distributions (Yeh, 2000; Søgaard, 2013). The standard significance testing methods in NLP are therefore rank- or randomization-based non- parametric tests (Yeh, 2000; Riezler and Maxwell, 2005; Berg-Kirkpatrick et al., 2012). Specifi- cally, most system comparisons across words, sen- tences or documents use bootstrap tests (Efron and Tibshirani, 1993) or approximate randomization (Noreen, 1989), while studies that compare perfor- mance across data sets use rank-based tests such as Wilcoxon’s test. The question we wish to address here is: how likely is a research finding in NLP to be false? Naively, we would expect all reported findings to be true, but significance tests have their weak- nesses, and sometimes researchers are forced to violate test assumptions and basic statistical methodology, e.g., when there is no one estab- lished metric, when we can’t run our models on full-length sentences, or when data is biased. For example, one such well-known bias from the tag- ging and parsing literature is what we may refer to as the WSJ FALLACY. This is the false belief that performance on the test section of the Wall Street Journal (WSJ) part of the English Penn treebank is representative for performance on other texts in English. In other words, it is the belief that our samples are always representative. …

Content truncated to 500 words