Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets Rotem Dror Gili Baumer Faculty of Industrial Engineering and Management, Technion, IIT {rtmdrr@campus|sgbaumer@campus|marinabo|roiri}.technion.ac.il Marina Bogomolov Roi Reichart Abstract With the ever growing amount of textual data from a large variety of languages, domains, and genres, it has become standard to evalu- ate NLP algorithms on multiple datasets in or- der to ensure a consistent performance across heterogeneous setups. However, such multi- ple comparisons pose significant challenges to traditional statistical analysis methods in NLP and can lead to erroneous conclusions. In this paper we propose a Replicability Analy- sis framework for a statistically sound anal- ysis of multiple comparisons between algo- rithms for NLP tasks. We discuss the theo- retical advantages of this framework over the current, statistically unjustified, practice in the NLP literature, and demonstrate its empirical value across four applications: multi-domain dependency parsing, multilingual POS tag- ging, cross-domain sentiment classification and word similarity prediction. 1 1 Introduction The field of Natural Language Processing (NLP) is going through the data revolution. With the persis- tent increase of the heterogeneous web, for the first time in human history, written language from mul- tiple languages, domains, and genres is now abun- dant. Naturally, the expectations from NLP algo- rithms also grow and evaluating a new algorithm on as many languages, domains, and genres as possible is becoming a de-facto standard. 1Our code is at: https://github.com/rtmdrr/replicability- analysis-NLP . For example, the phrase structure parsers of Char- niak (2000) and Collins (2003) were mostly evalu- ated on the Wall Street Journal Penn Treebank (Mar- cus et al., 1993), consisting of written, edited En- glish text of economic news. In contrast, modern dependency parsers are expected to excel on the 19 languages of the CoNLL 2006-2007 shared tasks on multilingual dependency parsing (Buchholz and Marsi, 2006; Nilsson et al., 2007), and additional challenges, such as the shared task on parsing mul- tiple English Web domains (Petrov and McDonald, 2012), are continuously proposed. Despite the growing number of evaluation tasks, the analysis toolbox employed by NLP researchers has remained quite stable. Indeed, in most exper- imental NLP papers, several algorithms are com- pared on a number of datasets where the perfor- mance of each algorithm is reported together with per-dataset statistical significance figures. However, with the growing number of evaluation datasets, it becomes more challenging to draw comprehensive conclusions from such comparisons. This is because although the probability of drawing an erroneous conclusion from a single comparison is small, with multiple comparisons the probability of making one or more false claims may be very high. The goal of this paper is to provide the NLP com- munity with a statistical analysis framework, which we term Replicability Analysis, which will allow us to draw statistically sound conclusions in evalua- tion setups that involve multiple comparisons. The classical goal of replicability analysis is to examine the consistency of findings across studies in order to address the basic dogma of science, that a find- 471 Transactions of the …

Content truncated to 500 words