Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2203–2212, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2203 Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing Piotr Szymański Department of Computational Intelligence Wrocław University of Science and Technology piotr.szymanski@pwr.edu.pl Kyle Gorman Graduate Program in Linguistics City University of New York kgorman@gc.cuny.edu Abstract Recent work raises concerns about the use of standard splits to compare natural language processing models. We propose a Bayesian statistical model comparison technique which uses k-fold cross-validation across multiple data sets to estimate the likelihood that one model will outperform the other, or that the two will produce practically equivalent results. We use this technique to rank six English part-of- speech taggers across two data sets and three evaluation metrics. 1 Introduction Gorman and Bedrick (2019) raise concerns about standard procedures used to compare speech and language processing models. They evaluate the performance of six English part-of-speech tag- gers using multiple randomly-generated training- testing splits; in some cases, they fail to reproduce previously-published system rankings established using a single “standard” split. They argue that point estimates of performance derived from a sin- gle training-testing splits are insufficient to estab- lish system rankings, even when null hypothesis significance testing is used for model comparison. In this study, we propose a technique for sys- tem comparison based on Bayesian statistical anal- ysis. Our approach, motivated in Section 2 and described in Section 3, allow us to infer the likeli- hood that one model will outperform the other, or even that both models’ performance will be prac- tically equivalent, something that is not possible with the frequentist statistical tests used by Gor- man and Bedrick. Our approach can also be ap- plied simultaneously across multiple data sets. As a proof of concept, in Sections 4–5 we apply the proposed method using the experimental setup of Gorman and Bedrick, and in Section 6, we use it to rank the six taggers, compare evaluation met- rics, and interpret the results. Our failure to repro- duce some of earlier reported results leads us to discuss the impact of repeating experiments, con- trasting performance in multiple measures and the advantages of comparing likelihoods in Section 6. We also discuss the notion of practical equivalence for speech and language technology. 2 Prior work Langley (1988) argues that machine learning should be viewed as an experimental science, and as such, machine learning technologies should be evaluated according to their performance on mul- tiple held-out data sets. Dietterich (1998) pro- poses a framework for comparing two supervised classifiers using a null hypothesis tests to deter- mine whether two classifiers have the same like- lihood of predicting a correct result. This study introduces several methods, including a paired t-test for k-fold cross-validation results. How- ever, he notes that the assumptions of normal- ity and independence may not be satisfied in all cases. Nadeau and Bengio (2000) propose a correlation-based correction for the Dietterich t- test procedure which adjusts for …

Content truncated to 500 words