We Need to Talk about Standard Splits

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2786 We need to talk about standard splits Kyle Gorman City University of New York kgorman@gc.cuny.edu Steven Bedrick Oregon Health & Science University bedricks@ohsu.edu Abstract It is standard practice in speech & language technology to rank systems according to per- formance on a test set held out for evalua- tion. However, few researchers apply statis- tical tests to determine whether differences in performance are likely to arise by chance, and few examine the stability of system ranking across multiple training-testing splits. We con- duct replication and reproduction experiments with nine part-of-speech taggers published be- tween 2000 and 2018, each of which reports state-of-the-art performance on a widely-used “standard split”. We fail to reliably repro- duce some rankings using randomly generated splits. We suggest that randomly generated splits should be used in system comparison. 1 Introduction Evaluation with a held-out test set is one of the few methodological practices shared across nearly all areas of speech and language processing. In this study we argue that one common instantiation of this procedure—evaluation with a standard split— is insufficient for system comparison, and propose an alternative based on multiple random splits. Standard split evaluation can be formalized as follows. Let G be a set of ground truth data, parti- tioned into a training set Gtrain, a development set Gdev and a test (evaluation) set Gtest. Let S be a system with arbitrary parameters and hyperparam- eters, and let M be an evaluation metric. Without loss of generality, we assume that M is a func- tion with domain G × S and that higher values of M indicate better performance. Furthermore, we assume a supervised training scenario in which the free parameters of S are set so as to maximize M(Gtrain, S), optionally tuning hyperparameters so as to maximize M(Gdev, S). Then, if S1 and S2 are competing systems so trained, we prefer S1 to S2 if and only if M(Gtest, S1) > M(Gtest, S2). 1.1 Hypothesis testing for system comparison One major concern with this procedure is that it treats M(Gtest, S1) and M(Gtest, S2) as exact quan- tities when they are better seen as estimates of ran- dom variables corresponding to true system perfor- mance. In fact many widely used evaluation met- rics, including accuracy and F-score, have known statistical distributions, allowing hypothesis test- ing to be used for system comparison. For instance, consider the comparison of two systems S1 and S2 trained and tuned to maximize accuracy. The difference in test accuracy, ˆδ = M(Gtest, S1) − M(Gtest, S2), can be thought of as estimate of some latent variable δ representing the true difference in system performance. While the distribution of ˆδ is not obvious, the probabil- ity that there is no population-level difference in system performance (i.e., δ = 0) can be com- puted indirectly using McNemar’s test (Gillick and Cox, 1989). Let …

Content truncated to 500 words