Please, Don’t Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5956–5962 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5956 Please, Don’t Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status Yves Bestgen Laboratoire d’analyse statistique des textes Universit´e catholique de Louvain Place Cardinal Mercier, 10 1348 Louvain-la-Neuve, Belgium yves.bestgen@uclouvain.be Abstract This paper argues for the widest possible use of bootstrap confidence intervals for comparing NLP system performances instead of the state-of-the-art status (SOTA) and statistical significance testing. Their main benefits are to draw attention to the difference in performance between two systems and to help assessing the degree of superiority of one system over another. Two cases studies, one comparing several systems and the other based on a K-fold cross-validation procedure, illustrate these benefits. Keywords: NLP system evaluation, statistical significance testing, bootstrap confidence interval, Fisher-Pitman test, SOTA, resampling approach 1. Introduction With the explosion of NLP studies, more and more systems are developed and evaluated using more and more shared tasks and freely available datasets. Com- paring systems has therefore become an essential ques- tion (Berg-Kirkpatrick et al., 2012; Dodge et al., 2019). Very often, it boils down to ranking systems by perfor- mance like in leaderboards or determining whether the new system scores better than the previous better one, the widely discussed ”glorification of benchmarks” and state-of-the-art status (SOTA) (Dehghani et al., 2021). The major advantage of this ranking criterion is that it only requires the performance level of each system. Using simple ranking has been criticized because the superiority of one system over another may result from chance alone rather than from any real difference (Koehn, 2004). The answer is obviously statistical sig- nificance testing, whose function is to reject the null hypothesis according to which there is no difference in the population to which the evaluation set belong. Un- like SOTA, it requires access to the predictions of the models to be compared. Yet, it has been widely em- phasized, but more often in other fields than NLP (Co- hen, 1994; Ioannidis, 2019; Kim and Robinson, 2019; Rao and Lovric, 2016; Schmidt and Hunter, 1997), that null-hypothesis significance testing presents many problems. When comparing NLP systems, one of the most important issues is that a statistical test leads to di- chotomizing the results of a study in significant vs. not significant, while statistically significant does not mean important. For most statistical tests, any difference, however small it may be and therefore as unimportant as it is, can be declared significant with enough data. This is for example the case when a chi-square test is (wrongly) used to compare the frequency of words or lexical bundles in large corpora: the smallest differ- ence in frequency gives rise to a statistically highly sig- nificant result (Cohen, 1994; Bestgen, 2014; Bestgen, 2020a). For these reasons, statisticians and researchers in many disciplines have strongly recommended reporting an effect size measure to help evaluating the importance of the result …

Content truncated to 500 words