Better than Average: Paired Evaluation of NLP systems

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2301–2315 August 1–6, 2021. ©2021 Association for Computational Linguistics 2301 Better than Average: Paired Evaluation of NLP Systems Maxime Peyrard∗, Wei Zhao†, Steffen Eger†, Robert West∗ ∗EPFL, Switzerland †Technische Universität Darmstadt, Germany {maxime.peyrard,robert.west}@epfl.ch {zhao,eger}@aiphes.tu-darmstadt.de Abstract Evaluation in NLP is usually done by com- paring the scores of competing systems inde- pendently averaged over a common set of test instances. In this work, we question the use of averages for aggregating evaluation scores into a final number used to decide which sys- tem is best, since the average, as well as alter- natives such as the median, ignores the pair- ing arising from the fact that systems are eval- uated on the same test instances. We illus- trate the importance of taking the instance- level pairing of evaluation scores into account and demonstrate, both theoretically and em- pirically, the advantages of aggregation meth- ods based on pairwise comparisons, such as the Bradley–Terry (BT) model, a mechanism based on the estimated probability that a given system scores better than another on the test set. By re-evaluating 296 real NLP evalua- tion setups across four tasks and 18 evaluation metrics, we show that the choice of aggrega- tion mechanism matters and yields different conclusions as to which systems are state of the art in about 30% of the setups. To facil- itate the adoption of pairwise evaluation, we release a practical tool for performing the full analysis of evaluation scores with the mean, median, BT, and two variants of BT (Elo and TrueSkill), alongside functionality for appro- priate statistical testing. 1 Introduction Research is driven by evaluation results, with at- tention and resources being focused on methods identified as state of the art (SotA). The proper design of evaluation methodology is thus crucial to ensure progress in the field. In NLP, evalua- tion usually consists in comparing the averaged scores of competing systems over a common set of test instances. Indeed, averaging scores inde- pendently for each system and declaring the one with the highest average to be best is particularly 1 2 3 4 5 Index of test instances 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Evaluation scores of systems (higher is better) System A Mean (same for all) System B Medians System C Figure 1: Motivating example (synthetic data). Eval- uation scores of systems A, B, and C for five test in- stances. All systems have the same mean. C is better than A on all instances but one, so BT declares C > A Also, B is better than A on all instances but one, so BT declares B > A, whereas the median of A is greater, and the means are the same. Overall, mean and median fail to capture the complex instance-level pairing. simple, well understood, and mirrors the expected risk minimization paradigm used to train systems. Here, we critically assess the specific choice of …

Content truncated to 500 words