Vote’n’Rank: Revision of Benchmarking with Social Choice Theory

VOTE’N’RANK: Revision of Benchmarking with Social Choice Theory Mark Rofin1∗, Vladislav Mikhailov1,5∗, Mikhail Florinskiy1∗, Andrey Kravchenko2, Elena Tutubalina1,3, Tatiana Shavrina3,4,5, Daniel Karabekyan1, Ekaterina Artemova6† 1HSE University, 2University of Oxford 3Artificial Intelligence Research Institute, 4Institute of Linguistics, RAS, 5SaluteDevices 6Center for Information and Language Processing, LMU Munich Correspondence: Ekaterina.Artemova@lmu.de Abstract The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisa- tion capabilities from multiple perspectives. Al- though the paradigm is shifting towards more fine-grained evaluation across diverse tasks, the delicate question of how to aggregate the per- formances has received particular interest in the community. In general, benchmarks fol- low the unspoken utilitarian principles, where the systems are ranked based on their mean av- erage score over task-specific metrics. Such aggregation procedure has been viewed as a sub-optimal evaluation protocol, which may have created the illusion of progress. This pa- per proposes VOTE’N’RANK, a framework for ranking systems in multi-task benchmarks un- der the principles of the social choice theory. We demonstrate that our approach can be effi- ciently utilised to draw new insights on bench- marking in several ML sub-fields and identify the best-performing systems in research and de- velopment case studies. The VOTE’N’RANK’s procedures are more robust than the mean av- erage whilst being able to handle missing per- formance scores and specify conditions under which the system becomes the winner. 1 Introduction Benchmarking has evolved as a conventional prac- tice for accelerating the development of general- isable systems in different applied areas of ma- chine learning (ML). Benchmarks are typically designed as a collection of datasets, correspond- ing task-specific evaluation metrics, and a crite- rion for summarising the overall performance on the tasks (Ruder, 2021). The benchmark holders provide public leaderboards, which are utilised by ML researchers and practitioners for comparing ∗Equal contribution. †Work done while at HSE University. novel systems against one another, and, if applica- ble, human baselines, as well as selecting the best- performing ones for practical purposes. According to the benchmark sharing platform PAPERSWITH- CODE1, the community has put much effort into creating more than 10, 000 influential benchmarks in natural language processing (NLP), computer vision, and knowledge graphs, to name a few. Criticism of the benchmark pillars. The bench- mark methodological foundations have received wide criticism from academic and industrial com- munities (Bowman and Dahl, 2021). The criticism covers various aspects of benchmarking, raising concerns about the construct validity (Raji et al., 2021), fragility of the design and task choices (De- hghani et al., 2021), data leakage and annotation ar- tifacts (Elangovan et al., 2021), SoTA-chasing ten- dencies at the cost of large carbon footprints (Ben- der et al., 2021), and low reproducibility of the re- ported results (Belz et al., 2021), inter alia. Recom- mendations proposed in these studies are of utmost importance to benchmark holders, system users, and developers. However, little attention has been paid to a more nuanced methodological question: how to aggregate performance scores …

Content truncated to 500 words