Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8761 - 8781 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons Akash Kumar Mohankumar∗ Microsoft Bangalore, India makashkumar@microsoft.com Mitesh M. Khapra Indian Institute of Technology Madras RBCDSAI, IIT Madras miteshk@cse.iitm.ac.in Abstract Recent studies have shown the advantages of evaluating NLG systems using pairwise comparisons as opposed to direct assessment. Given k systems, a naive approach for identi- fying the top-ranked system would be to uni- formly obtain pairwise comparisons from all �k 2 � pairs of systems. However, this can be very expensive as the number of human anno- tations required would grow quadratically with k. In this work, we introduce Active Evalu- ation, a framework to efficiently identify the top-ranked system by actively choosing sys- tem pairs for comparison using dueling ban- dit algorithms. We perform extensive experi- ments with 13 dueling bandits algorithms on 13 NLG evaluation datasets spanning 5 tasks and show that the number of human annota- tions can be reduced by 80%. To further reduce the number of human annotations, we propose model-based dueling bandit algorithms which combine automatic evaluation metrics with hu- man evaluations. Specifically, we eliminate sub-optimal systems even before the human annotation process and perform human evalu- ations only on test examples where the auto- matic metric is highly uncertain. This reduces the number of human annotations required fur- ther by 89%. In effect, we show that identi- fying the top-ranked system requires only a few hundred human annotations, which grow linearly with k. Lastly, we provide practical recommendations and best practices to iden- tify the top-ranked system efficiently. Our code has been made publicly available at https: //github.com/akashkm99/duelnlg 1 Introduction In the last few years, the field of NLG has made rapid progress with the advent of large-scale mod- els trained on massive amounts of data (Vaswani et al., 2017; Xue et al., 2020; Liu et al., 2020; Brown et al., 2020). However, evaluation of NLG ∗* Work done at Indian Institute of Technology Madras systems continues to be a challenge. On the one hand, we have automatic evaluation metrics which are easy to compute but unreliable. In particular, many studies have shown that they do not correlate well with human judgments (Novikova et al., 2017; Elliott and Keller, 2014; Sai et al., 2019, 2020a,b). On the other hand, we have human evaluations, which are relatively more reliable but tedious, ex- pensive, and time-consuming. Further, recent stud- ies have highlighted some limitations of human evaluations that involve direct assessment on an absolute scale, e.g., Likert scale. Specifically, hu- man evaluations using direct assessment have been shown to suffer from annotator bias, high vari- ance and sequence effects where the annotation of one item is influenced by preceding items (Kulikov et al., 2019; Sudoh et al., 2021; Liang et al., 2020; See et al., 2019; Mathur et al., 2017). In this work, we focus on reducing …

Content truncated to 500 words