Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3540 - 3557 July 10-15, 2022 ©2022 Association for Computational Linguistics Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand Jungo Kasai ∗ Keisuke Sakaguchi Ronan Le Bras Lavinia Dunagan Jacob Morrison Alexander R. Fabbri Yejin Choi Noah A. Smith Paul G. Allen School of Computer Science & Engineering, University of Washington Allen Institute for AI Department of Linguistics, University of Washington Salesforce Research billboard.nlp@gmail.com Abstract Natural language processing researchers have identified limitations of evaluation methodol- ogy for generation tasks, with new questions raised about the validity of automatic metrics and of crowdworker judgments. Meanwhile, efforts to improve generation models tend to depend on simple n-gram overlap metrics (e.g., BLEU, ROUGE). We argue that new advances on models and metrics should each more di- rectly benefit and inform the other. We there- fore propose a generalization of leaderboards, bidimensional leaderboards (BILLBOARDs), that simultaneously tracks progress in language generation models and metrics for their eval- uation. Unlike conventional unidimensional leaderboards that sort submitted systems by predetermined metrics, a BILLBOARD accepts both generators and evaluation metrics as com- peting entries. A BILLBOARD automatically creates an ensemble metric that selects and lin- early combines a few metrics based on a global analysis across generators. Further, metrics are ranked based on their correlation with hu- man judgments. We release four BILLBOARDs for machine translation, summarization, and image captioning.1 We demonstrate that a lin- ear ensemble of a few diverse metrics some- times substantially outperforms existing met- rics in isolation. Our mixed-effects model anal- ysis shows that most automatic metrics, espe- cially the reference-based ones, overrate ma- chine over human generation, demonstrating the importance of updating metrics as gener- ation models become stronger (and perhaps more similar to humans) in the future. 1 Introduction Recent modeling advances have led to improved natural language generation in applications such as machine translation and summarization (Ng et al., ∗Work was done during an internship at AI2. 1https://nlp.cs.washington.edu/ billboard/. metric.py Generator Developer Metric Developer output.txt Figure 1: Bidimensional leaderboard (BILLBOARD). When a generator developer submits output text (output.txt), BILLBOARD computes all metric scores. When a metric developer submits an executable program (e.g., metric.py), BILLBOARD computes correlation with the human judgments, updates the en- semble metric (§2.2), and measures how much the met- ric overrates machines (§2.3). 2019; Raffel et al., 2020; Brown et al., 2020, in- ter alia). This progress is typically measured with automatic scores, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), executed by model- ing researchers themselves. These metrics allow for fast, inexpensive development cycles. They were adopted based on reported correlations with human judgments at the time the metrics were introduced, but it has since been established that the correspon- dence can collapse when models of different types are compared (Callison-Burch et al., 2006) or mod- els become increasingly powerful (Ma et al., 2019; Edunov et al., 2020). Meanwhile, many evaluation metrics …

Content truncated to 500 words