BLEURT: Learning Robust Metrics for Text Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7881 BLEURT: Learning Robust Metrics for Text Generation Thibault Sellam Dipanjan Das Google ResearchAnkur P. Parikh New York, NY {tsellam, dipanjand, aparikh }@google.com Abstract Text generation has made significant advances in the last few years. Yet, evaluation met- rics have lagged behind, as the most popu- lar choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation met- ric based on BERT that can model human judgments with a few thousand possibly bi- ased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of- the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior re- sults even when the training data is scarce and out-of-distribution. 1 Introduction In the last few years, research in natural text generation (NLG) has made significant progress, driven largely by the neural encoder-decoder paradigm (Sutskever et al., 2014; Bahdanau et al., 2015) which can tackle a wide array of tasks including translation (Koehn, 2009), summariza- tion (Mani, 1999; Chopra et al., 2016), structured- data-to-text generation (McKeown, 1992; Kukich, 1983; Wiseman et al., 2017) dialog (Smith and Hipp, 1994; Vinyals and Le, 2015) and image cap- tioning (Fang et al., 2015). However, progress is increasingly impeded by the shortcomings of ex- isting metrics (Wiseman et al., 2017; Ma et al., 2019; Tian et al., 2019). Human evaluation is often the best indicator of the quality of a system. However, design- ing crowd sourcing experiments is an expensive and high-latency process, which does not easily fit in a daily model development pipeline. There- fore, NLG researchers commonly use automatic evaluation metrics, which provide an acceptable proxy for quality and are very cheap to compute. This paper investigates sentence-level, reference- based metrics, which describe the extent to which a candidate sentence is similar to a reference one. The exact definition of similarity may range from string overlap to logical entailment. The first generation of metrics relied on hand- crafted rules that measure the surface similarity between the sentences. To illustrate, BLEU (Pa- pineni et al., 2002) and ROUGE (Lin, 2004), two popular metrics, rely on N-gram overlap. Because those metrics are only sensitive to lexical vari- ation, they cannot appropriately reward seman- tic or syntactic variations of a given reference. Thus, they have been repeatedly shown to cor- relate poorly with human judgment, in particular when all the systems to compare have a similar level of accuracy (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018). Increasingly, NLG researchers have addressed those problems by injecting learned components in their metrics. To illustrate, consider the WMT Metrics Shared Task, an annual benchmark in which translation metrics are compared on their …

Content truncated to 500 words