ParBLEU: Augmenting Metrics with Automatic Paraphrases for the WMT’20 Metrics Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 887–894 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 887 ParBLEU: Augmenting Metrics with Automatic Paraphrases for the WMT’20 Metrics Shared Task Rachel Bawden1 Biao Zhang1 Andre T¨attar2 Matt Post3 1School of Informatics, University of Edinburgh, Scotland 2University of Tartu, Tartu, Estonia 3Johns Hopkins University, Baltimore, Maryland, USA Abstract We describe parBLEU, parCHRF++, and parESIM, which augment baseline metrics with automatically generated paraphrases pro- duced by PRISM (Thompson and Post, 2020a), a multilingual neural machine translation sys- tem. We build on recent work studying how to improve BLEU by using diverse automat- ically paraphrased references (Bawden et al., 2020), extending experiments to the multilin- gual setting for the WMT2020 metrics shared task and for three base metrics. We compare their capacity to exploit up to 100 additional synthetic references. We find that gains are possible when using additional, automatically paraphrased references, although they are not systematic. However, segment-level correla- tions, particularly into English, are improved for all three metrics and even with higher num- bers of paraphrased references. 1 Introduction One of the major challenges faced when automati- cally evaluating machine translation (MT) outputs is that there are almost always multiple correct translations of a sentence, and an automatic met- ric should be able to reward them all. Some of the most widely used MT metrics, including BLEU (Pa- pineni et al., 2002) and CHRF++ (Popovi´c, 2015, 2017), rely on a surface-form comparison of MT outputs to a human-produced reference translation. Both metrics support the use of multiple references. However, even for metrics that support multiple ref- erences, human-produced references are expensive to produce and so are rarely available. To overcome this problem, metrics that do not rely on the surface form of reference translations have been developed. One example is ESIM (Chen et al., 2017; Mathur et al., 2019), which uses contextual embeddings with the aim of creating an abstract meaning rep- resentation of the reference, with the potential of covering all translations with the correct meaning. We explore an alternative way of increasing the capacity of MT metrics to reward multiple valid translations: create additional references by automatically paraphrasing the original reference. There have been previous efforts to provide some sort of paraphrase support, mostly concentrating on synonyms (Banerjee and Lavie, 2005; Kauchak and Barzilay, 2006; Denkowski and Lavie, 2014). However, we base our work on a more recent at- tempt to improve BLEU using diverse automatic parahrasing with high quality MT-style sentential paraphrasing (Bawden et al., 2020). We put this to the test in the WMT’20 metrics shared task by applying Bawden et al.’s (2020) ap- proach to three different metrics: BLEU, CHRF++ and ESIM. We compare the different metrics’ ca- pacity to exploit automatically generated multi- ple references. We choose to use diverse para- phrases produced using PRISM (Thompson and Post, 2020a), since they are available in multiple languages, including most languages of the WMT shared task. We find that gains in correlation are possible, …

Content truncated to 500 words