Human-Paraphrased References Improve Neural Machine Translation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1183–1192 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1183 Human-Paraphrased References Improve Neural Machine Translation Markus Freitag, George Foster, David Grangier, Colin Cherry Google Research {freitag,fosterg,grangier,colincherry}@google.com Abstract Automatic evaluation comparing candidate translations to human-generated paraphrases of reference translations has recently been pro- posed by Freitag et al. (2020). When used in place of original references, the paraphrased versions produce metric scores that correlate better with human judgment. This effect holds for a variety of different automatic metrics, and tends to favor natural formulations over more literal (translationese) ones. In this pa- per we compare the results of performing end- to-end system development using standard and paraphrased references. With state-of-the-art English-German NMT components, we show that tuning to paraphrased references produces a system that is significantly better accord- ing to human judgment, but 5 BLEU points worse when tested on standard references. Our work confirms the finding that paraphrased ref- erences yield metric scores that correlate better with human judgment, and demonstrates for the first time that using these scores for system development can lead to significant improve- ments. 1 Introduction Machine Translation (MT) has shown impressive progress in recent years. Neural architectures (Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) have greatly contributed to this im- provement, especially for languages with abundant training data (Bojar et al., 2016, 2018; Barrault et al., 2019). This progress creates novel challenges for the evaluation of machine translation, both for human (Toral, 2020; L¨aubli et al., 2020) and auto- mated evaluation protocols (Lo, 2019; Zhang et al., 2019). Both types of evaluation play an important role in machine translation (Koehn, 2010). While hu- man evaluations provide a gold standard evaluation, they involve a fair amount of careful and hence expensive work by human assessors. Cost there- fore limits the scale of their application. On the other hand, automated evaluations are much less expensive. They typically only involve human la- bor when collecting human reference translations and can hence be run at scale to compare a wide range of systems or validate design decisions. The value of automatic evaluations therefore resides in their capacity to be used as a proxy for human evaluations for large scale comparisons and system development. The recent progress in MT has raised concerns about whether automated evaluation methodolo- gies reliably reflect human ratings in high accuracy ranges. In particular, it has been observed that the best systems according to humans might fare less well with automated metrics (Barrault et al., 2019). Most metrics such as BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) measure overlap between a system output and a human reference translation. More refined ways to compute such overlap have consequently been proposed (Baner- jee and Lavie, 2005; Lo, 2019; Zhang et al., 2019). Orthogonal to the work of building improved metrics, Freitag et al. (2020) hypothesized that hu- man references are also an important factor in the …

Content truncated to 500 words