A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 918–932 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 918 A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing Rachel Bawden1 Biao Zhang1 Lisa Yankovskaya2 Andre T¨attar2 Matt Post3 1School of Informatics, University of Edinburgh, Scotland 2University of Tartu, Tartu, Estonia 3Johns Hopkins University, Baltimore, Maryland, USA Abstract We investigate a long-perceived shortcoming in the typical use of BLEU: its reliance on a single reference. Using modern neural para- phrasing techniques, we study whether auto- matically generating additional diverse refer- ences can provide better coverage of the space of valid translations and thereby improve its correlation with human judgments. Our ex- periments on the into-English language direc- tions of the WMT19 metrics task (at both the system and sentence level) show that us- ing paraphrased references does generally im- prove BLEU, and when it does, the more di- verse the better. However, we also show that better results could be achieved if those para- phrases were to specifically target the parts of the space most relevant to the MT outputs being evaluated. Moreover, the gains remain slight even when using human paraphrases elicited to maximize diversity, suggesting in- herent limitations to BLEU’s capacity to cor- rectly exploit multiple references. Surpris- ingly, we also find that adequacy appears to be less important, as shown by the high results of a strong sampling approach, which even beats human paraphrases when used with sentence- level BLEU.1 1 Introduction There is rarely a single correct way to translate a sentence; work attempting to encode the en- tire translation space of a sentence suggests there may be billions of valid translations (Dreyer and Marcu, 2012). Despite this, in machine translation (MT), system outputs are usually evaluated against a single reference. This especially affects MT’s dominant metric, BLEU (Papineni et al., 2002), since it is a surface metric that operates on ex- plicit n-gram overlap (see. (1) showing two ade- 1Our code and outputs are available at https:// github.com/rbawden/paraphrasing-bleu. quate MT outputs, one with only minimal overlap with the reference):2 (1) Ref: This did not bother anybody . MT1: This didn ’t bother anybody . MT2: Nobody was bothered by this . Almost since its creation, BLEU’s status as the dominant metric for MT evaluation has been chal- lenged (e.g., Callison-Burch et al. (2006), Mathur et al. (2020)). Such work typically uses only a sin- gle reference, however, which is a deficient form of the metric, since one of BLEU’s raisons d’ˆetre was to permit the use of multiple references, in a bid to represent “legitimate differences in word choice and word order.” Unfortunately, multiple references are rarely available due to the high cost and effort of producing them. One way to inex- pensively create them is with automatic paraphras- ing. This has been tried before (Zhou et al., 2006; Kauchak and Barzilay, 2006), but only recently have paraphrase systems become good enough to generate fluent, high quality sentential para- phrases (with …

Content truncated to 500 words