Beyond BLEU:Training Neural Machine Translation with Semantic Similarity

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344–4355 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4344 Beyond BLEU: Training Neural Machine Translation with Semantic Similarity John Wieting1, Taylor Berg-Kirkpatrick2, Kevin Gimpel3, and Graham Neubig1 1Carnegie Mellon University, Pittsburgh, PA, 15213, USA 2University of California San Diego, San Diego, CA, 92093, USA 3Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {jwieting,gneubig}@cs.cmu.edu, tberg@eng.ucsd.edu, kgimpel@ttic.edu Abstract While most neural machine translation (NMT) systems are still trained using maximum like- lihood estimation, recent work has demon- strated that optimizing systems to directly im- prove evaluation metrics such as BLEU can substantially improve final translation accu- racy. However, training with BLEU has some limitations: it doesn’t assign partial credit, it has a limited range of output values, and it can penalize semantically correct hypotheses if they differ lexically from the reference. In this paper, we introduce an alternative reward function for optimizing NMT systems that is based on recent work in semantic similarity. We evaluate on four disparate languages trans- lated to English, and find that training with our proposed metric results in better translations as evaluated by BLEU, semantic similarity, and human evaluation, and also that the optimiza- tion procedure converges faster. Analysis sug- gests that this is because the proposed metric is more conducive to optimization, assigning partial credit and providing more diversity in scores than BLEU.1 1 Introduction In neural machine translation (NMT) and other natural language generation tasks, it is common practice to improve likelihood-trained models by further tuning their parameters to explicitly max- imize an automatic metric of system accuracy – for example, BLEU (Papineni et al., 2002) or ME- TEOR (Denkowski and Lavie, 2014). Directly op- timizing accuracy metrics involves backpropagat- ing through discrete decoding decisions, and thus is typically accomplished with structured predic- tion techniques like reinforcement learning (Ran- zato et al., 2016), minimum risk training (Shen 1Code and data to replicate results are available at https://www.cs.cmu.edu/˜jwieting. et al., 2015), and other specialized methods (Wise- man and Rush, 2016). Generally, these methods work by repeatedly generating a translation under the current parameters (via decoding, sampling, or loss-augmented decoding), comparing the gener- ated translation to the reference, receiving some reward based on their similarity, and finally updat- ing model parameters to increase future rewards. In the vast majority of work, discriminative training has focused on optimizing BLEU (or its sentence-factored approximation). This is not sur- prising given that BLEU is the standard metric for system comparison at test time. However, BLEU is not without problems when used as a training criterion. Specifically, since BLEU is based on n-gram precision, it aggressively penalizes lexical differences even when candidates might be syn- onymous with or similar to the reference: if an n-gram does not exactly match a sub-sequence of the reference, it receives no credit. While the pes- simistic nature of BLEU differs from human judg- ments and is therefore problematic, it may, in prac- tice, pose a …

Content truncated to 500 words