A Mixed Learning Objective for Neural Machine Translation

A Mixed Learning Objective for Neural Machine Translation Wenjie Lu, Leiying Zhou, Gongshen Liu* and Quanhai Zhang* School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China {jonsey,zhouleiying,lgshen,qhzhang}@sjtu.edu.cn Abstract Evaluation discrepancy and overcorrection phenomenon are two common problems in neural ma- chine translation (NMT). NMT models are generally trained with word-level learning objective, but evaluated by sentence-level metrics. Moreover, the cross-entropy loss function discourages model to generate synonymous predictions and overcorrect them to ground truth words. To address these two drawbacks, we adopt multi-task learning and propose a mixed learning ob- jective (MLO) which combines the strength of word-level and sentence-level evaluation without modifying model structure. At word-level, it calculates semantic similarity between predicted and ground truth words. At sentence-level, it computes probabilistic n-gram matching scores of generated translations. We also combine a loss-sensitive scheduled sampling decoding strategy with MLO to explore its extensibility. Experimental results on IWSLT 2016 German-English and WMT 2019 English-Chinese datasets demonstrate that our methodology can significantly promote translation quality. The ablation study shows that both word-level and sentence-level learning objective can improve BLEU scores. Furthermore, MLO is consistent with state-of-the- art scheduled sampling methods and can achieve further promotion. 1 Introduction In recent years, tremendous progresses have been made in the field of neural machine translation (NMT) (Sutskever et al., 2014; Luong et al., 2015). A typical NMT model can be formulated as an encoder- decoder-attention architecture (Forcada and ˜Neco, 1997; Bahdanau et al., 2015) with maximum likeli- hood estimation (MLE) objective. Given sufficient parallel corpora, NMT models can achieve promising performance. Despite much success, NMT models suffer from two major drawbacks. First, there exists a discrep- ancy between training objectives and evaluation metrics. Most NMT models are trained with MLE objective under the teacher forcing algorithm (Williams and Zipser, 1989), i.e., models calculate and accumulate cross-entropy loss between predicted and ground truth sentences word by word. A lower cross-entropy value means the predictions are closer to ground truth at word level. Model parameters are updated through backpropagation to minimize the value of loss function. However, translation quality is measured by sentence-level metrics such as BLEU (Shterionov et al., 2017), ROUGE (Lin, 2004), etc. This way of word-level optimization mismatches sentence-level evaluation metrics, which may mislead the promotion of translation performance. Second, the MLE training objective brings about overcor- rection phenomenon (Zhang et al., 2019). To be specific, models are trained to learn absolutely correct translations and overcorrect synonymous words and phrases. Once the model predicts a word different from the ground truth word, the cross-entropy loss will immediately punish it and lead the model to the correct direction. As for synonymous phrases, it may result in translating wrong phrases while reducing the diversity of translation. In this paper, we present a novel approach to solve the above problems. Instead of training NMT models with word-level cross-entropy loss, we propose to train models with a mixed learning objective c⃝2020 China National Conference on Computational Linguistics Published under Creative Commons Attribution 4.0 International …

Content truncated to 500 words