Uncertainty-Aware Semantic Augmentation for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2724–2735, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2724 Uncertainty-Aware Semantic Augmentation for Neural Machine Translation Xiangpeng Wei1,2∗, Heng Yu3, Yue Hu1,2†, Rongxiang Weng3, Luxi Xing1,2, Weihua Luo3 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Machine Intelligence Technology Lab, Alibaba Group, Hangzhou, China {weixiangpeng,huyue,xingluxi}@iie.ac.cn {yuheng.yh,wengrx,weihua.luowh}@alibaba-inc.com Abstract As a sequence-to-sequence generation task, neural machine translation (NMT) naturally contains intrinsic uncertainty, where a single sentence in one language has multiple valid counterparts in the other. However, the dom- inant methods for NMT only observe one of them from the parallel corpora for the model training but have to deal with adequate variations under the same meaning at infer- ence. This leads to a discrepancy of the data distribution between the training and the inference phases. To address this problem, we propose uncertainty-aware semantic aug- mentation, which explicitly captures the uni- versal semantic information among multiple semantically-equivalent source sentences and enhances the hidden representations with this information for better translations. Extensive experiments on various translation tasks reveal that our approach significantly outperforms the strong baselines and the existing methods. 1 Introduction In recent years neural machine translation (NMT) has demonstrated state-of-the-art performance on many language pairs with advanced architectures and large scale data (Bahdanau et al., 2015; Wu et al., 2016; Vaswani et al., 2017). At training time the parallel data only contains one source sentence as the input and the rest reasonable ones are ig- nored, while at inference the resulting model has to deal with adequate variations under the same meaning. This discrepancy of the data distribution poses a formidable learning challenge of the in- herent uncertainty in machine translation. Since typically there are several semantically-equivalent source sentences that can be translated to the same target sentence, but the model only observes one at ∗Work done at Alibaba Group. †Corresponding Author. training time. Thus it is natural to enable an NMT model trained with the token-level cross-entropy (CE) to capture such a rich distribution, which ex- actly motivates our work. Intuitively, the NMT model should be trained under the guidance of the same latent semantics that it will access at inference time. In their sem- inal work, the variational models (Blunsom et al., 2008; Zhang et al., 2016; Shah and Barber, 2018) introduce a continuous latent variable to serve as a global semantic signal to guide the generation of target translations. Wei et al. (2019) consider an universal topic representation for each sentence pair as global semantics for enhancing representa- tions learnt by NMT models. Yang et al. (2019) minimize the difference between the representa- tion of source and target sentences. Although their methods yield notable results, they are still limited to one-to-one parallel sentence pairs. To address these problems, we present a novel uncertainty-aware semantic augmentation method, which takes account of the intrinsic uncertainty sourced from the one-to-many nature of machine …

Content truncated to 500 words