ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems

Proceedings of NAACL-HLT 2019, pages 430–436 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 430 ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems Inigo Jauregi Unanue1,2, Ehsan Zare Borzeshi∗,2, Nazanin Esmaili2, Massimo Piccardi1 1 University of Technology Sydney, Sydney, Australia 2 Capital Markets Cooperative Research Centre, Sydney, Australia {ijauregi,ezborzeshi,nesmaili}@cmcrc.com massimo.piccardi@uts.edu.au Abstract Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this prob- lem, we propose regressing word embeddings (ReWE) as a new regularization technique in a system that is jointly trained to predict the next word in the translation (categorical value) and its word embedding (continuous value). Such a joint training allows the proposed sys- tem to learn the distributional properties rep- resented by the word embeddings, empirically improving the generalization to unseen sen- tences. Experiments over three translation datasets have showed a consistent improve- ment over a strong baseline, ranging between 0.91 and 2.54 BLEU points, and also a marked improvement over a state-of-the-art system. 1 Introduction The last few years have witnessed remarkable improvements in the performance of machine translation (MT) systems. These improvements are strongly linked to the development of neu- ral machine translation (NMT): based on encoder- decoder architectures (also known as seq2seq), NMT can use recurrent neural networks (RNNs) (Sutskever et al., 2014; Cho et al., 2014; Wu et al., 2016), convolutional neural networks (CNNs) (Gehring et al., 2017) or transformers (Vaswani et al., 2017) to learn how to map a sentence from the source language to an adequate translation in the target language. In addition, attention mecha- nisms (Bahdanau et al., 2015; Luong et al., 2015) help soft-align the encoded source words with the predictions, further improving the translation. NMT systems are usually trained via maxi- mum likelihood estimation (MLE). However, as ∗∗ The author has changed affiliation to Microsoft af- ter the completion of this work. His new email is: Ehsan.ZareBorzeshi@microsoft.com LSTM sj Linear Softmax ReWE ... s_dim vocab_dim word_emb_dim pj ej Decoder Figure 1: The proposed regularizer: the hidden vec- tor in the decoder, sj, transits through two paths: 1) a linear and a softmax layers that output vector vj (vo- cab dim) which is used for predicting the target word as usual, and 2) a two-layer network (ReWE) that outputs a vector, ej, of word embedding size (word emb dim). During training, ej is used in a regressive loss with the ground-truth embedding. pointed out by (Elbayad et al., 2018), MLE suf- fers from two obvious limitations: the first is that it treats all the predictions other than the ground truth as equally incorrect. As a consequence, syn- onyms and semantically-similar words — which are often regarded as highly interchangeable with the ground truth — are completely ignored dur- ing training. The second limitation is that MLE- trained systems suffer from “exposure bias” (Ben- gio et al., 2015; Ranzato et al., 2015) and do not generalize well over the large output space of translations. Owing to …

Content truncated to 500 words