Exploiting Source-side Monolingual Data in Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Exploiting Source-side Monolingual Data in Neural Machine Translation Jiajun Zhang† and Chengqing Zong†‡ †University of Chinese Academy of Sciences, Beijing, China National Laboratory of Pattern Recognition, CASIA, Beijing, China ‡CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China {jjzhang,cqzong}@nlpr.ia.ac.cn Abstract Neural Machine Translation (NMT) based on the encoder-decoder architecture has recently become a new paradigm. Researchers have proven that the target-side monolingual data can greatly enhance the decoder model of NMT. However, the source-side monolingual data is not fully explored although it should be useful to strengthen the encoder model of NMT, especially when the parallel corpus is far from sufficient. In this paper, we propose two approaches to make full use of the source- side monolingual data in NMT. The first ap- proach employs the self-learning algorithm to generate the synthetic large-scale parallel data for NMT training. The second approach ap- plies the multi-task learning framework using two NMTs to predict the translation and the reordered source-side monolingual sentences simultaneously. The extensive experiments demonstrate that the proposed methods ob- tain significant improvements over the strong attention-based NMT. 1 Introduction Neural Machine Translation (NMT) following the encoder-decoder architecture proposed by (Kalch- brenner and Blunsom, 2013; Cho et al., 2014) has become the novel paradigm and obtained state-of- the-art translation quality for several language pairs, such as English-to-French and English-to-German (Sutskever et al., 2014; Bahdanau et al., 2014; Lu- ong et al., 2015b; Sennrich et al., 2015). This end- to-end NMT typically consists of two recurrent neu- ral networks. The encoder network maps the source sentence of variable length into the context vector representation; and the decoder network generates the target translation word by word starting from the context vector. Currently, most NMT methods utilize only the sentence aligned parallel corpus for model train- ing, which limits the capacity of the model. Re- cently, inspired by the successful application of tar- get monolingual data in conventional statistical ma- chine translation (SMT) (Koehn et al., 2007; Chi- ang, 2007), Gulcehre et al. (2015) and Sennrich et al. (2015) attempt to enhance the decoder net- work model of NMT by incorporating the target- side monolingual data so as to boost the transla- tion fluency. They report promising improvements by using the target-side monolingual data. In con- trast, the source-side monolingual data is not fully explored. Luong et al. (2015a) adopt a simple autoencoder or skip-thought method (Kiros et al., 2015) to exploit the source-side monolingual data, but no significant BLEU gains are reported. Note that, in parallel to our efforts, Cheng et al. (2016b) have explored the usage of both source and target monolingual data using a similar semi-supervised reconstruction method, in which two NMTs are em- ployed. One translates the source-side monolingual data into target translations, and the other recon- structs the source-side monolingual data from the target translations. In this work, we investigate the …

Content truncated to 500 words