Semi-Supervised Learning for Neural Machine Translation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1965–1974, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Semi-Supervised Learning for Neural Machine Translation Yong Cheng#, Wei Xu#, Zhongjun He+, Wei He+, Hua Wu+, Maosong Sun† and Yang Liu† ∗ #Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China †State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China +Baidu Inc., Beijing, China chengyong3001@gmail.com weixu@tsinghua.edu.cn {hezhongjun,hewei06,wu hua}@baidu.com {sms,liuyang2011}@tsinghua.edu.cn Abstract While end-to-end neural machine transla- tion (NMT) has made remarkable progress recently, NMT systems only rely on par- allel corpora for parameter estimation. Since parallel corpora are usually limited in quantity, quality, and coverage, espe- cially for low-resource languages, it is appealing to exploit monolingual corpora to improve NMT. We propose a semi- supervised approach for training NMT models on the concatenation of labeled (parallel corpora) and unlabeled (mono- lingual corpora) data. The central idea is to reconstruct the monolingual corpora us- ing an autoencoder, in which the source- to-target and target-to-source translation models serve as the encoder and decoder, respectively. Our approach can not only exploit the monolingual corpora of the target language, but also of the source language. Experiments on the Chinese- English dataset show that our approach achieves significant improvements over state-of-the-art SMT and NMT systems. 1 Introduction End-to-end neural machine translation (NMT), which leverages a single, large neural network to directly transform a source-language sentence into a target-language sentence, has attracted increas- ing attention in recent several years (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). Free of latent structure design and feature engineering that are critical in conventional statistical machine translation (SMT) (Brown et al., 1993; Koehn et al., 2003; Chi- ang, 2005), NMT has proven to excel in model- ∗ Yang Liu is the corresponding author. ing long-distance dependencies by enhancing re- current neural networks (RNNs) with the gating (Hochreiter and Schmidhuber, 1993; Cho et al., 2014; Sutskever et al., 2014) and attention mecha- nisms (Bahdanau et al., 2015). However, most existing NMT approaches suf- fer from a major drawback: they heavily rely on parallel corpora for training translation mod- els. This is because NMT directly models the probability of a target-language sentence given a source-language sentence and does not have a sep- arate language model like SMT (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). Unfortunately, parallel corpora are usually only available for a handful of research- rich languages and restricted to limited domains such as government documents and news reports. In contrast, SMT is capable of exploiting abundant target-side monolingual corpora to boost fluency of translations. Therefore, the unavailability of large-scale, high-quality, and wide-coverage par- allel corpora hinders the applicability of NMT. As a result, several authors have tried to use abundant monolingual corpora to improve NMT. Gulccehre et al. (2015) propose two methods, which are referred to as shallow fusion and …

Content truncated to 500 words