Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1500–1505 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning Tobias Domhan and Felix Hieber Amazon Berlin, Germany {domhant,fhieber}@amazon.com Abstract The performance of Neural Machine Trans- lation (NMT) models relies heavily on the availability of sufficient amounts of paral- lel data, and an efficient and effective way of leveraging the vastly available amounts of monolingual data has yet to be found. We propose to modify the decoder in a neural sequence-to-sequence model to en- able multi-task learning for two strongly related tasks: target-side language model- ing and translation. The decoder predicts the next target word through two channels, a target-side language model on the lowest layer, and an attentional recurrent model which is conditioned on the source repre- sentation. This architecture allows joint training on both large amounts of mono- lingual and moderate amounts of bilingual data to improve NMT performance. Initial results in the news domain for three lan- guage pairs show moderate but consistent improvements over a baseline trained on bilingual data only. 1 Introduction In recent years, neural encoder-decoder models (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014) have significantly advanced the state of the art in NMT, and now con- sistently outperform Statistical Machine Transla- tion (SMT) (Bojar et al., 2016). However, their suc- cess hinges on the availability of sufficient amounts of parallel data, and contrary to the long line of research in SMT, there has only been a limited amount of work on how to effectively and effi- ciently make use of monolingual data which is typically amply available. We propose a modi- fied neural sequence-to-sequence model with atten- tion (Bahdanau et al., 2014; Luong et al., 2015b) that uses multi-task learning on the decoder side to jointly learn two strongly related tasks: target-side language modeling and translation. Our approach does not require any pre-translation or pre-training to learn from monolingual data and thus provides a principled way to integrate monolingual data re- sources into NMT training. 2 Related Work G¨ulc¸ehre et al. (2015) investigate two ways of inte- grating a pre-trained neural Language Model (LM) into a pre-trained NMT system: shallow fusion, where the LM is used at test time to rescore beam search hypothesis, requiring no additional fine- tuning and deep fusion, where hidden states of NMT decoder and LM are concatenated before making a prediction for the next word. Both com- ponents are pre-trained separately and fine-tuned together. More recently, Sennrich et al. (2016) have shown significant improvements by back-translating target-side monolingual data and using such syn- thetic data as additional parallel training data. One downside of this approach is the significantly in- creased training time, due to training of a model in the reverse direction and translation of monolingual data. In contrast, we propose to train NMT mod- els from scratch on both bilingual and target-side monolingual data in a multi-task …

Content truncated to 500 words