Unsupervised Pretraining for Sequence to Sequence Learning

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 383–391 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Unsupervised Pretraining for Sequence to Sequence Learning Prajit Ramachandran and Peter J. Liu and Quoc V. Le Google Brain {prajit, peterjliu, qvl}@google.com Abstract This work presents a general unsuper- vised learning method to improve the ac- curacy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We ap- ply this method to challenging bench- marks in machine translation and abstrac- tive summarization and find that it sig- nificantly improves the subsequent super- vised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the- art results on the WMT English→German task, surpassing a range of methods us- ing both phrase-based machine transla- tion and neural machine translation. Our method achieves a significant improve- ment of 1.3 BLEU from the previous best models on both WMT’14 and WMT’15 English→German. We also conduct hu- man evaluations on abstractive summa- rization and find that our method outper- forms a purely supervised learning base- line in a statistically significant manner. 1 Introduction Sequence to sequence (seq2seq) models (Sutskever et al., 2014; Cho et al., 2014; Kalchbrenner and Blunsom, 2013; Allen, 1987; ˜Neco and Forcada, 1997) are extremely effective on a variety of tasks that require a mapping between a variable-length input sequence to a variable-length output sequence. The main weakness of sequence to sequence models, and deep networks in general, lies in the fact that they can easily overfit when the amount of supervised training data is small. In this work, we propose a simple and effec- tive technique for using unsupervised pretraining to improve seq2seq models. Our proposal is to initialize both encoder and decoder networks with pretrained weights of two language models. These pretrained weights are then fine-tuned with the la- beled corpus. During the fine-tuning phase, we jointly train the seq2seq objective with the lan- guage modeling objectives to prevent overfitting. We benchmark this method on machine trans- lation for English→German and abstractive sum- marization on CNN and Daily Mail articles. Our main result is that a seq2seq model, with pre- training, exceeds the strongest possible baseline in both neural machine translation and phrase- based machine translation. Our model obtains an improvement of 1.3 BLEU from the previ- ous best models on both WMT’14 and WMT’15 English→German. On human evaluations for ab- stractive summarization, we find that our model outperforms a purely supervised baseline, both in terms of correctness and in avoiding unwanted repetition. We also perform ablation studies to understand the behaviors of the pretraining method. Our study confirms that among many other possible choices of using a language model in seq2seq with atten- tion, the above proposal works best. Our study also shows that, for translation, the main gains come from the improved …

Content truncated to 500 words