Pre-trained language model representations for language generation

Proceedings of NAACL-HLT 2019, pages 4052–4059 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4052 Pre-trained Language Model Representations for Language Generation Sergey Edunov∗, Alexei Baevski∗, Michael Auli Facebook AI Research Menlo Park, CA {edunov,abaevski,michaelauli}@fb.com Abstract Pre-trained language model representations have been successful in a wide range of lan- guage understanding tasks. In this paper, we examine different strategies to integrate pre- trained representations into sequence to se- quence models and apply it to neural ma- chine translation and abstractive summariza- tion. We find that pre-trained representa- tions are most effective when added to the en- coder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe im- provements when millions of sentence-pairs are available. Finally, on abstractive summa- rization we achieve a new state of the art on the full text version of CNN-DailyMail. 1 1 Introduction Pre-training of language models has been shown to provide large improvements for a range of lan- guage understanding tasks (Peters et al., 2018; Radford et al., 2018; Phang et al., 2018; Devlin et al., 2018). The key idea is to train a large gen- erative model on vast corpora and use the result- ing representations on tasks for which only limited amounts of labeled data is available. Pre-training of sequence to sequence models has been previ- ously investigated for text classification (Dai and Le, 2015) but not for text generation. In neural ma- chine translation, there has been work on transfer- ring representations from high-resource language pairs to low-resource settings (Zoph et al., 2016). In this paper, we apply pre-trained representa- tions from language models to language genera- ∗Equal contribution. 1Code and pre-trained models are available at https://github.com/pytorch/fairseq/tree/ bi_trans_lm/examples/pretraining tion tasks that can be modeled by sequence to se- quence architectures. Previous work on integrat- ing language models with sequence to sequence models focused on the decoder network and added language model representations right before the output of the decoder (Gulcehre et al., 2015). We extend their study by investigating several other strategies such as inputting ELMo-style represen- tations (Peters et al., 2018) or fine-tuning the lan- guage model (§2). Our experiments rely on strong transformer- based language models trained on up to six bil- lion tokens (§3). We present a detailed study of various strategies in different simulated labeled training data scenarios and observe the largest im- provements in low-resource settings but gains of over 1 BLEU are still possible when five million sentence-pairs are available. The most successful strategy to integrate pre-trained representations is as input to the encoder network (§4). 2 Strategies to add representations We consider augmenting a standard sequence to sequence model with pre-trained representations following an ELMo-style regime (§2.1) as well as by fine-tuning the language model (§2.2). 2.1 ELMo augmentation The ELMo approach of Peters et al. (2018) forms contextualized word embeddings based on lan- guage …

Content truncated to 500 words