Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks Sascha Rothe Google Research rothe@google.com Shashi Narayan Google Research shashinarayan@google.com Aliaksei Severyn Google Research severyn@google.com Abstract Unsupervised pre-training of large neural mod- elshas recentlyrevolutionizedNatural Language Processing. By warm-starting from the pub- licly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural LanguageUnderstanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Gen- eration. We developed a Transformer-based sequence-to-sequence model that is compati- ble with publicly available pre-trained BERT, GPT-2, and RoBERTa checkpoints and con- ductedan extensiveempirical study onthe utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion. 1 Introduction Unsupervised and self-supervised pre-training methods, such as ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018), and more recently BERT (Devlin et al., 2019), GPT and GPT-2 (Radford et al., 2018, 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019) have established a qualitatively new level of baseline performance for many widely used Natural Language Understanding (NLU) bench- marks including some of the most popular, like GLUE (Williams et al., 2018) and SQuAD (Rajpurkar et al., 2018). The most appealing part about this massive shift towards using large architectures pre-trained on large collections of texts is that the pre- trained checkpoints along with the inference code are made freely available. This saves hundreds of TPU/GPU hours, as warm-starting a model from a pre-trained checkpoint typically requires orders of magnitude fewer fine-tuning steps while delivering significant performance boosts. More importantly, the ability to bootstrap from a state-of-the-art performing model such as BERT (Devlin et al., 2019) motivates the community to greatly speed up the progress towards developing better and easily reusable NLU systems. While we continue to observe an increasing number of papers building on top of BERT and/or GPT models reporting encouraging improvements on GLUE, SQuAD, and other similar benchmarks, very little attention has been paid to using these pre-trained models to warm-start sequence-to- sequence (seq2seq) models. It has been argued that the pre-training objective used by BERT is not well-suited for tasks that require decoding texts, forexample, conditionaltextgeneration in machine translation and summarization (Yang et al., 2019). Nevertheless, it remains unclear to what extent using such large models pre-trained on large collections of text can be beneficial to warm-start seq2seq generation models. In this paper, we report on a Transformer-based seq2seq model that is compatible with publicly availablepre-trained BERT, GPT-2, and RoBERTa checkpoints. We aim to provide an empirical answer to the following research question: What is the best way to leverage publicly available pre- trained checkpoints for warm-starting sequence generation models? For example, one could imagine using a BERT checkpoint to initialize the encoder for better input understanding and choosing GPT-2 model as the decoder for better text generation. One of the main contributions of …

Content truncated to 500 words