BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7871 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension Mike Lewis*, Yinhan Liu*, Naman Goyal*, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer Facebook AI mikelewis@fb.com,yinhan@ai2incubator.com,naman@fb.com Abstract We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its sim- plicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pre- training schemes. We evaluate a number of noising approaches, finding the best perfor- mance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a sin- gle mask token. BART is particularly ef- fective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new state- of-the-art results on a range of abstractive di- alogue, question answering, and summariza- tion tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine transla- tion, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their ef- fect on end-task performance.1 1 Introduction Self-supervised methods have achieved remarkable success in a wide range of NLP tasks (Mikolov et al., 2013; Peters et al., 2018; Devlin et al., 2019; Joshi et al., 2019; Yang et al., 2019; Liu et al., 2019). The most successful approaches have been variants of masked language models, which are denoising autoen- coders that are trained to reconstruct text where a ran- dom subset of the words has been masked out. Recent work has shown gains by improving the distribution of 1Code and pre-trained models for BART are avail- able at https://github.com/pytorch/fairseq and https://huggingface.co/transformers masked tokens (Joshi et al., 2019), the order in which masked tokens are predicted (Yang et al., 2019), and the available context for replacing masked tokens (Dong et al., 2019). However, these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability. In this paper, we present BART, which pre-trains a model combining Bidirectional and Auto-Regressive Transformers. BART is a denoising autoencoder built with a sequence-to-sequence model that is applicable to a very wide range of end tasks. Pretraining has two stages (1) text is corrupted with an arbitrary nois- ing function, and (2) a sequence-to-sequence model is learned to reconstruct the original text. BART uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes (see Figure 1). …

Content truncated to 500 words