Large-Scale Transfer Learning for Natural Language Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6053–6058 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6053 Large-Scale Transfer Learning for Natural Language Generation Sergey Golovanov *1, Rauf Kurbanov *1, Sergey Nikolenko *12, Kyryl Truskovskyi *1, Alexander Tselousov *, and Thomas Wolf *3 1Neuromation OU, Liivalaia tn 45, 10145 Tallinn, Estonia 2Steklov Mathematical Institute at St. Petersburg, nab. r. Fontanki 27, St. Petersburg 191023, Russia 3Huggingface Inc., 81 Prospect St. Brooklyn, New York 11201, USA sergey.golovanov@neuromation.io, rauf.kurbanov@neuromation.io, snikolenko@neuromation.io, kyryl@neuromation.io, al.tselousov@gmail.com, thomas@huggingface.co *All authors contributed equally, names in alphabetical order. Abstract Large-scale pretrained language models define state of the art in natural language process- ing, achieving outstanding performance on a variety of tasks. We study how these archi- tectures can be applied and adapted for natu- ral language generation, comparing a number of architectural and training schemes. We fo- cus in particular on open-domain dialog as a typical high entropy generation task, present- ing and comparing different architectures for adapting pretrained models with state of the art results. 1 Introduction Over the past few years, the field of natural lan- guage processing (NLP) has witnessed the emer- gence of transfer learning methods which have significantly improved the state of the art (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018). These methods depart from classical super- vised machine learning where a predictive model for a given task is trained in isolation on a sin- gle dataset. Here, a model is pretrained on large text corpora and then fine-tuned on the target task. Such models are usually evaluated on natural lan- guage understanding (NLU) tasks such as text classification or question answering (Wang et al.; Rajpurkar et al., 2016), but natural language gen- eration (NLG) tasks such as summarization, dia- log, or machine translation remain relatively un- derexplored. At first glance, large-scale pretrained models appear to be a natural fit for NLG since their pretraining objectives are often derived from language modeling. However, interesting ques- tions and problems still arise. We consider a text-only NLG task where the generation of an output sequence of symbols y = (y1,...,ym) is conditioned on a context X = (x1,...,xK) composed of one or several sequences of symbols xk = (xk 1,...,xk n). Several types of con- texts may warrant different treatment in the model. E.g., in case of dialog generation they may in- clude: (i) facts from a knowledge base, (ii) dia- log history, and (iii) the sequence of already gen- erated output tokens (y1,...,ym−1). Thus, there arises a general question of how to adapt a single- input pretrained model to a multi-input down- stream generation task. In this work, we study two general schemes to adapt a pretrained language model to an NLG task. In the single-input setting, contexts are concate- nated to create a sequence prefix from which the output is decoded as a continuation by the pre- trained language model …

Content truncated to 500 words