Towards Zero-Shot Conditional Summarization with Adaptive Multi-Task Fine-Tuning

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3215–3226 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3215 Towards Zero-Shot Conditional Summarization with Adaptive Multi-Task Fine-Tuning Travis R. Goodwin, Max E. Savery, and Dina Demner-Fushman U.S. National Library of Medicine National Institutes of Health {firstname.lastname}@nih.gov Abstract Automatic summarization research has tra- ditionally focused on providing high qual- ity general-purpose summaries of documents. However, there are many applications that re- quire more specific summaries, such as sup- porting question answering or topic-based lit- erature discovery. In this paper, we study the problem of conditional summarization in which content selection and surface realization are explicitly conditioned on an ad-hoc natural language question or topic description. Be- cause of the difficulty in obtaining sufficient reference summaries to support arbitrary con- ditional summarization, we explore the use of multi-task fine-tuning (MTFT) on twenty- one natural language tasks to enable zero-shot conditional summarization on five tasks. We present four new summarization datasets, two novel “online” or adaptive task-mixing strate- gies, and report zero-shot performance using T5 and BART, demonstrating that MTFT can improve zero-shot summarization quality. 1 Introduction Transfer learning, in which a model is first pre- trained on one or more data-rich tasks before be- ing fine-tuned on a downstream task of interest, has been repeatedly shown to obtain remarkable performance on many natural language processing tasks (Yang et al., 2019; Dong et al., 2019; Liu et al., 2019b). The most successful models result- ing from this paradigm rely on self-supervised pre- training with prohibitively-large1 datasets to facili- tate adaptation to new tasks (i.e., fine-tuning) with less abundant data (Devlin et al., 2019; Lewis et al., 2020; Keskar et al., 2019; Raffel et al., 2019). Un- fortunately, the benefits of pretraining are reduced for tasks in which there is little direct knowledge 1As estimated by Strubell et al. (2019), the cost for training the 11 billion parameter variant of T5 (Raffel et al., 2019) can exceed $1.3 million USD for a single run. Document: Asthma is a condition in which your airways narrow and swell and produce extra mucus. This can make breathing difficult and trigger coughing, wheezing and shortness of breath. [...] Question: What is the consensus of medical doctors as to whether asthma can be cured? Summary: Asthma can’t be cured, but its symptoms can be controlled. Because asthma often changes over time, it’s im- portant that you work with your doctor to track your signs and symptoms and adjust treatment as needed [...] (a) Health Question (Savery et al., 2020) Document: The United Nations Thursday set aside $1 million to as- sess environmental damage caused by this week’s devastating tsunami, as reports of destroyed coral reefs and uprooted mangrove forests be- gan trickling in [...] Topic: Coral reefs Summary: The waves of the tsunami in southeast Asia wreaked tremendous damage on coral reefs, but much more damage occurred when the waves receded, carrying [...] Topic: Mangrove Forests Summary: The recent 26 December 2004 tsunami in the Indian …

Content truncated to 500 words