Adversarial Domain Adaptation Using Artificial Titles for Abstractive Title Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2197–2203 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2197 Adversarial Domain Adaptation Using Artificial Titles for Abstractive Title Generation Francine Chen FX Palo Alto Laboratory Palo Alto, CA chen@fxpal.com Yan-Ying Chen FX Palo Alto Laboratory Palo Alto, CA yanying@fxpal.com Abstract A common issue in training a deep learning, abstractive summarization model is lack of a large set of training summaries. This paper ex- amines techniques for adapting from a labeled source domain to an unlabeled target domain in the context of an encoder-decoder model for text generation. In addition to adversarial domain adaptation (ADA), we introduce the use of artificial titles and sequential training to capture the grammatical style of the unlabeled target domain. Evaluation on adapting to/from news articles and Stack Exchange posts indi- cates that the use of these techniques can boost performance for both unsupervised adaptation as well as fine-tuning with limited target data. 1 Introduction Many types of textual content, such as conver- sations and posts on chat, do not have a title or summary. While multi-sentence extractive sum- marization can give a sense of the content of an article, a title or highlight is more concise. Such short summaries can be generated using abstrac- tive summarization with an RNN encoder-decoder model, e.g., (Nallapati et al., 2016). A common issue when training models for abstractive summarization of conversations and posts is the lack of a large set of text with sum- maries. Obtaining good quality labeled data can be difficult and expensive, especially if author- generated summaries are desired. One option is to train on data from another domain with author- generated titles, but because of differences be- tween domains, the performance may be less than adequate. These differences include different vo- cabularies, different grammatical styles, and dif- ferent ways of expressing similar concepts. Vo- cabulary expansion may be used to address the dif- ferent vocabularies in source and target domains, and adversarial domain adaptation (ADA) may be used to merge the embedded feature representa- tions across domains. However, ADA does not adapt the decoder in an encoder-decoder genera- tion model. In this paper, we investigate the utility of these techniques in unsupervised domain adaptation for title generation. We also examine the use of a limited amount of labeled training data from the target domain, when high performance may be required but training data is not easily available. Our contributions include (1) proposing the use of artificial titles for unlabeled target documents to train a decoder to learn the grammatical style of titles in the new domain (2) proposing to train the decoder in a sequence of steps that encourages the source and target embedding spaces to remain aligned during adaptation, and (3) showing that our model improves performance over ADA and an expanded vocabulary alone and further, that a limited amount of labeled target data can achieve performance close to training on all labeled target data. …

Content truncated to 500 words