Deep Communicating Agents for Abstractive Summarization

Proceedings of NAACL-HLT 2018, pages 1662–1675 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Deep Communicating Agents for Abstractive Summarization Asli Celikyilmaz1, Antoine Bosselut2, Xiaodong He3 and Yejin Choi2,4 1Microsoft Research 2Paul G. Allen School of Computer Science & Engineering, University of Washington 3JD AI Research 4Allen Institute for Artificial Intelligence {aslicel}@microsoft.com {xiaodong.he}@jd.com {antoineb, yejin}@cs.washington.edu Abstract We present deep communicating agents in an encoder-decoder architecture to ad- dress the challenges of representing a long document for abstractive summariza- tion. With deep communicating agents, the task of encoding a long text is divided across multiple collaborating agents, each in charge of a subsection of the input text. These encoders are connected to a sin- gle decoder, trained end-to-end using rein- forcement learning to generate a focused and coherent summary. Empirical results demonstrate that multiple communicating encoders lead to a higher quality summary compared to several strong baselines, in- cluding those based on a single encoder or multiple non-communicating encoders. 1 Introduction We focus on the task of abstractive summariza- tion of a long document. In contrast to extractive summarization, where a summary is composed of a subset of sentences or words lifted from the in- put text as is, abstractive summarization requires the generative ability to rephrase and restructure sentences to compose a coherent and concise sum- mary. As recurrent neural networks (RNNs) are capable of generating fluent language, variants of encoder-decoder RNNs (Sutskever et al., 2014; Bahdanau et al., 2015) have shown promising re- sults on the abstractive summarization task (Rush et al., 2015; Nallapati et al., 2017). The fundamental challenge, however, is that the strong performance of neural models at encoding short text does not generalize well to long text. The motivation behind our approach is to be able to dynamically attend to different parts of the input to capture salient facts. While recent work in sum- … … … Tired of counting sheep ? Try one of these remedies and get a good nights sleep: Aroma .. How to use it: Massage a dab of aroma therapeutic balm or oil… … messages … … … … paragraphs 1 2 1 2 layer-1 layer-2 layer-3 agent a agent b Figure 1: Illustration of deep communicating agents presented in this paper. Each agent a and b encodes one paragraph in multiple layers. By passing new mes- sages through multiple layers the agents are able to co- ordinate and focus on the important aspects of the input text. marization addresses these issues using improved attention models (Chopra et al., 2016), pointer networks with coverage mechanisms (See et al., 2017), and coherence-focused training objectives (Paulus et al., 2018; Jaques et al., 2017), an effec- tive mechanism for representing a long document remains a challenge. Simultaneous work has investigated the use of deep communicating agents (Sukhbaatar et al., 2016) for collaborative tasks such as logic puzzles (Foerster et al., 2016), visual dialog (Das et al., 2017), and reference games (Lazaridou et al., 2016). Our work builds on these …

Content truncated to 500 words