Hierarchical Transformers for Multi-Document Summarization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070–5081 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5070 Hierarchical Transformers for Multi-Document Summarization Yang Liu and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh yang.liu2@ed.ac.uk, mlap@inf.ed.ac.uk Abstract In this paper, we develop a neural summa- rization model which can effectively process multiple input documents and distill abstrac- tive summaries. Our model augments a previ- ously proposed Transformer architecture (Liu et al., 2018) with the ability to encode docu- ments in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among tex- tual units, but can also take advantage of ex- plicit graph representations focusing on simi- larity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed architecture brings substantial im- provements over several strong baselines.1 1 Introduction Automatic summarization has enjoyed renewed interest in recent years, thanks to the popular- ity of neural network models and their ability to learn continuous representations without recourse to preprocessing tools or linguistic annotations. The availability of large-scale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018) containing hundreds of thousands of document- summary pairs has driven the development of neural architectures for summarizing single doc- uments. Several approaches have shown promis- ing results with sequence-to-sequence models that encode a source document and then decode it into an abstractive summary (See et al., 2017; Celiky- ilmaz et al., 2018; Paulus et al., 2018; Gehrmann et al., 2018). Multi-document summarization — the task of producing summaries from clusters of themati- 1Our code and data is available at https://github. com/nlpyang/hiersumm. cally related documents — has received signif- icantly less attention, partly due to the paucity of suitable data for the application of learning methods. High-quality multi-document summa- rization datasets (i.e., document clusters paired with multiple reference summaries written by hu- mans) have been produced for the Document Un- derstanding and Text Analysis Conferences (DUC and TAC), but are relatively small (in the range of a few hundred examples) for training neu- ral models. In an attempt to drive research fur- ther, Liu et al. (2018) tap into the potential of Wikipedia and propose a methodology for cre- ating a large-scale dataset (WikiSum) for multi- document summarization with hundreds of thou- sands of instances. Wikipedia articles, specifically lead sections, are viewed as summaries of various topics indicated by their title, e.g.,“Florence” or “Natural Language Processing”. Documents cited in the Wikipedia articles or web pages returned by Google (using the section titles as queries) are seen as the source cluster which the lead section purports to summarize. Aside from the difficulties in obtaining train- ing data, a major obstacle to the application of end-to-end models to multi-document summariza- tion is the sheer size and number of source doc- uments which …

Content truncated to 500 words