Document Summarization with Latent Queries

Document Summarization with Latent Queries Yumo Xu and Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, United Kingdom yumo.xu@ed.ac.uk mlap@inf.ed.ac.uk Abstract The availability of large-scale datasets has driven the development of neural models that create generic summaries for single or multiple documents. For query-focused summarization (QFS), labeled training data in the form of queries, documents, and summaries is not read- ily available. We provide a unified modeling framework for any kind of summarization, under the assumption that all summaries are a response to a query, which is observed in the case of QFS and latent in the case of generic summarization. We model queries as discrete latent variables over document tokens, and learn representations compatible with ob- served and unobserved query verbalizations. Our framework formulates summarization as a generative process, and jointly optimizes a latent query model and a conditional lan- guage model. Despite learning from generic summarization data only, our approach out- performs strong comparison systems across benchmarks, query types, document settings, and target domains.1 1 Introduction Recent years have witnessed substantial progress in generic summarization (See et al., 2017; Gehrmann et al., 2018; Liu and Lapata, 2019a, in- ter alia) thanks to neural architectures based on the encoder-decoder paradigm (Sutskever et al., 2014) and the availability of large-scale datasets contain- ing hundreds of thousands of document-summary pairs. Unfortunately, training data of this mag- nitude is not readily available for the related task of query-focused summarization (QFS; Dang 2005) which aims to create a summary from one or multiple document(s) that answers a specific query. Existing QFS benchmarks (Dang, 2005; Hoa, 2006; Nema et al., 2017; Baumel et al., 1Our code and models can be found at https:// github.com/yumoxu/lqsum. 2016) have been constructively used for evalu- ation but are relatively small for training large neural models. To make up for the absence of labeled QFS data, recent work has resorted to distant supervision provided by pretrained models, paraphrase identi- fication, and question-answering datasets (Xu and Lapata, 2020; Su et al., 2020; Laskar et al., 2020b). Other work induces proxy queries (Xu and Lapata, 2021) from generic summarization datasets, without additional question-answering resources that can be also extremely expen- sive to acquire (Bajaj et al., 2016). Despite this progress, building and scaling QFS systems re- mains challenging due to the many different ways natural language queries express users’ in- formation needs. For instance, queries can have one or multiple keyword(s) (Baumel et al., 2016; Zhu et al., 2019), a simple question (Nema et al., 2017), or a longer narrative composed of multi- ple sub-queries (Dang, 2006) (see the examples in Table 1). Although QFS systems can poten- tially handle queries resembling those seen in training, they are not expected to work well on out-of-distribution queries (Xu and Lapata, 2021), namely, queries with different surface forms from those seen in training. In order to cover new types of queries, it might be necessary to gather more data, re-design proxy …

Content truncated to 500 words