Long Document Summarization with Top-down and Bottom-up Inference

Findings of the Association for Computational Linguistics: EACL 2023, pages 1267–1284 May 2-6, 2023 ©2023 Association for Computational Linguistics Long Document Summarization with Top-down and Bottom-up Inference Bo Pang Erik Nijkamp Wojciech Kryscinski Silvio Savarese Yingbo Zhou Salesforce ResearchCaiming Xiong {b.pang, erik.nijkamp, wojciech.kryscinski}@salesforce.com {ssavarese, yingbo.zhou, cxiong}@salesforce.com Abstract Text summarization aims to condense long doc- uments and retain key information. Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents. Most recent models infer the latent represen- tations with a transformer encoder, which is purely bottom-up and thus does not capture long-distance context well. Also, self-attention- based models face the challenge of quadratic complexity with respect to sequence length. We propose a method to improve summarization models on these two aspects. Our method as- sumes a hierarchical latent structure of a doc- ument where the top-level captures the long range dependency at a coarser time scale and the bottom token level preserves the details. Critically, our method enables token represen- tations to be updated in both a bottom-up and top-down manner. In the bottom-up pass, to- ken representations are inferred with local self- attention to leverage its efficiency. Top-down correction is then applied to allow tokens to capture global context. We demonstrate the ef- fectiveness on a diverse set of summarization datasets, including narrative, conversational, scientific documents and news. Our model achieves state-of-the-art performance on a wide range of long document summarization bench- marks, compared to recent efficient transform- ers. We show that our model can summarize an entire book and achieve competitive per- formance using 0.27% parameters and much less training data, compared to a recent GPT-3- based model. These results indicate the general applicability and benefits of the framework. 1 Introduction An abstractive summarization system aims to gener- ate a semantically coherent and linguistically fluent summary by conditioning on the document. The dominant approach for abstractive summarization is to use a Seq2Seq model (Sutskever et al., 2014) with an encoder-decoder architecture instantiated with either RNNs (Hochreiter and Schmidhuber, 1997) or transformers (Vaswani et al., 2017). In such a model, an encoder computes or infers 1 la- tent representations of observed tokens (words or subwords) in a document, conditioning on which a decoder generates a summary. This paper studies the problem of how to compute informative latent representations, which in turn would improve sum- marization. We propose a method which synergizes bottom- up computation with top-down computation while assuming a multi-scale latent structure of a docu- ment. In a multi-scale structure, higher-level vari- ables (like those representing sentences, segments) model the document at a coarser time-scale and abstract away details, and are suitable for capturing long range dependency of the document; in con- trast, lower-level variables (like those representing tokens) preserve details, and prevent the summary from losing key details (such as the name of an en- tity). In our method, the summary is generated by conditioning on token representations (low-level variables), similar to recent abstractive summariza- …

Content truncated to 500 words