Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths

Proceedings of the 28th International Conference on Computational Linguistics, pages 6871–6882 Barcelona, Spain (Online), December 8-13, 2020 6871 Interpretable Multi-Headed Attention for Abstractive Summarization at Controllable Lengths Ritesh Sarkhel∗ Moniba Keymanesh∗ Department of Computer Science and Engineering The Ohio State University sarkhel.5,keymanesh.1,nandi.9,parthasarathy.2@osu.edu Arnab Nandi Srinivasan Parthasarathy Abstract Abstractive summarization at controllable lengths is a challenging task in natural language pro- cessing. It is even more challenging for domains where limited training data is available or scenarios in which the length of the summary is not known beforehand. At the same time, when it comes to trusting machine-generated summaries, explaining how a summary was constructed in human-understandable terms may be critical. We propose Multi-level Summarizer (MLS), a supervised method to construct abstractive summaries of a text document at controllable lengths. The key enabler of our method is an interpretable multi-headed attention mechanism that com- putes attention distribution over an input document using an array of timestep independent se- mantic kernels. Each kernel optimizes a human-interpretable syntactic or semantic property. Exhaustive experiments on two low-resource datasets in English language show that MLS out- performs strong baselines by up to 14.70% in the METEOR score. Human evaluation of the sum- maries also suggests that they capture the key concepts of the document at various length-budgets. 1 Introduction Great progress has been made in recent years on abstractive summarization of text documents. Among existing works, sequence-to-sequence networks with attention (Gehring et al., 2017; Liu et al., 2018a) have been one of the clear front-runners. Being able to constrain the length of a summary while preserv- ing its desirable properties has many real-world applications. One such application is content optimiza- tion for variable screen-sizes. Online content creators such as news portals, blogs, and advertisement agencies with audiences on multiple platforms customize their content based on display-area for best ex- perience. However, there has not been much work on summarization at controllable lengths until recently. High variance in screen-sizes often require extensive human supervision to perform these modifications. As most sequence-to-sequence networks (Rush et al., 2015; Nallapati et al., 2016) do not enforce the length of a summary, for scenarios as mentioned above, one may need to employ an ensemble of net- works to cover all possible lengths. There are two major challenges in following this approach for real- world applications. First, training sequence-to-sequence networks is a resource-intensive task (Strubell et al., 2019). To train a network for generating summaries budgeted at length b, we need a parallel corpus of text documents and their gold-standard summaries at length b. Constructing a large enough corpus with summaries budgeted at b, ∀b ∈ (0, 1) may not be possible and/or cost-efficient for a number of do- mains. This is one of the main reasons why most existing works on abstractive summarization evaluate their model on large-scale news corpus datasets (Nallapati et al., 2016; Hermann et al., 2015), leaving out a number of important but low-resource domains (Magooda and Litman, 2020; Parida and Motlicek, 2019) where the number of …

Content truncated to 500 words