RefSum: Refactoring Neural Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1437–1448 June 6–11, 2021. ©2021 Association for Computational Linguistics 1437 RefSum: Refactoring Neural Summarization Yixin Liu, Zi-Yi Dou, Pengfei Liu ∗ Carnegie Mellon University {yixinl2,zdou,pliu3}@cs.cmu.edu Abstract Although some recent works show potential complementarity among different state-of-the- art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this prob- lem. In this work, we highlight several limi- tations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summariza- tion and summaries combination. Experimen- tally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application sce- narios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE- 1), we also elaborate on how our proposed method addresses the limitations of the tra- ditional methods and the effectiveness of the Refactor model sheds light on insight for per- formance improvement. Our system can be directly used by other researchers as an off- the-shelf tool to achieve further performance improvements. We open-source all the code and provide a convenient interface to use it: https://github.com/yixinL7/ Refactoring-Summarization. 1 Introduction In neural text summarization, system designers commonly have flexible choices in model archi- tectures (Rush et al., 2015; Kedzie et al., 2018), decoding strategies (Paulus et al., 2018) (e.g. beam search) and etc. As a result, even on the same dataset, different selection biases of these choices will lead to diverse system outputs (Kedzie et al., 2018; Hossain et al., 2020). To combine complementarity of system’s output under different setups, researchers have made some preliminary efforts on two-stage learning (Collins and Koo, 2005; Huang, 2008; González-Rubio ∗Corresponding author. Figure 1: Illustration of two-stage learning. “Doc, Hypo, Ref” represent “input document, generated hypothesis, gold reference” respectively. “Hypo’” represents texts generated during test phase. ΘBase and ΘMeta represent learnable parameters in two stages. et al., 2011; Mizumoto and Matsumoto, 2016), con- sisting of (i) a base-stage: first generates different outputs under different setups, and (ii) a meta-stage: then aggregates them in diverse ways, exemplified by stacking that uses a high-level model to com- bine multiple low-level models (Ting and Witten, 1997), or reranking (Collins and Koo, 2005), which aims to rerank different outputs of one system. Al- though these methods each play a role in different scenarios, they suffer from following potential lim- itations: (i) Ad-hoc Methods: most existing methods are designed for a specific scenario. For example, Li et al. (2015) and Narayan et al. (2018b) resort to reranking techniques to select summary-worthy sentences that are usually generated from one sys- tem. By contrast, Hong et al. (2015) focus on sum- maries generated from different systems and use a non-neural system combination method to make their complementary advantages. Few works ex- plore if the complementarity existing in different scenarios could be utilized in a unified framework. …

Content truncated to 500 words