Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1667 - 1681 July 10-15, 2022 ©2022 Association for Computational Linguistics Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness Yun-Zhu Song and Yi-Syuan Chen and Hong-Han Shuai National Yang Ming Chiao Tung University, Taiwan {yunzhusong.eed07g,yschen.eed09g,hhshuai}@nctu.edu.tw Abstract A notable challenge in Multi-Document Sum- marization (MDS) is the extremely-long length of the input. In this paper, we present an extract-then-abstract Transformer framework to overcome the problem. Specifically, we leverage pre-trained language models to con- struct a hierarchical extractor for salient sen- tence selection across documents and an ab- stractor for rewriting the selected contents as summaries. However, learning such a frame- work is challenging since the optimal contents for the abstractor are generally unknown. Pre- vious works typically create pseudo extraction oracle to enable the supervised learning for both the extractor and the abstractor. Nev- ertheless, we argue that the performance of such methods could be restricted due to the insufficient information for prediction and in- consistent objectives between training and test- ing. To this end, we propose a loss weight- ing mechanism that makes the model aware of the unequal importance for the sentences not in the pseudo extraction oracle, and leverage the fine-tuned abstractor to generate summary references as auxiliary signals for learning the extractor. Moreover, we propose a reinforce- ment learning method that can efficiently ap- ply to the extractor for harmonizing the opti- mization between training and testing. Experi- ment results show that our framework substan- tially outperforms strong baselines with com- parable model sizes and achieves the best re- sults on the Multi-News, Multi-XScience, and WikiCatSum corpora.1 1 Introduction Neural multi-document summarization has drawn a lot of attention due to the wide applications, e.g., Wikipedia generation (Liu et al., 2018), news digest (Fabbri et al., 2019), or related-work sec- tion generation (Lu et al., 2020). Through con- 1The implementation code and trained models are available at https://github.com/yunzhusong/NAACL2022-REFLECT catenating the document clusters, it is applica- ble to adopt the approaches from single-document summarization (Zhang et al., 2020a; Fabbri et al., 2019) under multi-document settings. However, the input length from multiple documents is typi- cally long, which might be computationally infea- sible for Transformer-based models (Vaswani et al., 2017). One possible solution is to set a fixed length and truncate the input sequence. However, the trun- cation leads to information loss and performance drop. Moreover, even if the model can take such a long sequence as input, the attentions could be dis- persed over long sequences (Yang et al., 2018) and further degrade the performance (Jin et al., 2020; Liu and Lapata, 2019a; Cohan et al., 2018). To tackle the long sequence problem, another possible solution is to leverage the extract-then- abstract architecture in single document summa- rization (Pilault et al., 2020; Chen and Bansal, 2018; Gehrmann et al., 2018). In such an archi- tecture, an extractor first selects salient contents from the documents, and an abstractor is …

Content truncated to 500 words