Unsupervised Single Document Abstractive Summarization using Semantic Units

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 954–966 November 20–23, 2022. ©2022 Association for Computational Linguistics 954 Unsupervised Single Document Abstractive Summarization using Semantic Units Jhen-Yi Wu and Ying-Jia Lin and Hung-Yu Kao Intelligent Knowledge Management Lab Department of Computer Science and Information Engineering National Cheng Kung University Tainan, Taiwan jhenyiwu.d@gmail.com, yingjia.lin.public@gmail.com, hykao@mail.ncku.edu.tw Abstract In this work, we study the importance of content frequency on abstractive summariza- tion, where we define the content as "semantic units." We propose a two-stage training frame- work to let the model automatically learn the frequency of each semantic unit in the source text. Our model is trained in an unsupervised manner since the frequency information can be inferred from source text only. During infer- ence, our model identifies sentences with high- frequency semantic units and utilizes frequency information to generate summaries from the filtered sentences. Our model performance on the CNN/Daily Mail summarization task outperforms the other unsupervised methods under the same settings. Furthermore, we achieve competitive ROUGE scores with far fewer model parameters compared to several large-scale pre-trained models. Our model can be trained under low-resource language settings and thus can serve as a potential solution for real-world applications where pre-trained mod- els are not applicable. 1 Introduction Summarization is a task involving compressing a longer text into a shorter version while preserving the salient information in the original text. When given article-summary pairs, supervised models are able to learn corresponding implicit relationships, for example, where to focus or what to preserve. However, a lack of sufficient training pairs is a common issue in real-world applications. Creat- ing such high-quality training pairs can be costly. Although large pre-trained models for language generation or summarization may require less data for fine-tuning, they are often trained on English corpus only (e.g., Raffel et al., 2020; Song et al., 2019; Lewis et al., 2020; Zhang et al., 2020) and thus are not suitable for low-resource languages. Therefore, we seek the possibility of unsupervised summarization methods. Our idea is to utilize the frequency of contents in the source text. Intuitively, we expect some spe- cific contents to be included in a summary if they frequently occur in the source article. A similar concept of "content units" was first proposed by Nenkova and Passonneau (2004). They manually labeled the text by identifying similar text segments to form a content unit, where the contributing text segments of a content unit should have similar se- mantic meanings. In their results (Nenkova and Vanderwende, 2005), of the top 5 most frequent content units in the source documents, 96% appear in a human summary, and high percentages of 92 and 85 are observed for the top 8 and top 12 most frequent content units across 11 input sets. Their observation shows that content unit frequency can provide huge hints as to whether a specific unit of content will be …

Content truncated to 500 words