Breaking Down Walls of Text: How Can NLP Benefit Consumer Privacy?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 366–376 August 1–6, 2021. ©2021 Association for Computational Linguistics 366 Deep Differential Amplifier for Extractive Summarization Ruipeng Jia1,2, Yanan Cao1,2, Fang Fang1,2∗, Yuchen Zhou1, Zheng Fang1, Yanbing Liu1,2 and Shi Wang3∗ 1Institute of Information Engineering, Chinese Academy of Sciences 2School of Cyber Security, University of Chinese Academy of Sciences 3Institute of Computing Technology, Chinese Academy of Sciences 1,2{jiaruipeng, caoyanan, fangfang0703, zhouyuchen, fangzheng, liuyanbing}@iie.ac.cn 3wangshi@ict.ac.cn Abstract For sentence-level extractive summarization, there is a disproportionate ratio of selected and unselected sentences, leading to flatting the summary features when optimizing the clas- sification. The imbalanced sentence classi- fication in extractive summarization is inher- ent, which can’t be addressed by data sam- pling or data augmentation algorithms easily. In order to address this problem, we inno- vatively consider the single-document extrac- tive summarization as a rebalance problem and present a deep differential amplifier frame- work to enhance the features of summary sen- tences. Specifically, we calculate and amplify the semantic difference between each sentence and other sentences, and apply the residual unit to deepen the differential amplifier archi- tecture. Furthermore, the corresponding objec- tive loss of the minority class is boosted by a weighted cross-entropy. In this way, our model pays more attention to the pivotal information of one sentence, that is different from previ- ous approaches which model all informative context in the source document. Experimen- tal results on two benchmark datasets show that our summarizer performs competitively against state-of-the-art methods. Our source code will be available on Github. 1 Introduction Single-document extractive summarization forms summary by copying and concatenating the most important spans (usually sentences) in a document. Sentence-level summarization is a very challeng- ing task, because it arguably requires an in-depth understanding of the source document sentences, and current automatic solutions are still far from human performance. Recent approaches frame the task as a sequence labeling problem, taking advan- tage of the success of neural network architectures. ∗Corresponding authors: Fang Fang and Shi Wang 10 20 30 40 50 Number of Sentences 20 25 30 35 40 ROUGE-1 ROUGE-2 ROUGE-L Figure 1: ROUGE score for documents with differ- ent length. The result is calculated on the test set of CNN/DM and the trained model is based on BERT. However, there are still two inherent obstacles for sentence-level extractive summarization: 1) It should be detrimental to keep tangential information (West et al., 2019). The intuitive limi- tation of those approaches is that they always prefer to model and retain all informative content from the source document. This goes against the fundamen- tal goal of summarization, which crucially needs to forget all but the “pivotal” information. Recently, the Information Bottleneck principle (Tishby et al., 2000; West et al., 2019) is introduced to incorpo- rate a tradeoff between information selection and pruning. Length penalty and the topic loss (Bazio- tis et al., 2019) are used in the autoencoding system to augment …

Content truncated to 500 words