Noise-injected Consistency Training and Entropy-constrained Pseudo Labeling for Semi-supervised Extractive Summarization

Proceedings of the 29th International Conference on Computational Linguistics, pages 6447–6456 October 12–17, 2022. 6447 Noise-injected Consistency Training and Entropy-constrained Pseudo Labeling for Semi-supervised Extractive Summarization Yiming Wang1,2,3,†, Qianren Mao1,2,†, Junnan Liu1,2, Weifeng Jiang1,2, Hongdong Zhu1,2, Jianxin Li1,2,∗ 1 Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, China. 2 The State Key Laboratory of Software Development Environment, Beihang University, China. 3 Institute of Artificial Intelligence, Beihang University, China. {wangym,maoqr,liujn,jiangwf,zhuhd,lijx}@act.buaa.edu.cn Abstract Labeling large amounts of extractive summa- rization data is often prohibitive expensive due to time, financial, and expertise constraints, which poses great challenges to incorporat- ing summarization system in practical applica- tions. This limitation can be overcome by semi- supervised approaches: consistency-training and pseudo-labeling to make full use of un- labeled data. Researches on the two, how- ever, are conducted independently, and very few works try to connect them. In this paper, we first use the noise-injected consistency train- ing paradigm to regularize model predictions. Subsequently, we propose a novel entropy- constrained pseudo labeling strategy to obtain high-confidence labels from unlabeled predic- tions, which can obtain high-confidence labels from unlabeled predictions by comparing the entropy of supervised and unsupervised predic- tions. By combining consistency training and pseudo-labeling, this framework enforce a low- density separation between classes, which de- cently improves the performance of supervised learning over an insufficient labeled extractive summarization dataset. 1 Introduction Text summarization is a challenging task that gen- erates a condensed version of an input text that captures the original’s core meaning. In this pa- per, we focus on extractive summarization since it usually generates semantically and grammatically correct sentences (Liu and Lapata, 2019; Zhong et al., 2019a; Zhou et al., 2020; Zhong et al., 2020). The extractive summarization typically requires to label each sentence in massive documents for model training. However, acquiring well-annotated labels is a costly process, and labeling every sen- tence would be labor-intensive and error-prone due to subjective judgments of human. This motivates † These two authors contributed equally. ∗ Jianxin Li is the corresponding author. research on Semi-Supervised Learning (SSL) meth- ods which focus on how to effectively utilizes abun- dant unlabeled data, to further improve extractive summarization performances. Towards this goal, we first revisit an effective semi-supervised method, consistency training (Xie et al., 2020a). The consistency training leverages voluminous unlabeled data and employs advanced data augmentation methods to generate diverse and realistic noisy source text, forcing the model to be consistent with these noises. The consistency training has been extensively applied on the classi- fication problems, such as Text Classification (Xie et al., 2020a; Liu et al., 2021a), Image Recogni- tion (Laine and Aila, 2017; Tarvainen and Valpola, 2017; Miyato et al., 2019; Verma et al., 2019; Xie et al., 2020b). However, how does the consistency training work on semi-supervised extractive sum- marization tasks is still unclear. We investigate the noise-injected consistency training for semi-supervised extractive summariza- tion to encourage a consistent reason of model de- cision (summary and non-summary) under data perturbation. This framework makes …

Content truncated to 500 words