Noise Learning for Text Classification: A Benchmark

Proceedings of the 29th International Conference on Computational Linguistics, pages 4557–4567 October 12–17, 2022. 4557 Noise Learning for Text Classification: A Benchmark Bo Liu School of Sofrware Engineering, Xi’an Jiaotong University the CETC Key Laboratory of Smart City Model Simulation and Intelligent Technology The Smart City Research Institute of CETC and National Center for Applied Mathematics Shenzhen(NCAMS) boliu@stu.xjtu.edu.cn Wandi Xu Northeastern University xuwandi@stumail.neu.edu.cn Yuejia Xiang Tencent yuejiaxiang@tencent.com Xiaojun Wu the CETC Key Laboratory of Smart City Model Simulation and Intelligent Technology The Smart City Research Institute of CETC and National Center for Applied Mathematics Shenzhen(NCAMS) wuxiaojun@cetc.com.cn Lejian He Cornell University lh628@cornell.edu Bowen Zhang Shenzhen Technology University zhang_bo_wen@foxmail.com Li Zhu∗ School of Sofrware Engineering, Xi’an Jiaotong University zhuli@xjtu.edu.cn Abstract Noise Learning is important in the task of text classification which depends on massive la- beled data that could be error-prone. However, we find that noise learning in text classifica- tion is relatively underdeveloped: 1. many methods that have been proven effective in the image domain are not explored in text clas- sification, 2. it is difficult to conduct a fair comparison between previous studies as they do experiments in different noise settings. In this work, we adapt four state-of-the-art meth- ods of noise learning from the image domain to text classification. Moreover, we conduct comprehensive experiments on our benchmark of noise learning with seven commonly-used methods, four datasets, and five noise modes. Additionally, most previous works are based on an implicit hypothesis that the commonly- used datasets such as TREC, Ag-News and Chnsenticorp contain no errors. However, these datasets indeed contain 0.61% to 15.77% noise labels which we define as intrinsic noise that can cause inaccurate evaluation. Therefore, we build a new dataset Golden-Chnsenticorp (G-Chnsenticorp) without intrinsic noise to more accurately compare the effects of differ- ent noise learning methods. To the best of our knowledge, this is the first benchmark of noise learning for text classification. ∗† Corresponding author. 1 Introduction The fast development of text classification cannot be achieved without massive labeled data resources, espe- cially for supervised embedding-based methods. How- ever, not all training data are correctly labeled in practice (Wang et al., 2018; Zlateski et al., 2018). These incor- rectly labeled data are called noisy labels. To alleviate the interference caused by noisy labels, many noise learning methods have been proposed (Rolnick et al., 2017; Veit et al., 2017; Jiang et al., 2018; Yang Liu, 2019; Li et al., 2020; Curtis G. Northcutt, 2020; Garg et al., 2021). Although both CV and NLP domains have serious label noise problems, these work are mainly fo- cused on the CV domain, only (Garg et al., 2021) is dedicated to NLP domain. So in order to support the development of noise learning in NLP, we would like to propose a noise learning benchmark in the field of text classification. We find that the previous studies in noise learning for text classification tasks have two weaknesses. 1. The implicit hypothesis is unreasonable. 2. They lack horizontal comparison. Unreasonable implicit hypothesis. The …

Content truncated to 500 words