STGN: an Implicit Regularization Method for Learning with Noisy Labels in Natural Language Processing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7587 - 7598 December 7-11, 2022 ©2022 Association for Computational Linguistics STGN: an Implicit Regularization Method for Learning with Noisy Labels in Natural Language Processing Tingting Wu1, Xiao Ding1∗, Minji Tang1, Hao Zhang2, Bing Qin1, Ting Liu1 1Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China {ttwu, xding, mjtang,qinb tliu}@ir.hit.edu.cn 2Faculty of Computing, Harbin Institute of Technology, China zhh1000@hit.edu.cn Abstract Noisy labels are ubiquitous in natural language processing (NLP) tasks. Existing work, namely learning with noisy labels in NLP, is often lim- ited to dedicated tasks or specific training pro- cedures, making it hard to be widely used. To address this issue, SGD noise has been ex- plored to provide a more general way to al- leviate the effect of noisy labels by involving benign noise in the process of stochastic gra- dient descent. However, previous studies exert identical perturbation for all samples, which may cause overfitting on incorrect ones or opti- mizing correct ones inadequately. To facilitate this, we propose a novel stochastic tailor-made gradient noise (STGN), mitigating the effect of inherent label noise by introducing tailor-made benign noise for each sample. Specifically, we investigate multiple principles to precisely and stably discriminate correct samples from incor- rect ones and thus apply different intensities of perturbation to them. A detailed theoretical analysis shows that STGN has good properties, beneficial for model generalization. Experi- ments on three different NLP tasks demonstrate the effectiveness and versatility of STGN. Also, STGN can boost existing robust training meth- ods.1 1 Introduction Deep neural networks (DNNs) have notably suc- ceeded on various natural language processing (NLP) tasks. However, noisy labels induced by labeling errors are prevalent in a wide range of corpora (Jia et al., 2019; Alt et al., 2020). DNNs can gradually fit correct data (i.e., noise-free data) and eventually memorize all data, including incor- rect data (i.e., data with label noise) (Zhang et al., 2017), which would affect the generalization of deep models and result in the performance degrada- tion. Hence, it is pressing and necessary to develop ∗Corresponding author. 1The code is released at https://github.com/tangminji/ STGN-sst. 0 1 2 3 4 5 6 7 8 9 Epoch 0 5 10 15 20 Ratio of memorizing incorrect samples SLN σy = 0.5 SLN σy = 1.0 STGN 0 1 2 3 4 5 6 7 8 9 Epoch 10 20 30 40 50 Ratio of learning correct samples SLN σy = 0.5 SLN σy = 1.0 STGN Figure 1: Ratios of memorizing incorrect samples (Left, i.e., training accuracy from incorrect ones (%), smaller is better) and learning correct samples (Right, i.e., train- ing accuracy from correct ones (%), larger is better) during training on SST-5 with 40% label noise. Two ratios together form the training accuracy (%). Our method STGN gains more training accuracy on correct samples than existing method SLN but memorizes much less label noise. robust methods in NLP to mitigate the impact of …

Content truncated to 500 words