Context-based Virtual Adversarial Training for Text Classification with Noisy Labels

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6139–6146 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6139 Context-based Virtual Adversarial Training for Text Classification with Noisy Labels Do-Myoung Lee1∗, Yeachan Kim2∗, Chang-gyun Seo3 1ShinhanCard, 2Deargen Inc., 3GC Company 1domyoung89@shinhan.com, 2yeachan@deargen.me, 3ocean@gccompany.co.kr Abstract Deep neural networks (DNNs) have a high capacity to completely memorize noisy labels given sufficient training time, and its memorization unfortunately leads to performance degradation. Recently, virtual adversarial training (VAT) attracts attention as it could further improve the generalization of DNNs in semi-supervised learning. The driving force behind VAT is to prevent the models from overffiting to data points by enforcing consistency between the inputs and the perturbed inputs. These strategy could be helpful in learning from noisy labels if it prevents neural models from learning noisy samples while encouraging the models to generalize clean samples. In this paper, we propose context-based virtual adversarial training (ConVAT) to prevent a text classifier from overfitting to noisy labels. Unlike the previous works, the proposed method performs the adversarial training in the context level rather than the inputs. It makes the classifier not only learn its label but also its contextual neighbors, which alleviate the learning from noisy labels by preserving contextual semantics on each data point. We conduct extensive experiments on four text classification datasets with two types of label noises. Comprehensive experimental results clearly show that the proposed method works quite well even with extremely noisy settings. Keywords: Text Classification, Learning with Noisy Labels 1. Introduction Deep neural networks (DNNs) have shown human- level performance in various domains, such as image classification, machine translation, and speech recog- nition. To achieve such an ability, it is indisputably evident that we have to collect a large amount of train- ing data. As labeling such data is laborious and ex- pensive, previous works utilize search engine (Blum et al., 2003; Li et al., 2017) or crowdsourcing (Yan et al., 2014; Yu et al., 2018) to collect labeled dataset. Un- fortunately, these low-cost approaches introduce low- quality annotations with label noise. It causes DNNs to completely memorize such label noises (i.e., nearly 100% training accuracy) (Zhang et al., 2016), deterio- rating generalization capability (Fr´enay and Verleysen, 2013; Sukhbaatar et al., 2014). To deal with label noises, recent works primar- ily propose loss correction approaches. These meth- ods directly correct loss function (e.g., cross-entropy, mean squared error) or the probabilities used to com- pute it. For example, (Zhang et al., 2016) learns sam- ple weighting scheme via an auxiliary network (called MentorNet) and applies it to noisy labels such that cor- rupt data could get nearly zero sample weights on the loss function. On the one hand, (Han et al., 2018) utilizes an intervention between two same networks with small-loss samples which are considered as clean. However, there exists only a single work to handle label noise on natural language. (Jindal et al., 2019) utilize a noise transition matrix for text classification. This …

Content truncated to 500 words