Instance-adaptive training with noise-robust losses against noisy labels

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5647–5663 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5647 Instance-adaptive training with noise-robust losses against noisy labels Lifeng Jin Linfeng Song Kun Xu Dong Yu Tencent AI Lab Bellevue, WA, USA (lifengjin,lfsong,kxkunxu,dyu)@tencent.com Abstract In order to alleviate the huge demand for an- notated datasets for different tasks, many re- cent natural language processing datasets have adopted automated pipelines for fast-tracking usable data. However, model training with such datasets poses a challenge because pop- ular optimization objectives are not robust to label noise induced in the annotation genera- tion process. Several noise-robust losses have been proposed and evaluated on tasks in com- puter vision, but they generally use a sin- gle dataset-wise hyperparamter to control the strength of noise resistance. This work pro- poses novel instance-adaptive training frame- works to change dataset-wise hyperparame- ters of noise resistance in such losses to be instance-specific. Such instance-specific noise resistance hyperparameters are predicted by special instance-level label quality predictors, which are trained along with the main mod- els. Experiments on noisy and corrupted NLP datasets show that proposed instance-adaptive training frameworks help increase the noise- robustness provided by such losses, promot- ing the use of the frameworks and associated losses in training NLP models with noisy data. 1 Introduction The wide availability of neural network models has allowed development of novel and complex natural language processing tasks, many of which are in low-resource settings. With new defini- tions of tasks comes challenges of constructing new datasets, which is still an expensive and time- intensive endeavor. Many researchers have resorted to constructing datasets by using completely auto- mated pipelines (e.g. Lan et al., 2017; Joshi et al., 2017; Paul et al., 2019; Lange et al., 2019; Sousa et al., 2019; Wu et al., 2020). However, silver labels collected this way are still quite noisy compared to expert annotation. Because such methods have been gaining popularity and practicality, it is impor- tant to explore ways to ensure good performance in spite of noisy labels in training data. The widely-used cross entropy (CE) loss as the optimization objective in classification tasks has been shown to overfit to label noise (Ghosh et al., 2017). Several noise-robust losses have been de- signed for training models with noisy labels (Reed et al., 2015; Zhang and Sabuncu, 2018; Wang et al., 2019c), which were a convenient way to address the noisy label issue and shown to be more robust than CE. Experiments are usually conducted on com- puter vision datasets such as CIFAR (Krizhevsky, 2009) and MNIST (LeCun et al., 1998). These noise-robust losses usually have hyperpa- rameters for determining the strength of the noise- robustness at the dataset level. However, individual training instances may have different amounts of noise, derived from biases within models used in the automated pipeline. Moreover, noisy labels in natural language datasets potentially pose a greater challenge because instances of the same true label may not share similar surface features. Therefore, this …

Content truncated to 500 words