Weaker Than You Think: A Critical Look at Weakly Supervised Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14229–14253 July 9-14, 2023 ©2023 Association for Computational Linguistics Weaker Than You Think: A Critical Look at Weakly Supervised Learning Dawei Zhu1 Xiaoyu Shen2∗ Marius Mosbach1 Andreas Stephan3 Dietrich Klakow1 1Saarland University, Saarland Informatics Campus 2Amazon Alexa AI 3University of Vienna dzhu@lsv.uni-saarland.de Abstract Weakly supervised learning is a popular ap- proach for training machine learning models in low-resource settings. Instead of requesting high-quality yet costly human annotations, it allows training models with noisy annotations obtained from various weak sources. Recently, many sophisticated approaches have been pro- posed for robust training under label noise, re- porting impressive results. In this paper, we re- visit the setup of these approaches and find that the benefits brought by these approaches are significantly overestimated. Specifically, we find that the success of existing weakly super- vised learning approaches heavily relies on the availability of clean validation samples which, as we show, can be leveraged much more ef- ficiently by simply training on them. After using these clean labels in training, the advan- tages of using these sophisticated approaches are mostly wiped out. This remains true even when reducing the size of the available clean data to just five samples per class, making these approaches impractical. To understand the true value of weakly supervised learning, we thor- oughly analyze diverse NLP datasets and tasks to ascertain when and why weakly supervised approaches work. Based on our findings, we provide recommendations for future research.1 1 Introduction Weakly supervised learning (WSL) is one of the most popular approaches for alleviating the anno- tation bottleneck in machine learning. Instead of collecting expensive clean annotations, it leverages weak labels from various weak labeling sources such as heuristic rules, knowledge bases or lower- quality crowdsourcing (Ratner et al., 2017). These weak labels are inexpensive to obtain, but are often noisy and inherit biases from their sources. Deep learning models trained on such noisy data without ∗Work done outside Amazon. 1Our code is available at: https://github.com/ uds-lsv/critical_wsl SemEval AGNews OntoNotes IMDb Yelp ChemProt CoNLL-03 TREC 0 5 10 15 20 25 30 35 Performance Improvement (%) COSINE BOND L2R MLC MetaWN Denoise Train on the clean training set (performance upper bound) Train on the clean validation set Weak labels Figure 1: Performance improvement over weak la- bels on the test sets. Each point represents the average performance improvement of one approach over five runs. On various NLP datasets, weakly supervised meth- ods (dots) outperform weak labels (blue line) on the test sets. However, simply fine-tuning on the available clean validation data (light green crosses) outperforms all sophisticated weakly supervised methods in almost all cases. See Appendix D.2 for experimental details. regularization can easily overfit to the noisy la- bels (Zhang et al., 2017; Tänzer et al., 2022). Many advanced WSL techniques have recently been pro- posed to combat the noise in weak labels, and sig- nificant progress has been reported. On certain datasets, they even …

Content truncated to 500 words