Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming

Findings of the Association for Computational Linguistics: ACL 2022, pages 1188 - 1202 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming Ayush Maheshwari 1∗, Krishnateja Killamsetty 2∗, Ganesh Ramakrishnan 1, Rishabh Iyer2, Marina Danilevsky3 and Lucian Popa3 1Indian Institute of Technology Bombay, India 2 The University of Texas at Dallas 3 IBM Research – Almaden {ayusham, ganesh}@cse.iitb.ac.in {krishnateja.killamsetty, rishabh.iyer}@utdallas.edu {mdanile, lpopa}@us.ibm.com Abstract A critical bottleneck in supervised machine learning is the need for large amounts of labeled data which is expensive and time-consuming to obtain. Although a small amount of labeled data cannot be used to train a model, it can be used effectively for the generation of human- interpretable labeling functions (LFs). These LFs, in turn, have been used to generate a large amount of additional noisy labeled data in a paradigm that is now commonly referred to as data programming. Previous methods of gen- erating LFs do not attempt to use the given la- beled data further to train a model, thus missing opportunities for improving performance. Ad- ditionally, since the LFs are generated automat- ically, they are likely to be noisy, and naively aggregating these LFs can lead to suboptimal results. In this work, we propose an LF-based bi-level optimization framework WISDOM to solve these two critical limitations. WISDOM learns a joint model on the (same) labeled dataset used for LF induction along with any un- labeled data in a semi-supervised manner, and more critically, reweighs each LF according to its goodness, influencing its contribution to the semi-supervised loss using a robust bi-level op- timization algorithm. We show that WISDOM significantly outperforms prior approaches on several text classification datasets. The source code can be found at https://github.com/ ayushbits/robust-aggregate-lfs. 1 Introduction Supervised machine learning approaches require large amounts of labeled data to train robust ma- chine learning models. Human-annotated gold la- bels have become increasingly important to modern machine learning systems for tasks such as spam detection, (movie) genre classification, sequence la- beling, etc. The creation of labeled data is, however, a time-consuming and costly process that requires large amounts of human labor. Together with the ∗Equal contribution heavy reliance on labeled data for training models, this serves as a deterrent to achieving comparable performance on new tasks. As a result, various methods such as semi-supervision, distant super- vision, and crowdsourcing have been proposed to reduce reliance on human annotation. In particular, several recent data programming approaches (Bach et al., 2019; Maheshwari et al., 2021; Chatterjee et al., 2020; Awasthi et al., 2020) have proposed the use of human-crafted labeling functions to weakly associate labels with the train- ing data. Typically, users encode supervision as rules/guides/heuristics in the form of labeling func- tions (LFs) that assign noisy labels to the unlabeled data, thus reducing dependence on human-labeled data. The noisy labels were aggregated using La- bel aggregators, which often employ generative models, to assign a label to the data instance. Ex- amples of label aggregators are SNORKEL (Ratner …

Content truncated to 500 words