Self-Training with Weak Supervision

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 845–863 June 6–11, 2021. ©2021 Association for Computational Linguistics 845 Self-Training with Weak Supervision Giannis Karamanolakis§ ∗ Subhabrata Mukherjee† Guoqing Zheng† Ahmed Hassan Awadallah† §Columbia University, New York †Microsoft Research gkaraman@cs.columbia.edu {submukhe, zheng, hassanam}@microsoft.com Abstract State-of-the-art deep neural networks require large-scale labeled training data that is often expensive to obtain or not available for many tasks. Weak supervision in the form of domain- specific rules has been shown to be useful in such settings to automatically generate weakly labeled training data. However, learning with weak rules is challenging due to their inherent heuristic and noisy nature. An additional chal- lenge is rule coverage and overlap, where prior work on weak supervision only considers in- stances that are covered by weak rules, thus leaving valuable unlabeled data behind. In this work, we develop a weak supervision framework (ASTRA1) that leverages all the available data for a given task. To this end, we leverage task-specific unlabeled data through self-training with a model (student) that con- siders contextualized representations and pre- dicts pseudo-labels for instances that may not be covered by weak rules. We further develop a rule attention network (teacher) that learns how to aggregate student pseudo-labels with weak rule labels, conditioned on their fidelity and the underlying context of an instance. Fi- nally, we construct a semi-supervised learn- ing objective for end-to-end training with unla- beled data, domain-specific rules, and a small amount of labeled data. Extensive experi- ments on six benchmark datasets for text clas- sification demonstrate the effectiveness of our approach with significant improvements over state-of-the-art baselines. 1 Introduction The success of state-of-the-art neural networks cru- cially hinges on the availability of large amounts of annotated training data. While recent advances on language model pre-training (Peters et al., 2018; ∗Most of the work was done while the first author was an intern at Microsoft Research. 1ASTRA: weAkly-supervised Self-TRAining. Our code is publicly available at https://github.com/ microsoft/ASTRA. Student Weak Rules RegEx Patterns Domain Heuristics Knowledge Bases Lexicons ( |ˆ)(free)[ˆ\w] -> SPAM (\w+ )(talk) -> HAM Teacher (RAN) Unlabeled Data Few Labeled Data Figure 1: Our weak supervision framework, ASTRA, leverages domain-specific rules, a large amount of (task-specific) unlabeled data, and a small amount of labeled data via iterative self-training. Devlin et al., 2019; Radford et al., 2019) reduce the annotation bottleneck, they still require large amounts of labeled data for obtaining state-of-the- art performances on downstream tasks. However, it is prohibitively expensive to obtain large-scale labeled data for every new task, therefore posing a significant challenge for supervised learning. In order to mitigate labeled data scarcity, re- cent works have tapped into weak or noisy sources of supervision, such as regular expression pat- terns (Augenstein et al., 2016), class-indicative keywords (Ren et al., 2018b; Karamanolakis et al., 2019), alignment rules over existing knowledge bases (Mintz et al., 2009; Xu et al., 2013) or heuris- tic labeling functions (Ratner et al., …

Content truncated to 500 words