Denoising Multi-Source Weak Supervision for Neural Text Classification

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3739–3754 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3739 Denoising Multi-Source Weak Supervision for Neural Text Classification Wendi Ren1, Yinghao Li1, Hanting Su2, David Kartchner1, Cassie Mitchell1 and Chao Zhang1 1 Georgia Institute of Technology, Atlanta, USA 2 Renmin University, Beijing, China {wren44, yinghaoli, david.kartchner, chaozhang}@gatech.edu suhanting@ruc.edu.cn cassie.mitchell@bme.gatech.edu Abstract We study the problem of learning neural text classifiers without using any labeled data, but only easy-to-provide rules as multiple weak supervision sources. This problem is challenging because rule-induced weak labels are often noisy and incomplete. To address these two challenges, we design a label de- noiser, which estimates the source reliability using a conditional soft attention mechanism and then reduces label noise by aggregating rule-annotated weak labels. The denoised pseudo labels then supervise a neural classifier to predicts soft labels for unmatched samples, which address the rule coverage issue. We evaluate our model on five benchmarks for sentiment, topic, and relation classifications. The results show that our model outper- forms state-of-the-art weakly-supervised and semi-supervised methods consistently, and achieves comparable performance with fully-supervised methods even without any labeled data. Our code can be found at https://github.com/weakrules/ Denoise-multi-weak-sources. 1 Introduction Many NLP tasks can be formulated as text classifi- cation problems, such as sentiment analysis (Bad- jatiya et al., 2017), topic classification (Zhang et al., 2015), relation extraction (Krebs et al., 2018) and question answering like slot filling (Pilehvar and Camacho-Collados, 2018). Recent years have wit- nessed the rapid development of deep neural net- works (DNNs) for this problem, from convolutional neural network (CNN, Kim, 2014; Kalchbrenner et al., 2014), recurrent neural network (RNN, Lai et al., 2015) to extra-large pre-trained language models (Devlin et al., 2019; Dai et al., 2019; Liu et al., 2019). DNNs’ power comes from their capa- bilities of fitting complex functions based on large- scale training data. However, in many scenarios, labeled data are limited, and manually annotating them at a large scale is prohibitively expensive. Weakly-supervised learning is an attractive ap- proach to address the data sparsity problem. It la- bels massive data with cheap labeling sources such as heuristic rules or knowledge bases. However, the major challenges of using weak supervision for text classification are two-fold: 1) the created labels are highly noisy and imprecise. The label noise issue arises because heuristic rules are often too simple to capture rich contexts and complex semantics for texts; 2) each source only covers a small portion of the data, leaving the labels incomplete. Seed rules have limited coverage because they are de- fined over the most frequent keywords but real-life text corpora often have long-tail distributions, so the instances containing only long-tail keywords cannot be annotated. Existing works (Ratner et al., 2017; Meng et al., 2018; Zamani et al., 2018; Awasthi et al., 2020) attempt to use weak supervision for deep text clas- sification. Ratner et al. (2017) proposes a data programming method that uses labeling functions to automatically label data and then trains …

Content truncated to 500 words