Few Clean Instances Help Denoising Distant Supervision

Proceedings of the 29th International Conference on Computational Linguistics, pages 2528–2539 October 12–17, 2022. 2528 Few Clean Instances Help Denoising Distant Supervision Yufang Liu∗1, Ziyin Huang∗1, Yijun Wang2,Changzhi Sun3, Man Lan1, Yuanbin Wu1, Xiaofeng Mou4 and Ding Wang4 1School of Computer Science and Technology, East China Normal University 2Department of Computer Science and Engineering, Shanghai Jiao Tong University 3Bytedance AI Lab, 4AI Innovation Center, Midea Group {yfliu.antlp, zyhuang.cs}@gmail.com, ybwu@cs.ecnu.edu.cn Abstract Existing distantly supervised relation extrac- tors usually rely on noisy data for both model training and evaluation, which may lead to garbage-in-garbage-out systems. To alleviate the problem, we study whether a small clean dataset could help improve the quality of dis- tantly supervised models. We show that be- sides getting a more convincing evaluation of models, a small clean dataset also helps us to build more robust denoising models. Specif- ically, we propose a new criterion for clean instance selection based on influence functions. It collects sample-level evidence for recogniz- ing good instances (which is more informative than loss-level evidence). We also propose a teacher-student mechanism for controlling pu- rity of intermediate results when bootstrapping the clean set. The whole approach is model- agnostic and demonstrates strong performances on both denoising real (NYT) and synthetic noisy datasets.1 1 Introduction Distant supervision was introduced to tackle the lacking training data problem in information ex- traction tasks (Mintz et al., 2009). By aligning relation triples in knowledge bases (KB) with free texts, it automatically builds labelled sentence in- stances and easily extends the scale of training set to hundreds of thousands samples. Due to this great scalability, distantly supervised entity relation ex- tractors have been extensively studied in the past decade. Like other weak signals, the major problem about these automatically generated datasets is la- bel noise: not all aligned sentences carry the same semantic of a KB triple (e.g., not all sentences con- taining “Obama” and “United States” express a “born in” relation). Some applications (e.g., slot *These authors contributed equally. 1Our codes are publicly available at: https://github. com/Airuibadi/IF_DSRE. filling of the TAC KBP track (Ji and Grishman, 2011)) could be less affected with the help of in- stance bags, which only needs to seek one correct instance among a bag of aligned sentences. For a more general setting which aims to correctly de- tect relations on individual sentences (Miwa and Bansal, 2016; Sun et al., 2018; Wadden et al., 2019; Wang et al., 2020), however, the noisy labels make both learning and evaluation of models vulnerable: we may draw a flawed conclusion by using a dirty test set for a model learned with a dirty training set. Many methods have been proposed to reduce noise labels (denoise) in distant supervision. For bag-level applications, models often rely on atten- tion scores to either filter bad instances inside a bag (intra-bag attentions, Lin et al. 2016) or filter bags full of noisy instances (inter-bag attentions, Ye and Ling 2019). The dilemma there is that, while we ex- pect attention scores to indicate …

Content truncated to 500 words