Posterior-regularized REINFORCE for Instance Selection in Distant Supervision

Proceedings of NAACL-HLT 2019, pages 2831–2835 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2831 Posterior-regularized REINFORCE for Instance Selection in Distant Supervision Qi Zhang1, Siliang Tang1∗, Xiang Ren3,Fei Wu1, Shiliang Pu2 & Yueting Zhuang1 1Zhejiang University, 2Hikvision, 3University of Southern California, {zhangqihit,siliang,wufei,yzhuang}@zju.edu.cn, pushiliang@hikvision.com, xiangren@usc.edu Abstract This paper provides a new way to improve the efficiency of the REINFORCE training pro- cess. We apply it to the task of instance selec- tion in distant supervision. Modeling the in- stance selection in one bag as a sequential de- cision process, a reinforcement learning agent is trained to determine whether an instance is valuable or not and construct a new bag with less noisy instances. However unbiased methods, such as REINFORCE, could usually take much time to train. This paper adopts posterior regularization (PR) to integrate some domain-specific rules in instance selection us- ing REINFORCE. As the experiment results show, this method remarkably improves the performance of the relation classifier trained on cleaned distant supervision dataset as well as the efficiency of the REINFORCE training. 1 Introduction Relation extraction is a fundamental work in natu- ral language processing. Detecting and classifying the relation between entity pairs from the unstruc- tured document, it can support many other tasks such as question answering. While relation extraction requires lots of la- beled data and make methods labor intensive, (Mintz et al., 2009) proposes distant supervision (DS), a widely used automatic annotating way. In distant supervision, knowledge base (KB) , such as Freebase, is aligned with nature documents. In this way, the sentences which contain an en- tity pair in KB all express the exact relation that the entity pair has in KB. We usually call the set of instances that contain the same entity pair a bag. In this way, the training instances can be di- vided into N bags B = {B1, B2, ..., BN}. Each bag Bk are corresponding to an unique entity pair ∗Corresponding author Ek = (ek 1, ek 2) and contains a sequence of in- stances {xk 1, xk 2, ..., xk |Bk|} . However, distant su- pervision may suffer a wrong label problem. In other words, the instances in one bag may not ac- tually have the relation. To resolve the wrong label problem, just like Fig.2 shows, (Feng et al., 2018) model the instance selection task in one bag Bk as a sequential deci- sion process and train an agent π(a|s, θπ) denoting the probability Pπ(At = a, |St = s, θt = θπ) that action a is taken at time t given that the agent is in state s with parameter vector θπ by REINFORCE algorithm (Sutton and Barto, 1998). The action a can only be 0 or 1 indicating whether an instance xk i is truly expressing the relation and whether it should be selected and added to the new bag Bk. The state s is determined by the entity pair cor- responding to the bag, the candidate instance to be …

Content truncated to 500 words