Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 430–439 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1040 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 430–439 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1040 Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix Bingfeng Luo1, Yansong Feng∗1, Zheng Wang2, Zhanxing Zhu3, Songfang Huang4, Rui Yan1 and Dongyan Zhao1 1ICST, Peking University, China 2School of Computing and Communications, Lancaster University, UK 3Peking University, China 4IBM China Research Lab, China {bf luo,fengyansong,zhanxing.zhu,ruiyan,zhaody}@pku.edu.cn z.wang@lancaster.ac.uk huangsf@cn.ibm.com Abstract Distant supervision significantly reduces human efforts in building training data for many classification tasks. While promis- ing, this technique often introduces noise to the generated training data, which can severely affect the model performance. In this paper, we take a deep look at the appli- cation of distant supervision in relation ex- traction. We show that the dynamic transi- tion matrix can effectively characterize the noise in the training data built by distant supervision. The transition matrix can be effectively trained using a novel curricu- lum learning based method without any di- rect supervision about the noise. We thor- oughly evaluate our approach under a wide range of extraction scenarios. Experimen- tal results show that our approach consis- tently improves the extraction results and outperforms the state-of-the-art in various evaluation scenarios. 1 Introduction Distant supervision (DS) is rapidly emerging as a viable means for supporting various classification tasks – from relation extraction (Mintz et al., 2009) and sentiment classification (Go et al., 2009) to cross-lingual semantic analysis (Fang and Cohn, 2016). By using knowledge learned from seed ex- amples to label data, DS automatically prepares large scale training data for these tasks. While promising, DS does not guarantee per- fect results and often introduces noise to the gener- ated data. In the context of relation extraction, DS works by considering sentences containing both the subject and object of a <subj, rel, obj> triple as its supports. However, the generated data are not always perfect. For instance, DS could match the knowledge base (KB) triple, <Donald Trump, born-in, New York> in false positive contexts like Donald Trump worked in New York City. Prior works (Takamatsu et al., 2012; Ritter et al., 2013) show that DS often mistakenly labels real posi- tive instances as negative (false negative) or versa vice (false positive), and there could be confu- sions among positive labels as well. These noises can severely affect training and lead to poorly- performing models. Tackling the noisy data problem of DS is non- trivial, since there usually lacks of explicit super- vision to capture the noise. Previous works have tried to remove sentences containing unreliable syntactic patterns (Takamatsu et al., 2012), design new models to capture certain types of noise or aggregate multiple predictions under the at-least- one assumption that at least one of the aligned sentences supports the triple in KB (Riedel …

Content truncated to 500 words