Revisiting Distant Supervision for Relation Extraction

Revisiting Distant Supervision for Relation Extraction Tingsong Jiang†, Jing Liu‡, Chin-Yew Lin§, Zhifang Sui† †Key Laboratory of Computational Linguistics, Ministry of Education School of Electronics Engineering and Computer Science, Peking University ‡Baidu Inc., Beijing, China §Microsoft Research tingsong@pku.edu.cn, liujing46@baidu.com, cyl@microsoft.com,szf@pku.edu.cn Abstract Distant supervision has been widely used in the task of relation extraction (RE). However, when we carefully examine the experimental settings of previous work, we find two issues: (i) The compared models were trained on different training datasets. (ii) The existing testing data contains noise and bias issues. These issues may affect the conclusions in previous work. In this paper, our primary aim is to re-examine the distant supervision-based approaches under the experimental settings without the above issues. We approach this by training models on the same dataset and creating a new testing dataset annotated by the workers on Amzaon Mechanical Turk. We draw new conclusions based on the new testing dataset. The new testing data can be obtained from http://aka.ms/relationie. Keywords: relation extraction, distant supervision 1. Introduction In recent years, knowledge bases (KBs) like Freebase (Bol- lacker et al., 2008), DBpedia (Lehmann et al., 2015) and NELL (Carlson et al., 2010) have become extremely use- ful resources for many natural language processing (NLP) tasks. These KBs are mostly composed of relational facts between entities, which are typically represented as triples with the format (head entity, relation, tail entity), e.g., (Paris, capitalOf, France). Although existing KBs may contain billions of relational facts, they are still far from complete and missing many crucial facts. To en- rich KBs, relation extraction (RE), i.e., the task of extract- ing relations between entities from plain texts, has thus at- tracted increasing attention. For example, here is a sen- tence: Paris is the capital city of France. where Paris and France are two entity mentions. A re- lation extractor or classifier takes the sentence and the two entity mentions as inputs, and determines the semantic rela- tion that it expresses, if any. In the above example, a correct prediction may be capitalOf relation. Most existing approaches formulate RE as a classification task and use supervised learning on relation-specific train- ing data, which is very expensive to acquire. To address this issue, distant supervision is proposed to automatically gen- erate training data via aligning facts in KBs and texts (Wu and Weld, 2007; Mintz et al., 2009). The distant supervi- sion assumption is that if two entities preserve a relation in a KB, then all sentences that mention the two entities express the relation. Figure 1 shows an example of the automatic labeling of data via distant supervision. In this example, Paris and France are two entities with a rela- tion type capitalOf in a KB. All sentences contain these two entities are labeled with capitalOf. Although dis- tant supervision provides a cheap way to automatically la- bel training data, it leads to a noise problem with the data. The noisy data can be mainly classified into two categories: This work was done when Tingsong …

Content truncated to 500 words