Combining Distant and Direct Supervision for Neural Relation Extraction

Proceedings of NAACL-HLT 2019, pages 1858–1867 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1858 Combining Distant and Direct Supervision for Neural Relation Extraction Iz Beltagy Kyle Lo Waleed Ammar Allen Institute for Artificial Intelligence, Seattle, WA, USA {beltagy,kylel,waleeda}@allenai.org Abstract In relation extraction with distant supervision, noisy labels make it difficult to train quality models. Previous neural models addressed this problem using an attention mechanism that at- tends to sentences that are likely to express the relations. We improve such models by combining the distant supervision data with an additional directly-supervised data, which we use as supervision for the attention weights. We find that joint training on both types of supervision leads to a better model because it improves the model’s ability to identify noisy sentences. In addition, we find that sigmoidal attention weights with max pooling achieves better performance over the commonly used weighted average attention in this setup. Our proposed method1 achieves a new state-of-the- art result on the widely used FB-NYT dataset. 1 Introduction Early work in relation extraction from text used directly supervised methods, e.g., Bunescu and Mooney (2005), which motivated the development of relatively small datasets with sentence-level an- notations such as ACE 2004/2005, BioInfer and SemEval 2010 Task 8. Recognizing the difficulty of annotating text with relations, especially when the number of relation types of interest is large, others (Mintz et al., 2009; Craven and Kumlien, 1999) introduced the distant supervision approach of relation extraction, where a knowledge base (KB) and a text corpus are used to automatically generate a large dataset of labeled bags of sen- tences (a set of sentences that might express the relation) which are then used to train a relation classifier. The large number of labeled instances produced with distant supervision make it a prac- tical alternative to manual annotations. 1https://github.com/allenai/comb_dist_ direct_relex/ However, distant supervision implicitly as- sumes that all the KB facts are mentioned in the text (at least one of the sentences in each bag expresses the relation) and that all relevant facts are in the KB (use entities that are not related in the KB as negative examples). These two as- sumptions are generally not true, which introduces many noisy examples in the training set. Although many methods have been proposed to deal with such noisy training data (e.g., Hoffmann et al., 2011; Surdeanu et al., 2012; Roth et al., 2013; Fan et al., 2014; Zeng et al., 2015; Jiang et al., 2016; Liu et al., 2017), a rather obvious approach has been understudied: combine distant supervi- sion data with additional direct supervision. Intu- itively, directly supervising the model can improve its performance by helping it identify which of the input sentences for a given pair of entities are more likely to express a relation. A straightforward way to combine distant and direct supervision is to concatenate instances from both datasets into one large dataset. We show in Section 4.2 that this approach doesn’t help the model. Pershina et al. (2014) …

Content truncated to 500 words