Dual Supervision Framework for Relation Extraction with Distant Supervision and Human Annotation

Proceedings of the 28th International Conference on Computational Linguistics, pages 6411–6423 Barcelona, Spain (Online), December 8-13, 2020 6411 Dual Supervision Framework for Relation Extraction with Distant Supervision and Human Annotation Woohwan Jung Seoul National University Seoul, Korea whjung@kdd.snu.ac.kr Kyuseok Shim Seoul National University Seoul, Korea kshim@snu.ac.kr Abstract Relation extraction (RE) has been extensively studied due to its importance in real-world appli- cations such as knowledge base construction and question answering. Most of the existing works train the models on either distantly supervised data or human-annotated data. To take advantage of the high accuracy of human annotation and the cheap cost of distant supervision, we propose the dual supervision framework which effectively utilizes both types of data. However, simply combining the two types of data to train a RE model may decrease the prediction accuracy since distant supervision has labeling bias. We employ two separate prediction networks HA-Net and DS-Net to predict the labels by human annotation and distant supervision, respectively, to pre- vent the degradation of accuracy by the incorrect labeling of distant supervision. Furthermore, we propose an additional loss term called disagreement penalty to enable HA-Net to learn from distantly supervised labels. In addition, we exploit additional networks to adaptively assess the labeling bias by considering contextual information. Our performance study on sentence-level and document-level REs confirms the effectiveness of the dual supervision framework. 1 Introduction Relation extraction (RE) has been widely used in real-world applications such as knowledge base con- struction (Dong et al., 2014a; Dong et al., 2014b; Jung et al., 2019), question answering (Xu et al., 2016) and biomedical data mining (Ahmed et al., 2019). Given a pair of entities in a text (e.g., sentence or doc- ument), the goal of RE is to discover the relationships between the entities expressed in the text. More specifically, we aim to extract triples from the text in the form of ⟨eh, r, et⟩ where eh is a head entity, et is a tail entity and r is a relationship between the entities. To train a model for RE, we need a large volume of fully labeled training data in the form of text-triple pairs. Although human annotation provides high-quality labels to train the relation extraction models, it is difficult to produce a large-scale training data since manual labeling is expensive and time-consuming. Thus, Mintz et al. (2009) proposed distant supervision to automatically produce a large labeled data by using an external knowledge base (KB). For a text with a head entity eh and a tail entity et, when a triple ⟨eh, r, et⟩ exists in the KB for any relation type r, distant supervision produces a label ⟨eh, r, et⟩ even though the relationship is not expressed in the text. Thus, it suffers from the wrong labeling problem. For instance, if a triple ⟨UK, capital, London⟩ is in the KB, distant supervision labels the triple even for the sentence ‘London is the largest city of the UK’. Although each of the two labeling methods has a certain weakness, …

Content truncated to 500 words