Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction

Proceedings of the 28th International Conference on Computational Linguistics, pages 6436–6447 Barcelona, Spain (Online), December 8-13, 2020 6436 Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction Tong Zhu1, Haitao Wang1, Junjie Yu1, Xiabing Zhou1, Wenliang Chen1, Wei Zhang2, Min Zhang1 1Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China 2Alibaba Group, China {tzhu7, htwang2019, jjyu}@stu.suda.edu.cn {zhouxiabing, wlchen, minzhang}@suda.edu.cn lantu.zw@alibaba-inc.com Abstract In recent years, distantly-supervised relation extraction has achieved a certain success by using deep neural networks. Distant Supervision (DS) can automatically generate large-scale annotated data by aligning entity pairs from Knowledge Bases (KB) to sentences. However, these DS- generated datasets inevitably have wrong labels that result in incorrect evaluation scores during testing, which may mislead the researchers. To solve this problem, we build a new dataset NYT- H, where we use the DS-generated data as training data and hire annotators to label test data. Compared with the previous datasets, NYT-H has a much larger test set and then we can perform more accurate and consistent evaluation. Finally, we present the experimental results of several widely used systems on NYT-H. The experimental results show that the ranking lists of the comparison systems on the DS-labelled test data and human-annotated test data are different. This indicates that our human-annotated data is necessary for evaluation of distantly-supervised relation extraction. 1 Introduction There has been significant progress on Relation Extraction (RE) in recent years using models based on machine learning algorithms (Mintz et al., 2009; Hoffmann et al., 2011; Zeng et al., 2015; Zhou et al., 2016; Ji et al., 2017; Su et al., 2018; Qin et al., 2018b; Zhang et al., 2019). The task of RE is to identify semantic relationships among entities from texts. Traditional supervised methods require a massive amount of annotated data, which are often labelled by human annotators. However, it is hard to annotate data within strict time limits and hiring the annotators is non-scalable and costly. Knowledge Base Head Tail Relation Apple Steve Jobs founders Adele London place_of_birth Aragaki Yui Japanese nationality Corpus Sentence Distantly-Supervised Label Comment Steve Jobs left Apple in 1985 . founders ✗False Positive Adele was born in the UK . NA ✗False Negative Aragaki Yui is an Japanese actress . nationality ✓True Positive Jack , have you heard of Hemingway ? NA ✓True Negative Figure 1: Distant supervision alignment and noises. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 6437 In order to quickly obtain new annotated data, Mintz et al. (2009) use Distant Supervision (DS) to automatically generate labelled data. Given an entity pair and its relation from a Knowledge Base (KB) such as Freebase, they simply tag all sentences containing these two entities by this relation. Through this way, the framework has achieved great success and brought state-of-the-art performance in RE (Qin et al., 2018a; Liu et al., 2018; Feng et al., 2018; Chen et al., 2019). But as an exchange, the auto-generated data may be of …

Content truncated to 500 words