Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6241 - 6252 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED Quzhe Huang1, Shibo Hao1, Yuan Ye1, Shengqi Zhu2, Yansong Feng1∗, Dongyan Zhao1 1Wangxuan Institute of Computer Technology, Peking University, China 2University of Washington {huangquzhe,haoshibo,pkuyeyuan,fengyansong,zhaody} @pku.edu.cn sqzhu@uw.edu Abstract DocRED is a widely used dataset for document- level relation extraction. In the large-scale annotation, a recommend-revise scheme is adopted to reduce the workload. Within this scheme, annotators are provided with candi- date relation instances from distant supervi- sion, and they then manually supplement and remove relational facts based on the recom- mendations. However, when comparing Do- cRED with a subset relabeled from scratch, we find that this scheme results in a consid- erable amount of false negative samples and an obvious bias towards popular entities and relations. Furthermore, we observe that the models trained on DocRED have low recall on our relabeled dataset and inherit the same bias in the training data. Through the analy- sis of annotators’ behaviors, we figure out the underlying reason for the problems above: the scheme actually discourages annotators from supplementing adequate instances in the re- vision phase. We appeal to future research to take into consideration the issues with the recommend-revise scheme when designing new models and annotation schemes. The relabeled dataset is released at https://github. com/AndrewZhe/Revisit-DocRED, to serve as a more reliable test set of document RE models. 1 Introduction Relation Extraction (RE) is an important task which aims to identify relationships held between enti- ties in a given piece of text. While most previous methods focus on extracting relations from a sin- gle sentence (Lin et al., 2016; Zhang et al., 2018), recent studies begin to explore RE at document level (Peng et al., 2017; Zeng et al., 2020; Nan et al., 2020; Huang et al., 2021; Zhang et al., 2021), which is more challenging as it often requires rea- soning across multiple sentences. ∗Corresponding author. The rapid development of document-level RE in the past two years has benefited from the proposal of DocRED (Yao et al., 2019), the first large-scale and human-annotated dataset for this task. Notice- ably, longer documents introduce an unprecedented difficulty in annotating the relation instances: as the total number of entities dramatically increases in accordance to text length, the expected num- ber of entity pairs surges quadratically, intensively increasing the workload to check relationships be- tween every pair. To address this problem, Yao et al. (2019) applies a recommend-revise process: in the recommendation phase, a small set of candi- date relation instances is generated through distant supervision; then, annotators are required to revise the candidate set, removing the incorrect relation instances and supplementing the instances not iden- tified in the recommendation phase. Shifting the construction process from scratch to an edit-based task, it seems that the recommend- revise scheme cuts down the effort of annotating by a large margin. …

Content truncated to 500 words