CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4452–4472 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4452 CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild Yuan Yao1∗, Jiaju Du1∗, Yankai Lin2, Peng Li2, Zhiyuan Liu1† , Jie Zhou2, Maosong Sun1 1Department of Computer Science and Technology Institute for Artificial Intelligence, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology, China 2Pattern Recognition Center, WeChat AI, Tencent Inc. yuan-yao18@mails.tsinghua.edu.cn, i@dujiaju.me Abstract Existing relation extraction (RE) methods typ- ically focus on extracting relational facts be- tween entity pairs within single sentences or documents. However, a large quantity of re- lational facts in knowledge bases can only be inferred across documents in practice. In this work, we present the problem of cross- document RE, making an initial step towards knowledge acquisition in the wild. To fa- cilitate the research, we construct the first human-annotated cross-document RE dataset CodRED. Compared to existing RE datasets, CodRED presents two key challenges: Given two entities, (1) it requires finding the relevant documents that can provide clues for identi- fying their relations; (2) it requires reasoning over multiple documents to extract the rela- tional facts. We conduct comprehensive ex- periments to show that CodRED is challeng- ing to existing RE methods including strong BERT-based models. We make CodRED and the code for our baselines publicly available at https://github.com/thunlp/CodRED. 1 Introduction Relation extraction (RE), which aims to extract relations between entities from plain text, serves as an essential resource in populating knowledge bases (KBs) from large-scale corpora automati- cally. Existing RE systems typically focus on either sentence-level RE (Socher et al., 2012; Zeng et al., 2014, 2015; Lin et al., 2016; Qin et al., 2018) or document-level RE (Li et al., 2016; Peng et al., 2017; Quirk and Poon, 2017; Yao et al., 2019), and have achieved promising results on several pub- lic benchmarks. However, these works can only extract relational facts from single sentences or doc- uments containing both two target entities, which inevitably limits the coverage of knowledge acqui- sition. According to our statistics on Wikipedia ∗ indicates equal contribution † Corresponding author: Z.Liu (liuzy@tsinghua.edu.cn) documents, for over 57.6% of the relational facts in Wikidata (Erxleben et al., 2014; Vrandeˇci´c and Krötzsch, 2014), the head and tail entities do not co-occur in a single document. This inspires that it is crucial to break through the limitations of docu- ment boundaries to acquire knowledge in the wild. In this work, we make an initial step in this di- rection, presenting the problem of cross-document RE (cross-doc RE), which requires a RE system to infer the relation between two entities by retriev- ing and reasoning over multiple documents. Com- pared to conventional sentence/document-level RE, cross-doc RE presents new challenges in two lev- els of granularity: (1) at the coarse-grained level, given an entity pair, RE systems are required to find multiple informative documents for each en- tity, instead of restricted to the sentence/document containing both …

Content truncated to 500 words