DocRED: A Large-Scale Document-Level Relation Extraction Dataset

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764–777 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 764 DocRED: A Large-Scale Document-Level Relation Extraction Dataset Yuan Yao1∗, Deming Ye1∗, Peng Li2, Xu Han1, Yankai Lin1, Zhenghao Liu1, Zhiyuan Liu1† , Lixin Huang1, Jie Zhou2, Maosong Sun1 1Department of Computer Science and Technology, Tsinghua University, Beijing, China Institute for Artificial Intelligence, Tsinghua University, Beijing, China State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc. {yuan-yao18,ydm18}@mails.tsinghua.edu.cn Abstract Multiple entities in a document generally ex- hibit complex inter-sentence relations, and cannot be well handled by existing relation extraction (RE) methods that typically fo- cus on extracting intra-sentence relations for single entity pairs. In order to accelerate the research on document-level RE, we in- troduce DocRED, a new dataset constructed from Wikipedia and Wikidata with three fea- tures: (1) DocRED annotates both named en- tities and relations, and is the largest human- annotated dataset for document-level RE from plain text; (2) DocRED requires reading mul- tiple sentences in a document to extract en- tities and infer their relations by synthesiz- ing all information of the document; (3) along with the human-annotated data, we also of- fer large-scale distantly supervised data, which enables DocRED to be adopted for both su- pervised and weakly supervised scenarios. In order to verify the challenges of document- level RE, we implement recent state-of-the-art methods for RE and conduct a thorough eval- uation of these methods on DocRED. Empir- ical results show that DocRED is challeng- ing for existing RE methods, which indicates that document-level RE remains an open prob- lem and requires further efforts. Based on the detailed analysis on the experiments, we dis- cuss multiple promising directions for future research. We make DocRED and the code for our baselines publicly available at https: //github.com/thunlp/DocRED. 1 Introduction The task of relation extraction (RE) is to iden- tify relational facts between entities from plain text, which plays an important role in large-scale knowledge graph construction. Most existing RE ∗ indicates equal contribution † Corresponding author: Z.Liu(liuzy@tsinghua.edu.cn) [1] Kungliga Hovkapellet (The Royal Court Orchestra) is a Swedish orchestra, originally part of the Royal Court in Sweden's capital Stockholm. [2] The orchestra originally consisted of both musicians and singers. [3] It had only male members until 1727, when Sophia Schröder and Judith Fischer were employed as vocalists; in the 1850s, the harpist Marie Pauline Åhman became the first female instrumentalist. [4] From 1731, public concerts were performed at Riddarhuset in Stockholm. [5] Since 1773, when the Royal Swedish Opera was founded by Gustav III of Sweden, the Kungliga Hovkapellet has been part of the opera's company. Kungliga Hovkapellet Subject: Kungliga Hovkapellet; Royal Court Orchestra Object: Royal Swedish Opera Relation: part_of Subject: Riddarhuset Object: Sweden Relation: country Supporting Evidence: 5 Supporting Evidence: 1, 4 Figure 1: An example from DocRED. Each document in DocRED is annotated with named entity mentions, coreference information, intra- …

Content truncated to 500 words