HacRED: A Large-Scale Relation Extraction Dataset Toward Hard Cases in Practical Applications

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831 August 1–6, 2021. ©2021 Association for Computational Linguistics 2819 HacRED: A Large-Scale Relation Extraction Dataset Toward Hard Cases in Practical Applications Qiao Cheng1∗, Juntao Liu1∗, Xiaoye Qu2, Jin Zhao1, Jiaqing Liang1, Zhefeng Wang2, Baoxing Huai2, Nicholas Jing Yuan2, Yanghua Xiao1,3† 1Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, China 2Huawei Cloud, China 3Fudan-Aishu Cognitive Intelligence Joint Research Center, Shanghai, China {cq.qiaojim, l.j.q.light}@gmail.com, {jtliu19, shawyh, jinzhao20}@fudan.edu.cn, {quxiaoye, wangzhefeng, huaibaoxing, nicholas.yuan}@huawei.com Abstract Relation extraction (RE) is an essential topic in natural language processing and has at- tracted extensive attention. Current RE ap- proaches achieve fantastic results on common datasets, while they still struggle on practical applications. In this paper, we analyze the above performance gap, the underlying rea- son of which is that practical applications in- trinsically have more hard cases. To make RE models more robust on such practical hard cases, we propose a case-oriented construc- tion framework to build a Hard Case Relation Extraction Dataset (HacRED). The proposed HacRED consists of 65,225 relational facts an- notated from 9,231 documents with sufficient and diverse hard cases. Notably, HacRED is one of the largest Chinese document-level RE datasets and achieves a high 96% F1 score on data quality. Furthermore, we apply the state- of-the-art RE models on this dataset and con- duct a thorough evaluation. The results show that the performance of these models is far lower than humans, and RE applying on practi- cal hard cases still requires further efforts. Ha- cRED is publicly available at https://github. com/qiaojiim/HacRED. 1 Introduction Relation extraction (RE) is one of the core NLP tasks and plays an increasingly important role in knowledge graph completion (Bordes et al., 2013) and question answering (Dong et al., 2015). RE aims to extract structured relational facts, i.e., triples such as (Bill Gates, founder_of, Mi- crosoft) from plain texts. Recently, various models (Zeng et al., 2018; Takanobu et al., 2019; Fu et al., 2019; Wei et al., 2020) have been proposed to iden- tify the relational facts and achieved state-of-the- art (SOTA) performance, among which the latest ∗Equally contributed. †Corresponding author. Elliot See was born on July 23rd , 1927 in Dallas, and died in St. Louis on February 28th , 1966 . Case in WebNLG Elliot See , place_of_birth, Dallas Elliot See , place_of_death, St. Louis Triples Yang Jima (1986 -), ..., is a student of 2005 in the Department of …, Communication University of China ... In the semi- final of the Chinese Idol Show, Yang excellently performed the Lhasa Ballad, which was recognized by the judges and the audience. As a result, she got to the final competition. Case in Practice Yang Jima(Yang) , graduate_from, Communication University of China Lhasa Ballad , singer, Yang Jima Yang Jima(Yang) , invited_guest_of, Chinese Idol Show Triples Figure 1: Cases and corresponding triples in WebNLG and practical applications. method CasRel achieves notable 91.8% F1 score on WebNLG (Gardent et al., 2017) and 89.6% on NYT (Riedel …

Content truncated to 500 words