Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8472 - 8487 December 7-11, 2022 ©2022 Association for Computational Linguistics Revisiting DocRED – Addressing the False Negative Problem in Relation Extraction Qingyu Tan* 1, 2 Lu Xu* 1, 3 Lidong Bing † 1 Hwee Tou Ng2 Sharifah Mahani Aljunied1 1DAMO Academy, Alibaba Group 2Department of Computer Science, National University of Singapore 3Singapore University of Technology and Design {qingyu.tan,lu.x,l.bing,mahani.aljunied}@alibaba-inc.com {qtan6,nght}@comp.nus.edu.sg Abstract The DocRED dataset is one of the most popu- lar and widely used benchmarks for document- level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the annotation of DocRED is in- complete, i.e., false negative samples are preva- lent. We analyze the causes and effects of the overwhelming false negative problem in the DocRED dataset. To address the shortcom- ing, we re-annotate 4,053 documents in the DocRED dataset by adding the missed relation triples back to the original DocRED. We name our revised DocRED dataset Re-DocRED. We conduct extensive experiments with state-of- the-art neural models on both datasets, and the experimental results show that the mod- els trained and evaluated on our Re-DocRED achieve performance improvements of around 13 F1 points. Moreover, we conduct a compre- hensive analysis to identify the potential areas for further improvement.1 1 Introduction The field of relation extraction (RE) is related to knowledge bases (KBs). Most popular relation extraction datasets are constructed from knowl- edge triples in KBs. For example, the TACRED dataset (Zhang et al., 2017) is constructed by the TAC Knowledge Base Population challenge. The NYT10 (Riedel et al., 2013) dataset matched the Freebase Knowledge base (Bollacker et al., 2008) to the New York Times corpus (Sandhaus, 2008). Wiki20 (Gao et al., 2021) and DocRED (Yao et al., 2019) originated from distant supervision from the Wikidata knowledge base (Vrandeˇci´c and Krötzsch, 2014) and Wikipedia articles. By exploiting distant ∗ Equal contribution. Qingyu Tan and Lu Xu are under the Joint PhD Program between Alibaba and NUS/SUTD. † Corresponding author. 1Our dataset is publicly available at https://github. com/tonytan48/Re-DocRED. "I Knew You Were Trouble " is a song recorded by American singer - songwriter Taylor Swift for her fourth studio album , Red ( 2012 ) . It was released on October 9 , 2012 , in the United States by Big Machine Records as the third promotional single from the album . Later , " I Knew You Were Trouble " was released as the third single from Red on November 27 , 2012 , in the United States . It was written by Swift , Max Martin and Shell- back , with the production handled by the latter two ... It later peaked at number two in January 2013 , blocked from the top spot by Bruno Mars’ " Locked Out of Heaven " . At the inaugural YouTube Music Awards in 2013 , " I Knew You Were Trouble " won the award for YouTube …

Content truncated to 500 words