A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4123 - 4135 December 7-11, 2022 ©2022 Association for Computational Linguistics A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling Ye Wang1, Xinxin Liu1, Wenxin Hu1∗, Tao Zhang2 1East China Normal University, Shanghai, China 2Tsinghua University, Beijing, China {yewang, xxliu}@stu.ecnu.edu.cn, wxhu@cc.ecnu.edu.cn tao-zhan20@mails.tsinghua.edu.cn Abstract Document-level relation extraction (RE) aims to identify relations between entities across multiple sentences. Most previous methods focused on document-level RE under full super- vision. However, in real-world scenario, it is ex- pensive and difficult to completely label all re- lations in a document because the number of en- tity pairs in document-level RE grows quadrat- ically with the number of entities. To solve the common incomplete labeling problem, we propose a unified positive-unlabeled learning framework − shift and squared ranking loss positive-unlabeled (SSR-PU) learning. We use positive-unlabeled (PU) learning on document- level RE for the first time. Considering that labeled data of a dataset may lead to prior shift of unlabeled data, we introduce a PU learning under prior shift of training data. Also, us- ing none-class score as an adaptive threshold, we propose squared ranking loss and prove its Bayesian consistency with multi-label ranking metrics. Extensive experiments demonstrate that our method achieves an improvement of about 14 F1 points relative to the previous base- line with incomplete labeling. In addition, it outperforms previous state-of-the-art results un- der both fully supervised and extremely unla- beled settings as well.1 1 Introduction Relation extraction (RE) aims to identify the re- lations between two entities in a given text. It has rich applications in knowledge graph construc- tion, question answering, and biomedical text un- derstanding. Most of the previous work was to extract relations between entities in a single sen- tence (Miwa and Bansal, 2016; Zhang et al., 2018). Recently, document-level RE aiming to identify the relations among various entity pairs expressed in multiple sentences has received increasing research ∗Corresponding author. 1Code and data are available at https://github. com/www-Ye/SSR-PU. Figure 1: A case from DocRED. Entities are highlighted in different colors depending on their type. Black arrows indicate relations annotated with the original dataset, orange arrows indicate relations that are re-annotated by (Tan et al., 2022b). attention (Yao et al., 2019; Zhou et al., 2021; Xu et al., 2022). Previous document-level RE methods mainly deal with fully supervised scenarios. However, in real-world scenarios, incomplete labeling is a common problem in document-level RE because the number of entity pairs grows quadratically with the number of entities. DocRED (Yao et al., 2019) is a popular dataset for document-level RE. Re- cent studies (Huang et al., 2022; Tan et al., 2022b) found that DocRED, which annotates data with a 4123 recommend-revise scheme, contains a large num- ber of false negative samples, i.e., many positive samples being unlabeled. As shown in Figure 1, document Alecu Russo contains a large number of unlabeled positive relations. Consequently, the models trained on this dataset tend to overfit in real …

Content truncated to 500 words