Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation

Findings of the Association for Computational Linguistics: ACL 2022, pages 1672 - 1681 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation Qingyu Tan∗1,2 Ruidan He†1 Lidong Bing1 Hwee Tou Ng2 1DAMO Academy, Alibaba Group 2Department of Computer Science, National University of Singapore {qingyu.tan,ruidan.he,l.bing}@alibaba-inc.com {qtan6,nght}@comp.nus.edu.sg Abstract Document-level Relation Extraction (DocRE) is a more challenging task compared to its sentence-level counterpart. It aims to extract relations from multiple sentences at once. In this paper, we propose a semi-supervised frame- work for DocRE with three novel components. Firstly, we use an axial attention module for learning the interdependency among entity- pairs, which improves the performance on two- hop relations. Secondly, we propose an adap- tive focal loss to tackle the class imbalance problem of DocRE. Lastly, we use knowl- edge distillation to overcome the differences between human annotated data and distantly supervised data. We conducted experiments on two DocRE datasets. Our model consistently outperforms strong baselines and its perfor- mance exceeds the previous SOTA by 1.36 F1 and 1.46 Ign_F1 score on the DocRED leader- board.1 1 Introduction The problem of document-level relation extraction2 (DocRE) is highly important for information extrac- tion and NLP research. The DocRE task aims to extract relations among multiple entities within a document. The DocRE task is more challenging than its sentence-level counterpart in the following aspects: (1) The complexity of DocRE increases quadratically with the number of entities. If a doc- ument contains n entities, classification decisions must be made on n(n − 1) entity pairs and most of them do not contain any relation. (2) Aside from the imbalance of positive and negative examples, the distribution of relation types for the positive entity pairs is also highly imbalanced. Considering ∗† Qingyu Tan is under the Joint PhD Program between Alibaba and National University of Singapore. † Corresponding author 1Our code and data are available at https://github. com/tonytan48/KD-DocRE 2In this work, the task of relation extraction presumes that entities are given. the DocRED (Yao et al., 2019) dataset as an exam- ple, there are 96 relation types in total, where the top 10 relations take up 59.4% of all the relation labels. This imbalance significantly increases the difficulty of the document-level RE task. Most existing approaches of DocRE leverage dependency information to construct a document- level graph (Zeng et al., 2021; Zeng et al., 2020), and then use graph neural networks for reason- ing. Another popular strand of this field uses transformer-only (Vaswani et al., 2017) architec- ture (Zhou et al., 2021; Xu et al., 2021; Zhang et al., 2021). Such models are able to achieve state-of-the- art performance without explicit graph reasoning, showing that pre-trained language models (PrLMs) are able to implicitly capture long-distance relation- ships. However, there are three limitations of the existing DocRE methods. Firstly, existing meth- ods mainly focus on the syntactic features from PrLMs while neglecting the interactions between entity pairs. Zhang et al. (2021) and Li et al. (2021) have used …

Content truncated to 500 words