Consistent Representation Learning for Continual Relation Extraction

Findings of the Association for Computational Linguistics: ACL 2022, pages 3402 - 3411 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Consistent Representation Learning for Continual Relation Extraction Kang Zhao12, Hua Xu1∗, Jiangong Yang12, Kai Gao2∗ 1State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China 2School of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang 050018, China zhaok7878@gmail.com, xuhua@tsinghua.edu.cn,yjg219@163.com,gaokai@hebust.edu.cn Abstract Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding forgetting old ones. Some previous work has proved that storing a few typical samples of old relations and re- playing them when learning new relations can effectively avoid forgetting. However, these memory-based methods tend to overfit the memory samples and perform poorly on im- balanced datasets. To solve these challenges, a consistent representation learning method is proposed, which maintains the stability of the relation embedding by adopting contrastive learning and knowledge distillation when re- playing memory. Specifically, supervised con- trastive learning based on a memory bank is first used to train each new task so that the model can effectively learn the relation rep- resentation. Then, contrastive replay is con- ducted of the samples in memory and makes the model retain the knowledge of historical re- lations through memory knowledge distillation to prevent the catastrophic forgetting of the old task. The proposed method can better learn con- sistent representations to alleviate forgetting effectively. Extensive experiments on FewRel and TACRED datasets show that our method significantly outperforms state-of-the-art base- lines and yield strong robustness on the imbal- anced dataset. The code is publicly available at https://github.com/thuiar/CRL. 1 Introduction Relation extraction (RE) is an essential issue in in- formation extraction (IE), which can apply to many downstream NLP tasks, such as information re- trieval (Xiong et al., 2017) and question and answer (Tao et al., 2018). For example, given a sentence x with the annotated entities pairs e1 and e2, the RE aims to identify the relations between e1 and e2. However, traditional relation extraction models *Corresponding Author (Zhou et al., 2016; Soares et al., 2019a) always as- sume a fixed set of predefined relations and train on a fixed dataset, which cannot handle the growing relation types in real life well. To solve this situation, continual relation extrac- tion (CRE) is introduced (Wang et al., 2019; Han et al., 2020; Wu et al., 2021; Cui et al., 2021). Com- pared with traditional relation extraction, CRE aims to help the model learn new relations while main- taining accurate classification of old ones. Wang et al. (2019) shows that continual relation learn- ing needs to alleviate the catastrophic forgetting of old tasks when the model learns new tasks. Be- cause neural networks need to retrain a fixed set of parameters with each training, the most efficient solution to the problem of catastrophic forgetting is to store all the historical data and retrain the model with all the data each time a new relational …

Content truncated to 500 words