Improving Continual Relation Extraction through Prototypical Contrastive Learning

Proceedings of the 29th International Conference on Computational Linguistics, pages 1885–1895 October 12–17, 2022. 1885 Improving Continual Relation Extraction through Prototypical Contrastive Learning ∗ Chengwei Hu1†, Deqing Yang1†�, Haoliang Jin1‡, Zhen Chen1‡, Yanghua Xiao2† 1School of Data Science, Fudan University, Shanghai, China 2School of Computer Science, Fudan University, Shanghai, China †{cwhu20, yangdeqing, shawyh}@fudan.edu.cn ‡{hljin21, zhenchen21}@m.fudan.edu.cn Abstract Continual relation extraction (CRE) aims to ex- tract relations towards the continuous and itera- tive arrival of new data, of which the major chal- lenge is the catastrophic forgetting of old tasks. In order to alleviate this critical problem for en- hanced CRE performance, we propose a novel Continual Relation Extraction framework with Contrastive Learning, namely CRECL, which is built with a classification network and a pro- totypical contrastive network to achieve the incremental-class learning of CRE. Specifically, in the contrastive network a given instance is contrasted with the prototype of each candi- date relations stored in the memory module. Such contrastive learning scheme ensures the data distributions of all tasks more distinguish- able, so as to alleviate the catastrophic forget- ting further. Our experiment results not only demonstrate our CRECL’s advantage over the state-of-the-art baselines on two public datasets, but also verify the effectiveness of CRECL’s contrastive learning on improving CRE perfor- mance. 1 Introduction In some scenarios of relation extraction (RE), mas- sive new data including new relations emerges con- tinuously, which can not be solved by traditional RE methods. To handle such situation, continual relation extraction (CRE) (Wang et al., 2019) was proposed. Due to the limited storage and comput- ing resources, it is impractical to store all training data of previous tasks. As new tasks are learned where new relations emerge constantly, the model tends to forget the existing knowledge about old re- lations. Therefore, the problem of catastrophic for- getting damages CRE performance severely (Hass- abis et al., 2017; Thrun and Mitchell, 1995). In recent years, some efforts have focused on the alleviating catastrophic forgetting in CRE, which ∗ This work is supported by Shanghai Science and Tech- nology Innovation Action Plan (China) No.21511100401. Figure 1: The data distribution map (better viewed in color) after training a classification model for an old task and then a new task. Many different relation data (different colors) of the old (dots) and new (crosses) task are mixed due to the catastrophic forgetting, making it hard to distinguish the new task’s relations from the old task’s relations. can be divided into consolidation-based methods (Kirkpatrick et al., 2017), dynamic architecture methods (Chen et al., 2015; Fernando et al., 2017) and Memory-based methods (Chaudhry et al., 2018; Han et al., 2020; Cui et al., 2021). Despite these methods’ effectiveness on CRE, most of them have not taken full advantage of the negative rela- tion information in all tasks to alleviate catastrophic forgetting more thoroughly, result in suboptimal CRE performance. Through our empirical studies, we found that the catastrophic forgetting of a model results in the indistinguishability between the data (instances) distributions of all tasks, making …

Content truncated to 500 words