Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3602 - 3615 December 7-11, 2022 ©2022 Association for Computational Linguistics Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition Junhao Zheng, Zhanxian Liang, Haibin Chen, Qianli Ma* School of Computer Science and Engineering, South China University of Technology, Guangzhou, China junhaozheng47@outlook.com, qianlima@scut.edu.cn∗ Abstract Continual Learning for Named Entity Recogni- tion (CL-NER) aims to learn a growing number of entity types over time from a stream of data. However, simply learning Other-Class in the same way as new entity types amplifies the catastrophic forgetting and leads to a substan- tial performance drop. The main cause behind this is that Other-Class samples usually con- tain old entity types, and the old knowledge in these Other-Class samples is not preserved properly. Thanks to the causal inference, we identify that the forgetting is caused by the missing causal effect from the old data. To this end, we propose a unified causal frame- work to retrieve the causality from both new entity types and Other-Class. Furthermore, we apply curriculum learning to mitigate the im- pact of label noise and introduce a self-adaptive weight for balancing the causal effects between new entity types and Other-Class. Experimen- tal results on three benchmark datasets show that our method outperforms the state-of-the- art method by a large margin. Moreover, our method can be combined with the existing state- of-the-art methods to improve the performance in CL-NER. 1 1 Introduction Named Entity Recognition (NER) is a vital task in various NLP applications (Ma and Hovy, 2016). Traditional NER aims at extracting entities from un- structured text and classifying them into a fixed set of entity types (e.g., Person, Location, Organiza- tion, etc). However, in many real-world scenarios, the training data are streamed, and the NER sys- tems are required to recognize new entity types to support new functionalities, which can be formu- lated into the paradigm of continual learning (CL, a.k.a. incremental learning or lifelong learning) ∗*Corresponding author 1Our codes are publicly available at https://github.com/zzz47zzz/CFNER Figure 1: An illustration of Other-class in CL-NER. Suppose that a model learns four entity types in CoNLL2003 sequentially. “LOC”: Location; “MISC”: Miscellaneous; “ORG”: Organisation; “PER”: Person. (Thrun, 1998; Parisi et al., 2019). For instance, voice assistants such as Siri or Alexa are often re- quired to extract new entity types (e.g. Song, Band) for grasping new intents (e.g. GetMusic) (Monaikul et al., 2021). However, as is well known, continual learning faces a serious challenge called catastrophic for- getting in learning new knowledge (McCloskey and Cohen, 1989; Robins, 1995; Goodfellow et al., 2013; Kirkpatrick et al., 2017). More specifically, simply fine-tuning a NER system on new data usu- ally leads to a substantial performance drop on previous data. In contrast, a child can naturally learn new concepts (e.g., Song and Band) without forgetting the learned concepts (e.g., Person and Location). Therefore, continual learning for NER (CL-NER) is a ubiquitous issue and a big challenge in achieving human-level intelligence. In …

Content truncated to 500 words