Learn and Review: Enhancing Continual Named Entity Recognition via Reviewing Synthetic Samples

Findings of the Association for Computational Linguistics: ACL 2022, pages 2291 - 2300 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learn and Review: Enhancing Continual Named Entity Recognition via Reviewing Synthetic Samples Yu Xia1, ∗, Quan Wang2, Yajuan Lyu2, Yong Zhu2, Wenhao Wu1, Sujian Li1, †, Dai Dai2 1Key Laboratory of Computational Linguistics, Peking University, MOE, China 2Baidu Inc., Beijing, China {yuxia, waynewu, lisujian}@pku.edu.cn {wangquan05, lvyajuan, zhuyong, daidai}@baidu.com Abstract Traditional methods for named entity recog- nition (NER) classify mentions into a fixed set of pre-defined entity types. However, in many real-world scenarios, new entity types are incrementally involved. To investigate this problem, continual learning is introduced for NER. However, the existing method depends on the relevance between tasks and is prone to inter-type confusion. In this paper, we pro- pose a novel two-stage framework Learn-and- Review (L&R) for continual NER under the type-incremental setting to alleviate the above issues. Specifically, for the learning stage, we distill the old knowledge from teacher to a stu- dent on the current dataset. For the reviewing stage, we first generate synthetic samples of old types to augment the dataset. Then, we further distill new knowledge from the above student and old knowledge from the teacher to get an enhanced student on the augmented dataset. This stage has the following advan- tages: (1) The synthetic samples mitigate the gap between the old and new task and thus enhance the further distillation; (2) Different types of entities are jointly seen during training which alleviates the inter-type confusion. Ex- perimental results show that L&R outperforms the state-of-the-art method on CoNLL-03 and OntoNotes-5.0. 1 Introduction Traditional Named Entity Recognition (NER) aims at extracting mentions from a given text and clas- sifying them into a fixed set of pre-defined entity types such as Person, Location, Organization, etc (Ma and Hovy, 2016). However, in many real- world scenarios, new entity types emerge periodi- cally by demand and the models are required to rec- ognize new types of entities without forgetting the old ones, which can formulate into the paradigm of ∗ This work was done during internship at Baidu Inc. † Corresponding author. continual learning (a.k.a. lifelong learning or incre- mental learning) (Thrun, 1998; Parisi et al., 2019). For example, voice assistants such as Siri are often expected to grasp new intents (e.g. GetMovie) and thus new entity types (e.g. Actor, Genre) are contin- ually involved. The ability to learn from continuous streams of data after deployment is important for modern NER models in specific scenarios. However, continual learning, as it has long been recognized, suffers severely from catastrophic for- getting, i.e., the loss or disruption of previously learned knowledge when new patterns are learned (McCloskey and Cohen, 1989; Robins, 1995; Good- fellow et al., 2013; Kirkpatrick et al., 2017). Dif- ferent from human beings, an NER model (partic- ularly that based on deep neural networks) which stores knowledge by its parameters is vulnerable to catastrophic forgetting of old knowledge while updating parameters to learn new entity types. In …

Content truncated to 500 words