Multi-Grained Knowledge Distillation for Named Entity Recognition

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5704–5716 June 6–11, 2021. ©2021 Association for Computational Linguistics 5704 Multi-Grained Knowledge Distillation for Named Entity Recognition Xuan Zhou 1, Xiao Zhang1, Chenyang Tao2, Junya Chen2, Bing Xu1, Wei Wang1, and Jing Xiao1 1Ping An Technology (Shenzhen) Co., Ltd 2Duke University {zhouxuan553,zhangxiao585,xubing962,wangwei192,xiaojing661}@pingan.com.cn {chenyang.tao,junya.chen}@duke.edu Abstract Although pre-trained big models (e.g., BERT, ERNIE, XLNet, GPT3 etc.) have delivered top performance in Seq2seq modeling, their deployments in real-world applications are of- ten hindered by the excessive computations and memory demand involved. For many applications, including named entity recogni- tion (NER), matching the state-of-the-art re- sult under budget has attracted considerable attention. Drawing power from the recent advance in knowledge distillation (KD), this work presents a novel distillation scheme to ef- ficiently transfer the knowledge learned from big models to their more affordable counter- part. Our solution highlights the construction of surrogate labels through the k-best Viterbi algorithm to distill knowledge from the teacher model. To maximally assimilate knowledge into the student model, we propose a multi- grained distillation scheme, which integrates cross entropy involved in conditional random field (CRF) and fuzzy learning. To validate the effectiveness of our proposal, we conducted a comprehensive evaluation on five NER bench- marks, reporting cross-the-board performance gains relative to competing prior-arts. We further discuss ablation results to dissect our gains. 1 Introduction The task of named entity recognition (NER) aims to put named entity mentioned in a sentence into some pre-defined categories such as the person names, organizations, locations, etc. NER is a fundamen- tal task in nature language processing (NLP), and often serves as an information extraction tool em- bedded in solutions for downstream tasks such as event recognition and aspect-level sentiment analy- sis. And therefore, the computational and memory efficiency of an NER model is often considered crucial in many empirical settings. Given its practical significance, considerable re- search effort has been devoted to NER in recent years. One fruitful direction is to boost NER performance through the use of more sophisti- cated model architectures, such as Transformer and its variants (e.g., BERT (Devlin et al., 2019), ERNIE (Sun et al., 2019c), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019b), GPT3 (Brown et al., 2020), etc.). Many of these models are based on the attention mechanism, which allows the mod- els to adaptively focus on different parts of the sentence based on its current understanding. This enables more accurate parsing of the context, which is critical for the NER task. While such advanced models have delivered substantial improvements, a major drawback is that they are typically com- putationally expensive and memory intensive, pre- venting their applications in many cost-sensitive settings. As such, it is often desirable to reduce those big models into more affordable counterparts, prefer- ably without any significant performance drop. One strategy is to compress or truncate the original model, examples in this category include parame- ter pruning (Srinivas and Babu, …

Content truncated to 500 words