Wider & Closer: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5171 - 5183 December 7-11, 2022 ©2022 Association for Computational Linguistics WIDER & CLOSER: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition Jun-Yu Ma1∗, Beiduo Chen1∗, Jia-Chen Gu1, Zhen-Hua Ling1, Wu Guo1† Quan Liu2,3, Zhigang Chen4, Cong Liu1,3 1National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China 2State Key Laboratory of Cognitive Intelligence 3iFLYTEK Research, Hefei, China 4Jilin Kexun Information Technology Co., Ltd. {mjy1999,beiduo}@mail.ustc.edu.cn, {gujc,zhling,guowu}@ustc.edu.cn, {quanliu,zgchen,congliu2}@iflytek.com Abstract Zero-shot cross-lingual named entity recog- nition (NER) aims at transferring knowledge from annotated and rich-resource data in source languages to unlabeled and lean-resource data in target languages. Existing mainstream meth- ods based on the teacher-student distillation framework ignore the rich and complementary information lying in the intermediate layers of pre-trained language models, and domain- invariant information is easily lost during trans- fer. In this study, a mixture of short-channel distillers (MSD) method is proposed to fully interact the rich hierarchical information in the teacher model and to transfer knowledge to the student model sufficiently and efficiently. Concretely, a multi-channel distillation frame- work is designed for sufficient information transfer by aggregating multiple distillers as a mixture. Besides, an unsupervised method adopting parallel domain adaptation is pro- posed to shorten the channels between the teacher and student models to preserve domain- invariant features. Experiments on four datasets across nine languages demonstrate that the proposed method achieves new state-of-the-art performance on zero-shot cross-lingual NER and shows great generalization and compatibil- ity across languages and fields. 1 Introduction Named entity recognition (NER) is a fundamental and important task to locate and classify named entities in a text sequence. Recently, deep neural networks have achieved great performance on monolingual NER in rich-resource languages with abundant labeled data (Ye and Ling, 2018; Jia et al., 2020; Chen et al., 2022). However, it is too expensive to annotate a large amount of data in low-resource languages for supervised NER ∗Equal contribution. †Corresponding author. Layer 12 Layer 11 Layer 10 Θtea Layer 12 Layer 11 Layer 10 Θstu DT transfer Layer 12 Layer 11 Layer 10 Θtea Layer 12 Layer 11 Layer 10 Θstu DT transfer transfer transfer DS DT, DS shorten (a) Baseline Distillation (b) MSD Figure 1: Comparison between the previous mainstream method and the proposed MSD. (a) Baseline Distil- lation is the teacher-student distillation framework. The teacher model is utilized to predict the soft labels of unlabeled target language data, which are further utilized to distill a student model. (b) MSD constructs a dozen of channels and shortens the transmission route between the teacher and student models to transfer NER knowledge. Θtea / Θstu: teacher / student models; DS / DT: unlabeled source / target language data. training. This issue drives research on cross-lingual NER, which utilizes the rich-resource annotated data in source languages to alleviate the scarcity of unlabeled lean-resource data in target languages. In this paper, following Wu et al. (2020a), …

Content truncated to 500 words