Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4018–4031 July 9-14, 2023 ©2023 Association for Computational Linguistics Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning Ran Zhou∗1,2 Xin Li1† Lidong Bing1 Erik Cambria2 Chunyan Miao2 1DAMO Academy, Alibaba Group 2Nanyang Technological University, Singapore {ran.zhou, xinting.lx, l.bing}@alibaba-inc.com {cambria, ascymiao}@ntu.edu.sg Abstract In cross-lingual named entity recognition (NER), self-training is commonly used to bridge the linguistic gap by training on pseudo- labeled target-language data. However, due to sub-optimal performance on target languages, the pseudo labels are often noisy and limit the overall performance. In this work, we aim to improve self-training for cross-lingual NER by combining representation learning and pseudo label refinement in one coherent frame- work. Our proposed method, namely ContProto mainly comprises two components: (1) con- trastive self-training and (2) prototype-based pseudo-labeling. Our contrastive self-training facilitates span classification by separating clus- ters of different classes, and enhances cross- lingual transferability by producing closely- aligned representations between the source and target language. Meanwhile, prototype-based pseudo-labeling effectively improves the accu- racy of pseudo labels during training. We eval- uate ContProto on multiple transfer pairs, and experimental results show our method brings in substantial improvements over current state- of-the-art methods. 1 1 Introduction Cross-lingual named entity recognition (NER) (Tsai et al., 2016; Xie et al., 2018) has seen sub- stantial performance improvement since the emer- gence of large-scale multilingual pretrained lan- guage models (Devlin et al., 2019; Conneau et al., 2020). However, there is still a noticeable gap between zero-shot cross-lingual transfer and mono- lingual NER models trained with target-language labeled data. To further bridge the linguistic gap be- tween the source and target language, self-training ∗ Ran Zhou is under the Joint Ph.D. Program between Alibaba and Nanyang Technological University. †Corresponding author 1Our code is available at https://github.com/ DAMO-NLP-SG/ContProto. is widely adopted to exploit the abundant language- specific information in unlabeled target-language data (Wu et al., 2020b; Ye et al., 2020; Chen et al., 2021). In general, self-training (sometimes referred to as teacher-student learning (Wu et al., 2020a)) uses a weak tagger (i.e. teacher model) trained on source-language data to assign pseudo labels onto unlabeled target-language data, which is then com- bined with labeled source-language data to train the final model (i.e. student model). Nevertheless, due to sub-optimal performances on target languages, the pseudo-labeled data contains a large number of errors and might limit the performances of NER models trained on them. To optimize self-training for cross-lingual NER, several methods have been proposed to improve the quality of pseudo labels. One line of work focuses on selecting curated pseudo-labeled data for self- training via reinforcement learning (Liang et al., 2021a) or an adversarial discriminator (Chen et al., 2021). However, they do not fully utilize all the un- labeled data available. Wu et al. (2020a,b) exploit the full unlabeled dataset and alleviate the noise in pseudo labels by aggregating predictions from mul- tiple teacher models. Likewise, Liang et al. (2021a) develop …

Content truncated to 500 words