ConNER: Consistency Training for Cross-lingual Named Entity Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8438 - 8449 December 7-11, 2022 ©2022 Association for Computational Linguistics ConNER: Consistency Training for Cross-lingual Named Entity Recognition Ran Zhou∗1,2 Xin Li1† Lidong Bing1 Erik Cambria2 Luo Si1 Chunyan Miao2 1DAMO Academy, Alibaba Group 2Nanyang Technological University, Singapore {ran.zhou, xinting.lx, l.bing, luo.si}@alibaba-inc.com {cambria, ascymiao}@ntu.edu.sg Abstract Cross-lingual named entity recognition (NER) suffers from data scarcity in the target lan- guages, especially under zero-shot settings. Ex- isting translate-train or knowledge distillation methods attempt to bridge the language gap, but often introduce a high level of noise. To solve this problem, consistency training meth- ods regularize the model to be robust towards perturbations on data or hidden states. How- ever, such methods are likely to violate the consistency hypothesis, or mainly focus on coarse-grain consistency. We propose ConNER as a novel consistency training framework for cross-lingual NER, which comprises of: (1) translation-based consistency training on un- labeled target-language data, and (2) dropout- based consistency training on labeled source- language data. ConNER effectively leverages unlabeled target-language data and alleviates overfitting on the source language to enhance the cross-lingual adaptability. Experimental re- sults show our ConNER achieves consistent improvement over various baseline methods.1 1 Introduction With the emergence of large scale multilingual pretrained language models (Devlin et al., 2019; Conneau et al., 2020), cross-lingual named entity recognition (NER) performance has seen substan- tial improvement, especially under zero-shot set- ting where no labeled target-language training data is available (Tsai et al., 2016; Xie et al., 2018; Jain et al., 2019). Nevertheless, due to linguistic gaps between languages, NER models trained solely on the source-language data are likely to overfit the source language’s characteristics, and still suffer from sub-optimal performance when tested on tar- get languages directly. ∗ Ran Zhou is under the Joint Ph.D. Program between Alibaba and Nanyang Technological University. †Corresponding author 1Our code is available at https://github.com/ RandyZhouRan/ConNER. To further enhance cross-lingual adaptability, various methods have been proposed to explic- itly incorporate target-language information dur- ing training. These approaches can be roughly categorized into: (1) translate-train and (2) knowl- edge distillation. Translate-train produces paral- lel target-language training data by translating the source-language training data and mapping the la- bels (Xie et al., 2018; Bari et al., 2020; Li et al., 2020). However, the translated sentences are less natural and the label mapping is also error-prone. Moreover, without utilizing the abundant unlabeled target-language data, their performance is usually sub-optimal (Wu et al., 2020b). On the other hand, knowledge distillation trains the student model on target-language data with soft labels from a teacher model (Wu et al., 2020a,b; Chen et al., 2021; Liang et al., 2021; Zhong et al., 2022). Given the rela- tively low performance of the teacher model on target languages, the soft labels are noisy and limit the efficacy of unlabeled target-language data when using these labels as supervision signals. In this paper, we primarily explore how to better utilize unlabeled target-language data. Instead of adopting knowledge distillation, which …

Content truncated to 500 words