On the Robustness of Question Rewriting Systems to Questions of Varying Hardness

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 170 - 179 May 22-27, 2022 c⃝2022 Association for Computational Linguistics An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition Zhuoran Li1, Chunming Hu1∗, Xiaohui Guo2 Junfan Chen1, Wenyi Qin1, Richong Zhang1 1SKLSDE, School of Computer Science and Engineering, Beihang University, Beijing, China 2Hangzhou Innovation Institute, Beihang University, Hangzhou, China {lizhuoranget, hucm}@buaa.edu.cn, wenyi.qin@hotmail.com {guoxh, chenjf, zhangrc}@act.buaa.edu.cn Abstract Cross-lingual named entity recognition task is one of the critical problems for evaluating the potential transfer learning techniques on low resource languages. Knowledge distilla- tion using pre-trained multilingual language models between source and target languages have shown their superiority in transfer. How- ever, existing cross-lingual distillation models merely consider the potential transferability between two identical single tasks across both domains. Other possible auxiliary tasks to im- prove the learning performance have not been fully investigated. In this study, based on the knowledge distillation framework and multi- task learning, we introduce the similarity met- ric model as an auxiliary task to improve the cross-lingual NER performance on the target domain. Specifically, an entity recognizer and a similarity evaluator are first trained in par- allel as two teachers from the source domain. Then, two tasks in the student model are super- vised by these teachers simultaneously. Empir- ical studies on the three datasets across 7 dif- ferent languages confirm the effectiveness of the proposed model. 1 Introduction Named entity recognition, NER in short, refers to identifying entity types, i.e. location, person, orga- nization, etc., in a given sentence. The exploiting of deep neural networks, such as Bi-LSTM-CRF (Lample et al., 2016), Bi-LSTM-CNN (Chiu and Nichols, 2016) makes this task achieve significant performances. However, since deep neural net- works highly rely on a large amount of labelled training data, the annotation acquiring process is expensive and time consuming. This situation is more severe for zero-resource languages. With the help of transfer learning (Ruder et al., 2019) and multilingual BERT (short as mBERT) (Devlin et al., ∗Corresponding author NERstu TSL MTMT NERtea {X,P}tgt NER {X,Y}src NERtea Directly {X}tgt {X}tgt NERstu training training soft labeling SIMtea {X,S}tgt training soft labeling similarity score {X,P}tgt collaboration Figure 1: Comparison between previous cross- lingual NER models. Directly: direct model trans- fer; TSL: teacher-student learning model; MTMT: proposed multiple-task and multiple-teacher Model. NER / NERtea: learned NER model for source lan- guage; NERstu: learned NER model for target lan- guage; SIMtea learned similarity model for source language; {X, Y }src: labeled data in source lan- guage; {X}tgt: unlabeled data in target language; {X, P}tgt: labeled data in target language with probability; {X, S}tgt: labeled data in target lan- guage with entity similarity score. 2019), it is possible to transfer the annotated train- ing samples or trained models from a rich-resource domain to a zero-resource domain. Many studies have been done to solve this cross- lingual NER problem. Existing models can be sep- arated into three categories, shared feature space based, translation based and …

Content truncated to 500 words