Structure-Level Knowledge Distillation For Multilingual Sequence Labeling

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3317–3330 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3317 Structure-Level Knowledge Distillation For Multilingual Sequence Labeling Xinyu Wang⋄, Yong Jiang†, Nguyen Bach†, Tao Wang†, Fei Huang†, Kewei Tu⋄∗ ⋄School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences †DAMO Academy, Alibaba Group {wangxy1,tukw}@shanghaitech.edu.cn {yongjiang.jy,nguyen.bach,leeo.wangt,f.huang}@alibaba-inc.com Abstract Multilingual sequence labeling is a task of pre- dicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, us- ing a multilingual model has the benefit of a smaller model size, easier in online serv- ing, and generalizability to low-resource lan- guages. However, current multilingual mod- els still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to re- duce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance be- tween the student’s and the teachers’ structure- level probability distributions, (2) aggregates the structure-level knowledge to local distri- butions and minimizes the distance between two local probability distributions. Our experi- ments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models. 1 Introduction Sequence labeling is an important task in natural language processing. Many tasks such as named entity recognition (NER) and part-of-speech (POS) tagging can be formulated as sequence labeling problems and these tasks can provide extra informa- tion to many downstream tasks and products such as searching engine, chat-bot and syntax parsing (Jurafsky and Martin, 2009). Most of the previ- ∗ Kewei Tu is the corresponding author. This work was conducted when Xinyu Wang was interning at Alibaba DAMO Academy. ous work on sequence labeling focused on mono- lingual models, and the work on multilingual se- quence labeling mainly focused on cross-lingual transfer learning to improve the performance of low-resource or zero-resource languages (Johnson et al., 2019; Huang et al., 2019a; Rahimi et al., 2019; Huang et al., 2019b; Keung et al., 2019), but their work still trains monolingual models. How- ever, it would be very resource consuming con- sidering if we train monolingual models for all the 7,000+ languages in the world. Besides, there are languages with limited labeled data that are required for training. Therefore it is beneficial to have a single unified multilingual sequence label- ing model to handle multiple languages, while less attention is paid to the unified multilingual mod- els due to the significant difference between dif- ferent languages. Recently, Multilingual BERT (M-BERT) (Devlin et al., 2019) is surprisingly good at zero-shot cross-lingual model transfer on tasks such as NER and POS tagging (Pires …

Content truncated to 500 words