Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3097 - 3106 December 7-11, 2022 ©2022 Association for Computational Linguistics Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model Mingqi Li1, Fei Ding1, Dan Zhang1, Long Cheng1, Hongxin Hu2, Feng Luo1∗ 1Clemson University, 2University at Buffalo {mingqil,feid,dzhang4,lcheng2,luofeng}@clemson.edu hongxinh@buffalo.edu Abstract Pre-trained multilingual language models play an important role in cross-lingual natural lan- guage understanding tasks. However, existing methods did not focus on learning the semantic structure of representation, and thus could not optimize their performance. In this paper, we propose Multi-level Multilingual Knowledge Distillation (MMKD), a novel method for im- proving multilingual language models. Specifi- cally, we employ a teacher-student framework to adopt rich semantic representation knowl- edge in English BERT. We propose token-, word-, sentence-, and structure-level alignment objectives to encourage multiple levels of con- sistency between source-target pairs and corre- lation similarity between teacher and student models. We conduct experiments on cross- lingual evaluation benchmarks including XNLI, PAWS-X, and XQuAD. Experimental results show that MMKD outperforms other baseline models of similar size on XNLI and XQuAD and obtains comparable performance on PAWS- X. Especially, MMKD obtains significant per- formance gains on low-resource languages. 1 Introduction Pre-training a large-scale language model and fine- tuning it on downstream tasks has shown great suc- cess in natural language processing. Most works (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019b) focused on English, since it is easy to get a large amount of training data for English. This paradigm has recently emerged as a promising means for cross-lingual tasks. Some multilingual language models (Devlin et al., 2018; Conneau et al., 2019) trained on monolingual data from over 100 languages using Masked Language Modeling (MLM) objective performed surprisingly well with- out any explicit alignment. On the other hand, XLM (Lample and Conneau, 2019) extended MLM objective to a parallel corpus version - Translation ∗*Corresponding author. Language Modeling (TLM), and achieved impres- sive results. This inspired researchers to develop alignment methods using parallel corpora. Follow-up works of XLM (Yang et al., 2020; Wei et al., 2020; Chi et al., 2020; Ouyang et al., 2020) leveraged various of training objectives to align par- allel sentences at different granularity. These works were usually trained using both large amounts of monolingual data and additional parallel corpora, which are time-consuming and require consider- able computational resources. Another line of research (Cao et al., 2020; Pan et al., 2020; Hu et al., 2020a) only used limited parallel data to improve existing pre-trained lan- guage models rather than training new models from scratch. They depended on new alignment methods for parallel pairs of words and sentences, which could further achieve performance gains over cur- rent state-of-the-art pre-trained language models. However, these approaches neglected vector space properties when aligning across languages, and thus generating sub-optimal results. We hypothe- size that a large-scale English corpus can provide more semantic and structural information than most other languages used to train multilingual language models. Moreover, BERT …

Content truncated to 500 words