Soft Language Clustering for Multilingual Model Pre-training

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7021–7035 July 9-14, 2023 ©2023 Association for Computational Linguistics Soft Language Clustering for Multilingual Model Pre-training Jiali Zeng1∗, Yufan Jiang3, Yongjing Yin2, Yi Jing4, Fandong Meng1, Binghuai Lin3, Yunbo Cao3, Jie Zhou1 1Pattern Recognition Center, WeChat AI, Tencent Inc 2Westlake University 3Tencent Cloud Xiaowei 4Northeastern University {lemonzeng,fandongmeng,withtomzhou}@tencent.com yinyongjing@westlake.edu.cn Abstract Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross- lingual transfer abilities, however, their per- formance is hindered when the target lan- guage has distant typology from source lan- guages or when pre-training data is limited in size. In this paper, we propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight model- ing of language-invariant and language-specific knowledge across languages, and (2) easy in- tegration with other multilingual pre-training methods. On the tasks of XTREME including text classification, sequence labeling, question answering, and sentence retrieval, both base- and large-size language models pre-trained with our proposed method exhibit consistent performance improvement. Furthermore, it pro- vides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer1. 1 Introduction Multilingual pre-trained language models (mPLMs) such as mBERT (Devlin et al., 2019), mBART (Liu et al., 2020), XLM-R (Conneau et al., 2020) and mT5 (Xue et al., 2021) have lately produced notable advancements in a number of downstream NLP tasks. In particular, the use of mPLMs significantly enhances few-shot fine-tuning and makes possible efficient zero-shot cross-lingual transfer (Hu et al., 2020). Essentially, an ideal mPLM should satisfy two properties: alignment between language pairs, which has been widely studied in the literature (Chi et al., 2022; Ouyang et al., 2021; Chi et al., 2021a); and a good trade-off between high-resource and low-resource ∗Corresponding author. 1Code and model are available at https://github.com/lemon0830/XLMP.git languages, which remains largely unexplored despite the success of mPLMs. In this paper, we focus on the second property, specially the potential for model performance to suffer when a large number of languages are added. This can occur due to restricted model capacity or computational limitations, resulting in underrep- resented languages being allocated less capacity (Conneau et al., 2020). Furthermore, the model’s coverage of world’s languages remains inadequate, limiting the range of language technology applica- tions it can support (Ansell et al., 2021). A typical solution for the coverage-performance trade-off in multilingual learning is to assign additional model parameters to specific languages, such as language identity embeddings (Conneau and Lample, 2019), adaptors (Houlsby et al., 2019; Üstün et al., 2022; Ansell et al., 2021), and language-aware layers (Zhang et al., 2021). However, it is impractical for multilingual pre-training to maintain a separate component for each language, which can lead to more complicated and challenging optimization, especially for low-resource languages. We propose to approach the above language- aware components from a different perspective. In linguistic typology, some patterns such as nominative-accusative alignment have broad global distributions, whereas …

Content truncated to 500 words