Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3203–3215 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3203 Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training Bo Zheng†∗, Li Dong‡, Shaohan Huang‡, Saksham Singhal‡, Wanxiang Che†, Ting Liu†, Xia Song‡, Furu Wei‡ †Harbin Institute of Technology ‡Microsoft Corporation {bzheng,car,tliu}@ir.hit.edu.cn {lidong1,shaohanh,saksingh,xiaso,fuwei}@microsoft.com Abstract Compared to monolingual models, cross- lingual models usually require a more expres- sive vocabulary to represent all languages ad- equately. We find that many languages are under-represented in recent cross-lingual lan- guage models due to the limited vocabulary ca- pacity. To this end, we propose an algorithm VOCAP to determine the desired vocabulary capacity of each language. However, increas- ing the vocabulary size significantly slows down the pre-training speed. In order to ad- dress the issues, we propose k-NN-based tar- get sampling to accelerate the expensive soft- max. Our experiments show that the multi- lingual vocabulary learned with VOCAP bene- fits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mit- igates the side-effects of increasing the vo- cabulary size while achieving comparable per- formance and faster pre-training speed. The code and the pretrained multilingual vocab- ularies are available at https://github. com/bozheng-hit/VoCapXLM. 1 Introduction Pretrained cross-lingual language models (Con- neau and Lample, 2019; Conneau et al., 2020; Chi et al., 2021b; Xue et al., 2020) have recently shown great success in improving cross-lingual transfer- ability. These models encode texts from differ- ent languages into universal representations with a shared multilingual vocabulary and a shared Trans- former encoder (Vaswani et al., 2017). By pre- training cross-lingual language models on the large- scale multilingual corpus, the models achieve state- of-the-art performance on various downstream tasks, e.g., cross-lingual question answering and cross-lingual sentence classification. Although the Transformer architecture used in most pretrained monolingual and cross-lingual lan- guage models are almost identical, the vocabularies ∗Contribution during internship at Microsoft Research. are quite different. The vocabulary sizes in exist- ing pretrained monolingual language models typi- cally range from 30K to 60K subword units (Devlin et al., 2019; Liu et al., 2019; Dong et al., 2019; Bao et al., 2020). Meanwhile, state-of-the-art pretrained cross-lingual language models use the shared mul- tilingual vocabulary of 250K subword units to rep- resent more than 100 languages (Conneau et al., 2020; Chi et al., 2021b; Xue et al., 2020). Although some subword units are shared across languages, no more than 2.5K language-specific subword units on average are allocated for each language, which is still relatively small. Besides, the multilingual vocabulary is trained on the combined multilingual corpus with subword segmentation algorithms like BPE (Sennrich et al., 2015) and unigram language model (Kudo, 2018). During vocabulary construc- tion, these algorithms tend to select more subword units shared across languages with common scripts like Latin and Cyrillic (Chung et al., 2020b), but have a lower chance to select language-specific subword units. It is hard to determine how much vocabulary capacity a particular language requires and whether the shared multilingual vocabulary has allocated enough vocabulary capacity to …

Content truncated to 500 words