Improving Multilingual Models with Language-Clustered Vocabularies

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4536–4546, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4536 Improving Multilingual Models with Language-Clustered Vocabularies Hyung Won Chung∗ Dan Garrette Kiat Chuan Tan Jason Riesa Google Research {hwchung,dhgarrette,kiatchuan,riesa}@google.com Abstract State-of-the-art multilingual models depend on vocabularies that cover all of the languages the model will expect to see at inference time, but the standard methods for generating those vocabularies are not ideal for massively mul- tilingual applications. In this work, we intro- duce a novel procedure for multilingual vocab- ulary generation that combines the separately trained vocabularies of several automatically derived language clusters, thus balancing the trade-off between cross-lingual subword shar- ing and language-specific vocabularies. Our experiments show improvements across lan- guages on key multilingual benchmark tasks TYDI QA (+2.9 F1), XNLI (+2.1%), and WikiAnn NER (+2.8 F1) and factor of 8 re- duction in out-of-vocabulary rate, all without increasing the size of the model or data. 1 Introduction Multilingual models such as mBERT (Devlin et al., 2019), XLM (Lample and Conneau, 2019), and XLM-R (Conneau et al., 2020) have built on the advances of deep contextualized language model- ing by pretraining on texts from many languages at once. One trait common to all of these mod- els is the use of a single vocabulary containing subwords from all languages, used to segment the input text before transforming it into a sequence of embeddings. Conneau et al. (2020) showed that increasing the vocabulary size can produce qual- ity gains, but unlike similar monolingual models, the vocabulary embedding matrix in each of these multilingual models constitutes a significant frac- tion of its total parameters; for example, 47% of XLM-R’s parameters are in its embedding matrix. Therefore, scaling up a model’s vocabulary size requires the construction of inductive biases that ∗Work done as a member of the Google AI Residency Program. will guide the training procedure to effectively and efficiently learn these parameters. The multilingual subword vocabularies used by the state-of-the-art models are generated by algo- rithms such as WordPiece (Schuster and Nakajima, 2012; Wu et al., 2016), SentencePiece (Kudo and Richardson, 2018),1 or Byte Pair Encoding (BPE) (Sennrich et al., 2016). Given a desired vocabu- lary size, these algorithms select an inventory of subwords that compactly represents the training corpora, which means preferring subwords that oc- cur frequently, and, by extension for multilingual models, occur frequently across languages. Because these algorithms look at overall sub- word frequencies in the combined multilingual cor- pus, they may learn suboptimal decompositions for low-resource languages that happen to have character combinations that resemble subwords of high-resources languages.2 Additionally, subwords in common scripts like Latin and Cyrillic have a higher chance of selection since their counts are combined across a large number of languages (Wu and Dredze, 2019). By attempting to optimize for the best overall vocabulary across all languages without regard for the differences among those languages, the joint procedure over-emphasizes cross-lingual subword sharing—even across languages with little lexi- cal overlap—which is …

Content truncated to 500 words