How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3118–3135 August 1–6, 2021. ©2021 Association for Computational Linguistics 3118 How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models Phillip Rust∗1†, Jonas Pfeiffer∗1, Ivan Vuli´c2, Sebastian Ruder3, Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt 2Language Technology Lab, University of Cambridge 3DeepMind www.ukp.tu-darmstadt.de Abstract In this work, we provide a systematic and comprehensive empirical comparison of pre- trained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the correspond- ing monolingual representation model of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data, with monolingually and multilingually trained tok- enizers. We find that while the pretraining data size is an important factor, a designated mono- lingual tokenizer plays an equally important role in the downstream performance. Our re- sults show that languages that are adequately represented in the multilingual model’s vocab- ulary exhibit negligible performance decreases over their monolingual counterparts. We fur- ther find that replacing the original multilin- gual tokenizer with the specialized monolin- gual tokenizer improves the downstream per- formance of the multilingual model for almost every task and language. 1 Introduction Following large transformer-based language mod- els (LMs, Vaswani et al., 2017) pretrained on large English corpora (e.g., BERT, RoBERTa, T5; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020), similar monolingual language models have been in- troduced for other languages (Virtanen et al., 2019; ∗Both authors contributed equally to this work. †PR is now affiliated with the University of Copenhagen. Our code is available at https://github.com/Adapter-Hub/hgiyt. Antoun et al., 2020; Martin et al., 2020, inter alia), offering previously unmatched performance in all NLP tasks. Concurrently, massively multilingual models with the same architectures and training procedures, covering more than 100 languages, have been proposed (e.g., mBERT, XLM-R, mT5; Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2021). The “industry” of pretraining and releasing new monolingual BERT models continues its operations despite the fact that the corresponding languages are already covered by multilingual models. The common argument justifying the need for mono- lingual variants is the assumption that multilingual models—due to suffering from the so-called curse of multilinguality (Conneau et al., 2020, i.e., the lack of capacity to represent all languages in an eq- uitable way)—underperform monolingual models when applied to monolingual tasks (Virtanen et al., 2019; Antoun et al., 2020; R¨onnqvist et al., 2019, inter alia). However, little to no compelling em- pirical evidence with rigorous experiments and fair comparisons have been presented so far to support or invalidate this …

Content truncated to 500 words