A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 1725–1735 May 2-6, 2023 ©2023 Association for Computational Linguistics A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models Jimin Sun1,2 Patrick Fernandes1 Xinyi Wang1 Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University 2Kakao Enterprise {jimins2,pfernand,xinyiw1,gneubig}@cs.cmu.edu Abstract Recent works on tokenizer-free multilingual pretrained models show promising results in improving cross-lingual transfer and reducing engineering overhead compared to subword- based alternatives. However, previous work mainly focuses on reporting accuracy on a lim- ited set of tasks and data settings, placing less emphasis on other important factors when tun- ing and deploying the models in practice, such as memory usage, inference speed, and fine- tuning data efficiency. We attempt to fill this gap by performing a comprehensive empirical comparison of multilingual tokenizer-free and subword-based models considering the various dimensions. Surprisingly, we find that subword- based models might still be the most practical choice in many settings, achieving better perfor- mance for lower inference latency and memory usage. Based on these results, we encourage fu- ture work in tokenizer-free methods to consider these factors when designing and evaluating new models.1 1 Introduction Several recent results (Clark et al., 2022; Xue et al., 2022) have excited the research commu- nity with the possibility of “tokenizer-free” models, character-level and byte-level models, as an alter- native to more traditional subword-based models. Tokenizer-free models are especially appealing to practitioners as they can eschew the two-step pro- cessing pipeline of subword segmentation and re- duce the corresponding difficulties in cross-lingual transfer (Hu et al., 2020; Maronikolakis et al., 2021; Rust et al., 2021; Wang et al., 2021) or domain adaptation (Sato et al., 2020; Liu et al., 2021) due to inconsistent subword units. However, upon several attempts to apply tokenizer-free methods, our analysis reveals sev- eral practical difficulties in applying these methods. 1We will release code to train and evaluate models upon de-anonymization. This paper is a chronicle of some of the concerns we uncovered; we highlight some challenges with applying these models and propose best practices for future results reporting in this area. Specifically, we perform experiments finetuning pretrained multilingual models, evaluating them with respect to finetuning data efficiency, inference time, and memory consumption. Based on these multiple dimensions, we come to the somewhat surprising conclusion that subword-based models, in particular mBERT (Devlin et al., 2019), might still be the most practical choice in most settings, as they perform best while maintaining a relatively low inference cost. 2 Tokenizer-free Multilingual Models While multilingual pretrained models (Devlin et al., 2019; Lample and Conneau, 2019; Liu et al., 2020; Xue et al., 2021) have led to impressive perfor- mance improvements for low-resource languages through cross-lingual transfer, the standard word representation method in these models relies on subword segmentation (Sennrich et al., 2016; Kudo, 2018). In multilingual settings, subword tokeniza- tion can be sub-optimal as supporting hundreds of languages with various scripts and vocabularies causes segmentation mismatch between languages and over-segmentation in the lower-resourced lan- guages (Wang et al., 2020). To alleviate …

Content truncated to 500 words