What is the best recipe for character-level encoder-only modelling?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5924–5938 July 9-14, 2023 ©2023 Association for Computational Linguistics What is the best recipe for character-level encoder-only modelling? Kris Cao DeepMind, London, UK kriscao@deepmind.com Abstract This paper aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. Many such modelling architectures and methods to train those architectures have been proposed, but it is currently unclear what the relative contributions of the architecture vs. the pretraining objective are to final model perfor- mance. We explore the design space of such models, comparing architectural innovations (Clark et al., 2022; Jaegle et al., 2022; Tay et al., 2021), and a variety of different pretrain- ing objectives on a suite of evaluation tasks in order to find the optimal way to build and train character-level BERT-like models. We find that the best recipe combines the Char- former and CANINE model architectures, and follows the CANINE training procedure. This model exceeds the performance of a token- based model trained with the same settings on the same data, suggesting that character-level models are ready for more widespread adop- tion. Unfortunately, the best method to train character-level models still relies on a learnt tokeniser during pretraining, and final model performance is highly dependent on tokeniser quality. We believe our results demonstrate the readiness of character-level models for multi- lingual language representation, and encourage NLP practitioners to try them for their needs. 1 Introduction The first stage of almost all NLP modelling pipelines is to convert input text strings into a se- quence of symbols that the model can ingest. This step, called tokenisation, can be highly non-trivial and introduces significant theoretical and practical complexities to both training and using these mod- els. One particular issue for massively multilingual models is that many languages have to compete for space given a fixed vocabulary size, which limits the effective vocabulary size per language: as an illustration, the WordPiece tokeniser that multilin- gual BERT uses tokenises ‘hello’ as two tokens: ‘hell’ and ‘##o’.1 We are therefore interested in al- ternative approaches which use lightweight tokeni- sation schemes (in particular character-level seg- mentation) coupled with powerful neural-network based composition functions to build language models (see Section 5.1 for a discussion of the benefits of character-level modelling). In this pa- per, we aim to determine the best way to build such models, focussing on models which output vector representations for each input character. However, as the field of pretrained character- level modelling is relatively new, comparisons are complicated by the fact that recently proposed methods use different model architectures, pretrain on different data using different training objectives, and evaluate on different downstream tasks. With so many variables changing simultaneously, it is difficult to disentangle the effect of each individ- ual choice in the modelling pipeline, and there- fore decide on an overall best model configuration. To answer this question, we tested many model architectures and pretraining objectives from …

Content truncated to 500 words