You should evaluate your language model on marginal likelihood over tokenisations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104–2114 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2104 You should evaluate your language model on marginal likelihood over tokenisations Kris Cao and Laura Rimell DeepMind, London, UK {kriscao, laurarimell}@deepmind.com Abstract Neural language models typically tokenise in- put text into sub-word units to achieve an open vocabulary. The standard approach is to use a single canonical tokenisation at both train and test time. We suggest that this approach is un- satisfactory and may bottleneck our evaluation of language model performance. Using only the one-best tokenisation ignores tokeniser un- certainty over alternative tokenisations, which may hurt model out-of-domain performance. In this paper, we argue that instead, lan- guage models should be evaluated on their marginal likelihood over tokenisations. We compare different estimators for the marginal likelihood based on sampling, and show that it is feasible to estimate the marginal likeli- hood with a manageable number of samples. We then evaluate pretrained English and Ger- man language models on both the one-best- tokenisation and marginal perplexities, and show that the marginal perplexity can be signif- icantly better than the one best, especially on out-of-domain data. We link this difference in perplexity to the tokeniser uncertainty as mea- sured by tokeniser entropy. We discuss some implications of our results for language model training and evaluation, particularly with re- gard to tokenisation robustness. 1 Introduction Neural end-to-end language models have largely done away with traditional pipeline approaches to- wards building NLP systems. However, one compo- nent which stubbornly remains is the tokenisation step, used right at the start of preprocessing. At the time of writing, the most widely used tokenisers, such as BPE (Sennrich et al., 2016) and unigram (Kudo, 2018), break up the input text into subword units, potentially backing off to character-level seg- mentation if necessary. This allows for coverage of every possible input sequence; on the downside, a single input sequence may now have multiple possible tokenisations. Typically, language models are trained and eval- uated using a single canonical tokenisation out of the multitude of possible ones, but this tokenisation may be suboptimal (Bostrom and Durrett, 2020) for many reasons. For example, different tokenisations – that is, different surface segmentations – can re- veal different morphological analyses of the word in question (think un-ion-izeable vs. union-izable), and committing to a particular analysis can discard useful information, particularly if the best analysis from the tokeniser is erroneous (Dyer, 2010). Further, tokenisers themselves are trained us- ing an objective which optimises the likelihood of the data. This can be explicit (the unigram to- keniser of Kudo (2018) optimises a unigram lan- guage modelling objective) or implicit (BPE aims to minimise the description length of the training data, which has close connections to probabilistic methods; MacKay 2003). In this sense they are also language models, albeit far less powerful than the neural language models we train on their out- puts. This raises a difficult question: to what extent are our …

Content truncated to 500 words