Improving Tokenisation by Alternative Treatment of Spaces

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11430 - 11443 December 7-11, 2022 ©2022 Association for Computational Linguistics Improving Tokenisation by Alternative Treatment of Spaces Edward Gow-Smith1, Harish Tayyar Madabushi2, Carolina Scarton1 and Aline Villavicencio1 1Department of Computer Science, University of Sheffield 2Department of Computer Science, University of Bath egow-smith1@sheffield.ac.uk Abstract Tokenisation is the first step in almost all NLP tasks, and state-of-the-art transformer-based language models all use subword tokenisation algorithms to process input text. Existing al- gorithms have problems, often producing to- kenisations of limited linguistic validity and representing equivalent strings differently de- pending on their position within a word. We hypothesise that these problems hinder the ability of transformer-based models to handle complex words, and suggest that these prob- lems are a result of allowing tokens to include spaces. We thus experiment with an alternative tokenisation approach where spaces are always treated as individual tokens. Specifically, we apply this modification to the BPE and Uni- gram algorithms. We find that our modified algorithms lead to improved performance on downstream NLP tasks that involve handling complex words, whilst having no detrimental effect on performance in general natural lan- guage understanding tasks. Intrinsically, we find that our modified algorithms give more morphologically correct tokenisations, in par- ticular when handling prefixes. Given the re- sults of our experiments, we advocate for al- ways treating spaces as individual tokens as an improved tokenisation method. 1 Introduction Tokenisation is a key initial step in processing nat- ural language, as it identifies the linguistic units to be processed, converting them to numerical IDs which can then be vectorised and manipulated by mathematical operations. Earlier NLP approaches used simple string- searching techniques with regular expressions to tokenise text; however, these pattern-matching to- kenisation methods have drawbacks: they require large vocabulary sizes to cover the training data, they cannot handle out-of-vocabulary words, and they do not work for languages without spaces as word boundaries. To address these issues, subword tokenisation was introduced. The first explicit men- tion (and popularisation) of this approach was by Sennrich et al. (2015), though it was indirectly in- troduced earlier by Schuster and Nakajima (2012). This method works by learning from training data to build a vocabulary (of a fixed size) and then tokenising text at inference time using this vocabu- lary (and possibly other learnt parameters). More frequent words are represented as single tokens, with rare words being broken down into multiple subword tokens, possibly down to the character level. State-of-the art transformer-based language mod- els all use subword tokenisation algorithms based on either byte-pair encoding (BPE) (Sennrich et al., 2015) or Unigram (Kudo, 2018). The original trans- former model (Vaswani et al., 2017) uses BPE, whilst BERT (Devlin et al., 2018), which consists of a transformer encoder pretrained with a masked language modelling objective, uses WordPiece to- kenisation (Schuster and Nakajima, 2012), which is a variant of BPE with a language model loss func- tion. WordPiece is also used by ERNIE (Sun et …

Content truncated to 500 words