Incorporating Context into Subword Vocabularies

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 623–635 May 2-6, 2023 ©2023 Association for Computational Linguistics Incorporating Context into Subword Vocabularies Shaked Yehezkel Blavatnik School of Computer Science Tel-Aviv University Tel-Aviv, Israel shakedy@mail.tau.ac.il Yuval Pinter Department of Computer Science Ben-Gurion University of the Negev Beer Sheva, Israel uvp@cs.bgu.ac.il Abstract Most current popular subword tokenizers are trained based on word frequency statistics over a corpus, without considering informa- tion about co-occurrence or context. Neverthe- less, the resulting vocabularies are used in lan- guage models’ highly contextualized settings. We present SAGE, a tokenizer that tailors sub- words for their downstream use by baking in the contextualized signal at the vocabulary cre- ation phase. We show that SAGE does a bet- ter job than current widespread tokenizers in keeping token contexts cohesive, while not in- curring a large price in terms of encoding effi- ciency or domain robustness. SAGE improves performance on English GLUE classification tasks as well as on NER, and on Inference and NER in Turkish, demonstrating its robustness to language properties such as morphological exponence and agglutination. 1 Introduction Much of the research space in current NLP is focused on advancing models: modifying pre- training objectives, improving network architec- tures, adding tasks and schemes for downstream evaluation. Limited work is dedicated to a crucial step underlying all modern large language models (LLMs), namely the tokenization phase. In order to process a given string of text, an LLM must first obtain a vector representation of the input by seg- menting it into tokens. Since out-of-vocabulary (OOV) items inhibit the performance of models, current tokenizers produce tokens which are possi- bly proper subsegments of input words, known as subwords. This method, popularized by systems such as WordPiece (Schuster and Nakajima, 2012), Byte-Pair Encoding (BPE; Sennrich et al., 2016) and UNIGRAMLM (Kudo, 2018), allows any word to be represented by one or more tokens, removing the OOV problem while allowing more flexibility in determining the token vocabulary size, which ultimately affects model speed (mostly through BPE His son Raj ash ri Sud h ak ar has p enn ed dial og ues and songs for some films that were dubbed into Telugu . SAGE His son Raj ash r i Sud h a k a r has penn e d dial ogues and songs for some films that were dubbed into Telugu . BPE This gene is a pseud og ene in humans and most other prim ates . SAGE This gene is a pseud ogene in humans and most other prim ates . BPE The St o og es work for Mir acle Det ective Agency , SAGE The St o o g e s work for Mir acle Det ective Agency , Table 1: The token og is selected by BPE (vocabulary of size 16,000) for achieving the frequency objective, but is discarded by SAGE for failing to be contextually coherent. These examples from the corpus demonstrate some different contexts. the softmax …

Content truncated to 500 words