From characters to words: the turning point of BPE merges

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3454–3468 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3454 From characters to words: the turning point of BPE merges Ximena Gutierrez-Vasques1 Christian Bentz2 Olga Sozinova1 Tanja Samardˇzi´c1 1URPP Language and Space, University of Z¨urich 2Department of General Linguistics, University of T¨ubingen {ximena.gutierrezvasques, olga.sozinova, tanja.samardzic}@uzh.ch chris@christianbentz.de Abstract The distributions of orthographic word types are very different across languages due to ty- pological characteristics, different writing tra- ditions, and other factors. The wide range of cross-linguistic diversity is still a major chal- lenge for NLP, and for the study of language more generally. We use BPE and information- theoretic measures to investigate if distribu- tions become more similar under specific lev- els of subword tokenization. We perform a cross-linguistic comparison, following incre- mental BPE merges (we go from characters to words) for 47 diverse languages. We show that text entropy values (a feature of probabil- ity distributions) converge at specific subword levels: relatively few BPE merges (around 200 for our corpus) lead to the most similar dis- tributions across languages. Additionally, we analyze the interaction between subword and word-level distributions and show that our find- ings can be interpreted in light of the ongoing discussion about different morphological com- plexity types.1 1 Introduction In NLP, one of the predominant methods for ob- taining subword units is Byte-Pair Encoding (BPE). These subwords have proven to be useful for im- proving several NLP tasks, most likely because they capture morphological patterns to some extent (and also phonological and orthographic ones). BPE is based on a compression algorithm which finds frequently ocurring patterns in a text by means of incrementally merging adjacent symbols into longer strings (Gage, 1994; Sennrich et al., 2015). The granularity of the subword units is con- trolled by the number of merge operations applied to the text (few merges lead to a text tokenization closer to the character level, while more merges lead to a tokenization closer to the word level). 1Data and code available at https://github.com/ ximenina/theturningpoint Usually the number of BPE merges is chosen arbi- trarily depending on the application. It is rarely analyzed how the distribution of these subwords changes across different merge opera- tions. Our goal is to investigate if languages get ‘closer’ in terms of their subword distributions un- der specific levels of tokenization. We quantify this cross-linguistic variation using information- theoretic measures. Information theory provides a useful tool for exploring variation, and for quantifying the pre- dictability/organization of patterns, e.g., in mor- phological systems (Ackerman and Malouf, 2013). We measure Shannon entropy and redundancy over varied subword tokenizatons of texts obtained with BPE. At each incremental merge, we compare the values across 47 typologically diverse languages. Cross-linguistic corpora are widely used as a means of quantifying linguistic diversity. For in- stance, the range of entropy values measured over word-level types varies greatly across languages. This is a reflection of the diversity of morpholog- ical systems. However, …

Content truncated to 500 words