Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT Benoist Wolleb1, Romain Silvestri1, Giorgos Vernikos1,2, Ljiljana Dolamic3 and Andrei Popescu-Belis1,2 1HEIG-VD / HES-SO 2EPFL 3Armasuisse W+T Yverdon-les-Bains Lausanne Thun Switzerland Switzerland Switzerland name.surname@heig-vd.ch ljiljana.dolamic@armasuisse.ch Abstract Subword tokenization is the de facto stan- dard for tokenization in neural language models and machine translation systems. Three advantages are frequently cited in favor of subwords: shorter encoding of frequent tokens, compositionality of sub- words, and ability to deal with unknown words. As their relative importance is not entirely clear yet, we propose a tokeniza- tion approach that enables us to separate frequency (the first advantage) from com- positionality. The approach uses Huffman coding to tokenize words, by order of fre- quency, using a fixed amount of symbols. Experiments with CS-DE, EN-FR and EN- DE NMT show that frequency alone ac- counts for 90%-95% of the scores reached by BPE, hence compositionality has less importance than previously thought. 1 Introduction Tokenization into subwords has become an unchal- lenged standard used in virtually all NMT sys- tems and language models. Since the proposal by Sennrich et al. (2016) to use Byte-Pair En- coding (BPE) (Gage, 1994) to create subword vo- cabularies, followed by the use of a unigram lan- guage model and the SentencePiece implemen- tation (Kudo, 2018), no alternative models have taken over. While subwords have been empirically demonstrated to outperform character and word- level tokenization (Sennrich et al., 2016; Wu et al., 2016; Denkowski and Neubig, 2017), the factors © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. contributing to their success have not been fully understood yet. Some studies have investigated the performance of subwords with regard to com- pression (Gall´e, 2019; Libovick´y et al., 2022), sug- gesting that better compression may be associated with improved performance. However, other fac- tors such as compositionality have yet to be thor- oughly explored. In this paper, we use an alternative algorithm for creating subword vocabularies, which retains only one of the features that have been invoked to explain the effectiveness of BPE, namely the fact that frequent words are encoded as unique sub- words, while less frequent ones are encoded us- ing several subwords, possibly up to the charac- ter level. The algorithm is based on Huffman cod- ing (Huffman, 1952), a different text compression method than the one used by BPE. The algorithm differs from BPE in two key aspects: while cer- tain BPE subwords convey compositional linguis- tic properties (e.g., meaning or morphology), Huff- man coding is fundamentally non-compositional, and cannot tokenize words not seen during train- ing. When using Huffman coding to tokenize data for Transformer-based MT, we reach scores that are within 10-12% of those obtained using BPE when measured by BLEU and within 4-8% when measured by COMET, for vocabulary sizes of 32k symbols. This demonstrates that the main factor accounting for the success of BPEs is word fre- quency, and not subword compositionality. Our …

Content truncated to 500 words