A Statistical Extension of Byte-Pair Encoding

Proceedings of the 18th International Conference on Spoken Language Translation, pages 263–275 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 263 A Statistical Extension of Byte-Pair Encoding David Vilar∗ Amazon Marcello Federico Amazon Abstract Sub-word segmentation is currently a stan- dard tool for training neural machine transla- tion (MT) systems and other NLP tasks. The goal is to split words (both in the source and target languages) into smaller units which then constitute the input and output vocabularies of the MT system. The aim of reducing the size of the input and output vocabularies is to increase the generalization capabilities of the translation model, enabling the system to trans- late and generate infrequent and new (unseen) words at inference time by combining previ- ously seen sub-word units. Ideally, we would expect the created units to have some linguistic meaning, so that words are created in a compo- sitional way. However, the most popular word- splitting method, Byte-Pair Encoding (BPE), which originates from the data compression lit- erature, does not include explicit criteria to fa- vor linguistic splittings, nor to find the optimal sub-word granularity for the given training data. In this paper, we propose a statistically moti- vated extension of the BPE algorithm and an effective convergence criterion that avoids the costly experimentation cycle needed to select the best sub-word vocabulary size. Experimen- tal results with morphologically rich languages show that our model achieves nearly-optimal BLEU scores and produces morphologically better word segmentations, which allows to out- perform BPE’s generalization in the translation of sentences containing new words, as shown via human evaluation. 1 Introduction Sub-word segmentation is currently a standard tool for machine translation systems (see e.g. the sys- tems submitted to WMT and IWSLT evaluations (Barrault et al., 2019; Niehues et al., 2019), as well as systems for a wide variety of NLP tasks (see e.g. Devlin et al. (2018) and derived works). The goal ∗ Now at Google. is to split words (both in the source and target lan- guage) into smaller units which then constitute the input and output of the machine translation system. The goal is twofold: On the one hand, sub-word splitting reduces the size of the input and output vo- cabularies. This is specially important when using neural models, as the size of the input layer is fixed and thus the vocabulary size cannot be dynamically adjusted. On the other hand, it tries to increase the generalization capabilities of the translation model, enabling the system to accept and/or generate new words at translation time by combining previously seen units. The most widespread method used for sub-word splitting in neural machine translation is Byte Pair Encoding (BPE), introduced by Sennrich et al. (2016). Since then, BPE has become a default preprocessing step for many NLP tasks. The BPE extraction algorithm is an adaptation of the algorithm introduced by Gage (1994) for data compression. The main idea of this algorithm is to replace the most frequent pair of bytes found in the …

Content truncated to 500 words