BPE-Dropout: Simple and Effective Subword Regularization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1882 BPE-Dropout: Simple and Effective Subword Regularization Ivan Provilkov∗ 1,2 Dmitrii Emelianenko∗ 1,3 Elena Voita4,5 1Yandex, Russia 2Moscow Institute of Physics and Technology, Russia 3National Research University Higher School of Economics, Russia 4University of Edinburgh, Scotland 5University of Amsterdam, Netherlands {iv-provilkov, dimdi-y, lena-voita}@yandex-team.ru Abstract Subword segmentation is widely used to ad- dress the open vocabulary problem in ma- chine translation. The dominant approach to subword segmentation is Byte Pair Encod- ing (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmenta- tions are possible even with the same vocabu- lary, BPE splits words into unique sequences; this may prevent a model from better learn- ing the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfec- tion, its deterministic nature, was to create an- other subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself in- corporates the ability to produce multiple seg- mentations of the same word. We introduce BPE-dropout – simple and effective subword regularization method based on and compat- ible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmenta- tions within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves trans- lation quality up to 2.3 BLEU compared to BPE and up to 0.9 BLEU compared to the pre- vious subword regularization. 1 Introduction Using subword segmentation has become de-facto standard in Neural Machine Translation (Bojar et al., 2018; Barrault et al., 2019). Byte Pair En- coding (BPE) (Sennrich et al., 2016) is the domi- nant approach to subword segmentation. It keeps the common words intact while splitting the rare and unknown ones into a sequence of subword units. This potentially allows a model to make ∗Equal contribution. use of morphology, word composition and translit- eration. BPE effectively deals with an open- vocabulary problem and is widely used due to its simplicity. There is, however, a drawback of BPE in its de- terministic nature: it splits words into unique sub- word sequences, which means that for each word a model observes only one segmentation. Thus, a model is likely not to reach its full potential in exploiting morphology, learning the composition- ality of words and being robust to segmentation errors. Moreover, as we will show further, sub- words into which rare words are segmented end up poorly understood. A natural way to handle this problem is to en- able multiple segmentation candidates. This was initially proposed by Kudo (2018) as a subword regularization – a regularization method, which is implemented as an on-the-fly data sampling and is not specific to NMT architecture. Since stan- dard BPE produces single segmentation, to realize this regularization the author had to propose a new subword segmentation, different from BPE. How- ever, the …

Content truncated to 500 words