A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation

A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation Shuoyang Ding† Adithya Renduchintala† Kevin Duh†‡ † Center for Language and Speech Processing ‡ Human Language Technology Center of Excellence Johns Hopkins University {dings, adi.r}@jhu.edu kevinduh@cs.jhu.edu Abstract Most neural machine translation systems are built upon subword units extracted by methods such as Byte-Pair Encoding (BPE) or wordpiece. However, the choice of num- ber of merge operations is generally made by following existing recipes. In this pa- per, we conduct a systematic exploration on different numbers of BPE merge oper- ations to understand how it interacts with the model architecture, the strategy to build vocabularies and the language pair. Our exploration could provide guidance for se- lecting proper BPE configurations in the future. Most prominently: we show that for LSTM-based architectures, it is neces- sary to experiment with a wide range of different BPE operations as there is no typ- ical optimal BPE configuration, whereas for Transformer architectures, smaller BPE size tends to be a typically optimal choice. We urge the community to make prudent choices with subword merge operations, as our experiments indicate that a sub-optimal BPE configuration alone could easily re- duce the system performance by 3–4 BLEU points. 1 Introduction While achieving state-of-the-art results, it is a com- mon constraint that Neural Machine Translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017) sys- tems are only capable of generating a closed set of © 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. symbols. Systems with large vocabulary sizes are too hard to fit onto GPU for training, as the word embedding is generally the most parameter-dense component in the NMT architecture. For that rea- son, subword methods, such as Byte-Pair Encoding (BPE) (Sennrich et al., 2016), are very widely used for building NMT systems. The general idea of these methods is to exploit the pre-defined vocab- ulary space optimally by performing a minimum amount of word segmentations in the training set. However, very few existing literature carefully examines what is the best practice regarding appli- cation of subword methods. As hyper-parameter search is expensive, there is a tendency to simply use existing recipes. This is especially true for the number of merge operations when people are using BPE, although this configuration is closely correlated with the granularity of the segmentation on the training corpus, thus having direct influ- ence on the final system performance. Prior to this work, Denkowski and Neubig (2017) recom- mended 32k BPE merge operation in their work on trustable baselines for NMT, while Cherry et al. (2018) contradicted their study by showing that character-based models outperform 32k BPE. Both of these studies are based on the LSTM-based ar- chitectures (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015). To the best of our knowledge, there is no work that looks into the same problem for the Transformer architecture …

Content truncated to 500 words