Unifying Input and Output Smoothing in Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4361–4372 Barcelona, Spain (Online), December 8-13, 2020 4361 Unifying Input and Output Smoothing in Neural Machine Translation Yingbo Gao* Baohao Liao* Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {gao|bliao|ney}@i6.informatik.rwth-aachen.de Abstract Soft contextualized data augmentation is a recent method that replaces one-hot representation of words with soft posterior distributions of an external language model, smoothing the input of neural machine translation systems. Label smoothing is another effective method that penalizes over-confident model outputs by discounting some probability mass from the true target word, smoothing the output of neural machine translation systems. Having the benefit of updating all word vectors in each optimization step and better regularizing the models, the two smoothing methods are shown to bring significant improvements in translation performance. In this work, we study how to best combine the methods and stack the improvements. Specifically, we vary the prior distributions to smooth with, the hyperparameters that control the smoothing strength, and the token selection procedures. We conduct extensive experiments on small datasets, evaluate the recipes on larger datasets, and examine the implications when back-translation is further used. Our results confirm cumulative improvements when input and output smoothing are used in combination, giving up to +1.9 BLEU scores on standard machine translation tasks and reveal reasons why these smoothing methods should be preferred. 1 Introduction Nowadays, neural network models are commonly used for the task of machine translation. The Trans- former (Vaswani et al., 2017) architecture is the default choice for many competitive systems (Bojar et al., 2018; Barrault et al., 2019; Ott et al., 2018). In order to make use of large amount of available data in the target language, among others, back-translation is a frequently used method (Sennrich et al., 2016a; Edunov et al., 2018; Grac¸a et al., 2019). Recently, a method under the name “soft contextualized data augmentation” (Gao et al., 2019) is introduced and focus on the input side of neural machine translation models. Intuitively, the method smoothes both the source input and the target input to the model. While one-hot vectors are traditionally used to feed the word / token information to the network, this method instead uses an external language model (LM) trained on the parallel data without additional monolingual data, and achieve a “smoother” input where a “linear interpolation” operation is done on the word embedding matrices, instead of the usual “table lookup”. Conceptually, the “input smoothing” method above largely resembles the “label smoothing” method used in the output side. Initially introduced in the field of image recognition (Szegedy et al., 2016), label smoothing is included in the original Transformer setup (Vaswani et al., 2017) by default. Deducting a certain probability mass from the true target and redistributing uniformly across the vocabulary, label smoothing can also be thought of as penalizing over-confident system outputs. From here on, to highlight our motivation, we will use “input smoothing” and “output smoothing” to refer to …

Content truncated to 500 words