Soft Contextual Data Augmentation for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5539–5544 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5539 Soft Contextual Data Augmentation for Neural Machine Translation Fei Gao1,∗, Jinhua Zhu2,∗, Lijun Wu3, Yingce Xia4, Tao Qin4, Xueqi Cheng1, Wengang Zhou2, Tie-Yan Liu4 1Institute of Computing Technology, Chinese Academy of Sciences; 2University of Science and Technology of China, 3Sun Yat-sen University, 4Microsoft Reserach Asia; 1{gaofei17b, cxq}@ict.ac.cn, 2{teslazhu@mail., zhwg@}ustc.edu.cn, 3wulijun3@mail2.sysu.edu.cn, 4{Yingce.Xia, taoqin, tyliu}@microsoft.com Abstract While data augmentation is an important trick to boost the accuracy of deep learning meth- ods in computer vision tasks, its study in nat- ural language tasks is still very limited. In this paper, we present a novel data augmen- tation method for neural machine translation. Different from previous augmentation meth- ods that randomly drop, swap or replace words with other words in a sentence, we softly aug- ment a randomly chosen word in a sentence by its contextual mixture of multiple related words. More accurately, we replace the one- hot representation of a word by a distribu- tion (provided by a language model) over the vocabulary, i.e., replacing the embedding of this word by a weighted combination of mul- tiple semantically similar words. Since the weights of those words depend on the contex- tual information of the word to be replaced, the newly generated sentences capture much richer information than previous augmenta- tion methods. Experimental results on both small scale and large scale machine transla- tion datasets demonstrate the superiority of our method over strong baselines1. 1 Introduction Data augmentation is an important trick to boost the accuracy of deep learning methods by gener- ating additional training samples. These methods have been widely used in many areas. For ex- ample, in computer vision, the training data are augmented by transformations like random rota- tion, resizing, mirroring and cropping (Krizhevsky et al., 2012; Cubuk et al., 2018). While similar random transformations have also been explored in natural language processing (NLP) tasks (Xie et al., 2017), data augmentation ∗The first two authors contributed equally to this work. 1Our code can be found at https://github.com/ teslacool/SCA is still not a common practice in neural machine translation (NMT). For a sentence, existing meth- ods include randomly swapping two words, drop- ping word, replacing word with another one and so on. However, due to text characteristics, these random transformations often result in significant changes in semantics. A recent new method is contextual augmenta- tion (Kobayashi, 2018; Wu et al., 2018), which re- places words with other words that are predicted using language model at the corresponding word position. While such method can keep seman- tics based on contextual information, this kind of augmentation still has one limitation: to gener- ate new samples with adequate variation, it needs to sample multiple times. For example, given a sentence in which N words are going to be re- placed with other words predicted by one language model, there could be as many as exponential can- …

Content truncated to 500 words