Semantically Consistent Data Augmentation for Neural Machine Translation via Conditional Masked Language Model

Proceedings of the 29th International Conference on Computational Linguistics, pages 5148–5157 October 12–17, 2022. 5148 Semantically Consistent Data Augmentation for Neural Machine Translation via Conditional Masked Language Model Qiao Cheng, Jin Huang, Yitao Duan NetEase Youdao Beijing, China {chengqiao, huangjin, duan}@rd.netease.com Abstract This paper introduces a new data augmenta- tion method for neural machine translation that can enforce stronger semantic consistency both within and across languages. Our method is based on Conditional Masked Language Model (CMLM) which is bi-directional and can be conditional on both left and right context, as well as the label. We demonstrate that CMLM is a good technique for generating context- dependent word distributions. In particular, we show that CMLM is capable of enforcing se- mantic consistency by conditioning on both source and target during substitution. In addi- tion, to enhance diversity, we incorporate the idea of soft word substitution for data augmen- tation which replaces a word with a probabilis- tic distribution over the vocabulary. Experi- ments on four translation datasets of different scales show that the overall solution results in more realistic data augmentation and better translation quality. Our approach consistently achieves the best performance in comparison with strong and recent works and yields im- provements of up to 1.90 BLEU points over the baseline. 1 1 Introduction Neural network models have achieved remarkable results in many fields such as computer vision, nat- ural language processing, and speech. In order to obtain adequate expressivity, the models usually come with a large number of parameters. How- ever, such models are prone to overfitting if trained with an insufficient amount of training data. Data Augmentation (DA) is an effective technique that has been used in many areas to augment existing labeled data and boost the performance of machine learning models. For example, in computer vision, training data is often augmented by ways such as horizontal flipping, random cropping, tilting, and 1Our code is available at https://github.com/ netease-youdao/cmlm_da. color shifting (Krizhevsky et al., 2012; Cubuk et al., 2018). While DA has become a standard technique to train deep networks for image processing, it is relatively under-explored in Natural Language Pro- cessing (NLP). The exact mechanisms and theoretical founda- tions of data augmentation are still under investi- gation. Most studies show empirically that data augmentation is effective and provide some intu- itive explanations. A recent work in the field of vision (Gontijo-Lopes et al., 2020) demonstrates that affinity (the distributional shift caused by DA) and diversity (the complexity of the augmentation) can predict the performance of data augmentation methods. However, neither metric can be measured without completing the entire DA process. There- fore, it is still challenging to evaluate the goodness of a DA technique without full-fledged experimen- tation and it is not clear how the result can be used to guide the design of data augmentation schemes. Generally speaking, DA can be classified into two categories. The first tries to produce realistic samples that resemble the inherent semantics of naturally generated data. In areas such as com- …

Content truncated to 500 words