Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7930 - 7944 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation Xiangpeng Wei†, Heng Yu†, Yue Hu‡,§, Rongxiang Weng†, Weihua Luo†, Rong Jin† †Machine Intelligence Technology Lab, Alibaba DAMO Academy, Hangzhou, China ‡Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China §School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China pemywei@gmail.com https://github.com/pemywei/csanmt Abstract The principal task in supervised neural ma- chine translation (NMT) is to learn to gener- ate target sentences conditioned on the source inputs from a set of parallel sentence pairs, and thus produce a model capable of gener- alizing to unseen instances. However, it is com- monly observed that the generalization perfor- mance of the model is highly influenced by the amount of parallel data used in training. Al- though data augmentation is widely used to enrich the training data, conventional meth- ods with discrete manipulations fail to gener- ate diverse and faithful training samples. In this paper, we present a novel data augmen- tation paradigm termed Continuous Semantic Augmentation (CSANMT), which augments each training instance with an adjacency se- mantic region that could cover adequate vari- ants of literal expression under the same mean- ing. We conduct extensive experiments on both rich-resource and low-resource settings involving various language pairs, including WMT14 English→{German,French}, NIST Chinese→English and multiple low-resource IWSLT translation tasks. The provided empiri- cal evidences show that CSANMT sets a new level of performance among existing augmenta- tion techniques, improving on the state-of-the- art by a large margin.1 1 Introduction Neural machine translation (NMT) is one of the core topics in natural language processing, which aims to generate sequences of words in the target language conditioned on the source in- puts (Sutskever et al., 2014; Cho et al., 2014; Wu et al., 2016; Vaswani et al., 2017). In the common supervised setting, the training objective is to learn a transformation from the source space to the target space X �→ Y : f(y|x; Θ) with the usage of paral- lel data. In this way, NMT models are expected to 1The core codes are contained in Appendix E. be capable of generalizing to unseen instances with the help of large scale training data, which poses a big challenge for scenarios with limited resources. To address this problem, various methods have been developed to leverage abundant unlabeled data for augmenting limited labeled data (Sen- nrich et al., 2016a; Cheng et al., 2016; He et al., 2016; Hoang et al., 2018; Edunov et al., 2018; He et al., 2020; Song et al., 2019). For example, back- translation (BT) (Sennrich et al., 2016a) makes use of the monolingual data on the target side to syn- thesize large scale pseudo parallel data, which is further combined with the real parallel corpus in machine translation task. Another line of research is to introduce adversarial inputs to improve the generalization …

Content truncated to 500 words