Active Learning Approaches to Enhancing Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1796–1806 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1796 Active Learning Approaches to Enhancing Neural Machine Translation Yuekai Zhao1 Haoran Zhang1 Shuchang Zhou2 Zhihua Zhang3 1 Academy for Advanced Interdisciplinary Studies, Peking University 2 Megvii Inc. 3 School of Mathematical Sciences, Peking University {yuekaizhao, haoran zhang}@pku.edu.cn zsc@megvii.com zhzhang@math.pku.edu.cn Abstract Active learning is an efficient approach for mit- igating data dependency when training neural machine translation (NMT) models. In this pa- per we explore new training frameworks by in- corporating active learning into various tech- niques such as transfer learning and iterative back-translation (IBT) under a limited human translation budget. We design a word fre- quency based acquisition function and com- bine it with a strong uncertainty based method. The combined method steadily outperforms all other acquisition functions in various scenar- ios. As far as we know, we are the first to do a large-scale study on actively training Trans- former (Vaswani et al., 2017) for NMT. Specif- ically, with a human translation budget of only 20% of the original parallel corpus, we man- age to surpass Transformer trained on the en- tire parallel corpus in three language pairs. 1 Introduction Many impressive progresses have been made in neural machine translation (NMT) in the past few years (Luong et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Wu et al., 2019). However, the general training procedure requires tremendous amounts of high-quality parallel corpus to achieve a deep model’s full potential. The scarcity of the training corpus is a common problem for many lan- guage pairs, which might lead to the NMT model’s poor performance. However, constructing a parallel corpus is a slow and laborious process. Professional human trans- lators and well-trained proofreaders are needed. Although several dual learning (He et al., 2016; Bi et al., 2019) and unsupervised learning (Artetxe et al., 2018; Lample et al., 2017; Lample and Con- neau, 2019) approaches have been successfully used, they are often inferior to the supervised mod- els. In such cases, active learning might be a good choice. The goal of active learning in NMT is to train a well-performing model under a limited hu- man translation budget. We achieve this goal by using some particularly designed acquisition func- tions to select informative sentences to construct a training corpus. Acquisition functions can be categorized into two types: model related and model agnostic. For the former, the methods we use are all based on the idea of uncertainty. For the latter, we devise a word frequency based method which takes lin- guistic features into consideration. Both types of acquisition functions have been proven to be bene- ficial in active NMT training, especially when they are appropriately combined. Data augmentation techniques that consume no human translation budget are worth exploring in active NMT training. If the parallel corpus of a related language pair is available, transfer learning (Zoph et al., 2016; Kim et al., 2019) might be a good choice. …

Content truncated to 500 words