Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation

Proceedings of the 28th International Conference on Computational Linguistics, pages 3413–3421 Barcelona, Spain (Online), December 8-13, 2020 3413 Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation Fahimeh Saleh Monash University first.last@monash.edu Wray Buntine Monash University first.last@monash.edu Gholamreza Haffari Monash University first.last@monash.edu Abstract Scarcity of parallel sentence-pairs poses a significant hurdle for training high-quality Neural Ma- chine Translation (NMT) models in bilingually low-resource scenarios. A standard approach is transfer learning, which involves taking a model trained on a high-resource language-pair and fine-tuning it on the data of the low-resource MT condition of interest. However, it is not clear generally which high-resource language-pair offers the best transfer learning for the target MT setting. Furthermore, different transferred models may have complementary semantic and/or syntactic strengths, hence using only one model may be sub-optimal. In this paper, we tackle this problem using knowledge distillation, where we propose to distill the knowledge of ensemble of teacher models to a single student model. As the quality of these teacher models varies, we propose an effective adaptive knowledge distillation approach to dynamically adjust the contri- bution of the teacher models during the distillation process. Experiments on transferring from a collection of six language pairs from IWSLT to five low-resource language-pairs from TED Talks demonstrate the effectiveness of our approach, achieving up to +0.9 BLEU score improvement compared to strong baselines. 1 Introduction Neural models have been revolutionising machine translation (MT), and have achieved state-of-the-art for many high-resource language pairs (Chen et al., 2018; Stahlberg, 2019; Maruf et al., 2019). How- ever, the scarcity of bilingual parallel corpora is still a major challenge for training high-quality NMT models (Koehn and Knowles, 2017). Transfer learning by fine-tuning, from a model trained for a high- resource language-pair, is a standard approach to tackle the scarcity of the data in the target low-resource language-pair (Dabre et al., 2017; Kocmi and Bojar, 2018; Saleh et al., 2019; Kim et al., 2019). How- ever, this is a one-to-one approach, which is not able to exploit models trained for multiple high-resource language-pairs for the target language-pair of interest. Furthermore, models transferred from different high-resource language-pairs may have complementary syntactic and/or semantic strengths, hence using a single model may be sub-optimal. Another appealing approach is multilingual NMT, whereby a single NMT model is trained by com- bining data from multiple high-resource and low-resource language-pairs (Johnson et al., 2017; Ha et al., 2016; Neubig and Hu, 2018). However, the performance of a multilingual NMT model is highly dependent on the types of languages used to train the model. Indeed, if languages are from very distant language families, they lead to negative transfer, causing low translation quality in the multilingual sys- tem compared to the counterparts trained on the individual language-pairs (Tan et al., 2019a; Oncevay et al., 2020). To address this problem, (Tan et al., 2019b) has proposed a knowledge distillation approach to effectively train a multilingual model, by selectively distilling the knowledge from individual teacher models to the multilingual student model. However, …

Content truncated to 500 words