Combining Sequence Distillation and Transfer Learning for Efficient Low-Resource Neural Machine Translation Models

Proceedings of the 5th Conference on Machine Translation (WMT), pages 492–502 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 492 Combining Sequence Distillation and Transfer Learning for Efficient Low-Resource Neural Machine Translation Models Raj Dabre Atsushi Fujita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan firstname.lastname@nict.go.jp Abstract In neural machine translation (NMT), se- quence distillation (SD) through creation of distilled corpora leads to efficient (compact and fast) models. However, its effectiveness in extremely low-resource (ELR) settings has not been well-studied. On the other hand, trans- fer learning (TL) by leveraging larger helping corpora greatly improves translation quality in general. This paper investigates a combina- tion of SD and TL for training efficient NMT models for ELR settings, where we utilize TL with helping corpora twice: once for distill- ing the ELR corpora and then during compact model training. We experimented with two ELR settings: Vietnamese–English and Hindi– English from the Asian Language Treebank dataset with 18k training sentence pairs. Us- ing the compact models with 40% smaller pa- rameters trained on the distilled ELR corpora, greedy search achieved 3.6 BLEU points im- provement in average while reducing 40% of decoding time. We also confirmed that using both the distilled ELR and helping corpora in the second round of TL further improves trans- lation quality. Our work highlights the impor- tance of stage-wise application of SD and TL for efficient NMT modeling for ELR settings. 1 Introduction Neural machine translation (NMT) (Bahdanau et al., 2015; Sutskever et al., 2014) enables end- to-end training of translation models and is known to give state-of-the-art results for a large variety of language pairs. NMT models with large hid- den sizes or deep stacked layers tend to give better translations than those with small hidden sizes or fewer layers. Large models inevitably need more storage space and computation, and are difficult to deploy on low-computation and low-memory devices. Additionally, beam search decoding is known to improve translation quality but needs more computation and is unacceptable in a low- latency real-time application where faster decoding is as valuable as if not more valuable than trans- lation quality. Consequently, neural models that are compact and fast are extremely important and a growing body of research known as neural model efficiency focuses on this issue. One of the most popular techniques to train ef- ficient models is knowledge distillation (Hinton et al., 2015) which relies on transferring the knowl- edge learned by a large model (called teacher) into a smaller model (called student). Sequence distilla- tion (SD) (Kim and Rush, 2016) is a special case of knowledge distillation for sequence-to-sequence models, such as those used for NMT. Not only does it help in the training of compact and fast models with high translation quality, it sometimes helps in eliminating the need for beam search which further increases decoding speed. SD relies on the creation of distilled parallel corpora by translating the train- ing source sentences into the target language by using …

Content truncated to 500 words