An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models

An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models Varun Gumma1, Raj Dabre2, Pratyush Kumar3 Indian Institute of Technology, Madras1,3 Microsoft3 AI4Bharat1,2,3 National Institute of Information and Communications Technology2 1varun230999@gmail.com 2raj.dabre@nict.go.jp 3pratykumar@microsft.com Abstract Knowledge distillation (KD) is a well- known method for compressing neural models. However, works focusing on dis- tilling knowledge from large multilingual neural machine translation (MNMT) mod- els into smaller ones are practically nonex- istent, despite the popularity and superior- ity of MNMT. This paper bridges this gap by presenting an empirical investigation of knowledge distillation for compressing MNMT models. We take Indic to English translation as a case study and demonstrate that commonly used language-agnostic and language-aware KD approaches yield models that are 4-5× smaller but also suf- fer from performance drops of up to 3.5 BLEU. To mitigate this, we then experi- ment with design considerations such as shallower versus deeper models, heavy pa- rameter sharing, multi-stage training, and adapters. We observe that deeper compact models tend to be as good as shallower non-compact ones, and that fine-tuning a distilled model on a High-Quality subset slightly boosts translation quality. Over- all, we conclude that compressing MNMT models via KD is challenging, indicating immense scope for further research. 1 Introduction Neural Machine Translation (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017) is a state-of-the- art approach to machine translation that has gained © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. Figure 1: A comparison of the major distillation techniques and models we experimented with. Note that the red incre- ments in the bar plots denote the improvements due to HQ fine-tuning for those models. significant attention in recent years. With the avail- ability of large corpora and compute, Multilingual NMT (MNMT) (Zhang et al., 2019; Firat et al., 2016; Aharoni et al., 2019) has gained popularity since it enables a single model to translate between multiple languages. Large MNMT models trained on substantial data have shown higher levels of performance. However, these models are impracti- cal for deployment on a commercial or production scale due to their size, which contains millions, if not billions, of parameters. Therefore, they need to be compressed into smaller models for efficient and convenient usage. In practice, models are compressed via two methods: Firstly, by stripping unnecessary and re- dundant parameters from the existing model (Bu- ciluˇa et al., 2006), and secondly, by transferring knowledge from the larger “teacher” model to a smaller “student” model using distillation (Hin- ton et al., 2015). This study focuses on the lat- ter, as the former can be done post-hoc (Diddee et al., 2022). Although existing literature mainly discusses bilingual-to-multilingual or bilingual-to- bilingual distillation, to the best of our knowledge, there is no work in end-to-end multilingual-to- multilingual knowledge distillation for compres- sion in a setting with a mix of low, medium, and high resource languages. Therefore, we aim to distill a large MNMT model into a …

Content truncated to 500 words