Analyzing Knowledge Distillation in Neural Machine Translation

Analyzing Knowledge Distillation in Neural Machine Translation Dakun Zhang, Josep Crego, Jean Senellart SYSTRAN / 5 rue Feydeau, 75002 Paris, France firstname.lastname@systrangroup.com Abstract Knowledge distillation has recently been successfully ap- plied to neural machine translation. It allows for building shrunk networks while the resulting systems retain most of the quality of the original model. Despite the fact that many authors report on the benefits of knowledge distillation, few have discussed the actual reasons why it works, especially in the context of neural MT. In this paper, we conduct sev- eral experiments aimed at understanding why and how dis- tillation impacts accuracy on an English-German translation task. We show that translation complexity is actually reduced when building a distilled/synthesised bi-text when compared to the reference bi-text. We further remove noisy data from synthesised translations and merge filtered synthesised data together with original reference, thus achieving additional gains in terms of accuracy. 1. Introduction Neural machine translation (NMT) achieves state-of-the-art results in several translation tasks and for multiple language pairs [1, 2]. Equivalent to its phrase-based predecessor, neu- ral networks directly learn from parallel bi-texts, consisting of large amounts of human created sentences with their cor- responding translations. Therefore, the quality of an MT en- gine is heavily dependent on the amount and quality of par- allel sentences. Several techniques aimed at boosting the quality [3, 4] and quantity [5] of training data are successfully applied to neural MT. Parallel to these techniques, knowledge distilla- tion [6] has attracted the focus of many researchers given its simplicity and the quality of the results. However, despite the fact that a growing number of private entities have begun to include distillation into their NMT systems [7, 8, 9] and that knowledge distillation has demonstrated its performance for multiple tasks [10, 11, 12], none of them give a detailed analysis of the reasons why it works. In most cases, the availability of parallel corpora is a pre- requisite to build Neural MT systems. The process of com- piling parallel bi-texts is usually composed of several steps: crawling, filtering, cleaning, etc. As a result, parallel corpora usually contain parallel sentences that are often not as par- allel as one might assume. And even for parallel sentences that truly convey the same meaning, in some cases transla- tions follow a more or less word-for-word pattern (more lit- eral translations). While in many other cases, translations show greater latitude of expression (more flexible transla- tions) with higher degrees of variability, which humans often judge as good. However, machine translations are usually “closer” in terms of syntactic structure and present lower lev- els of variability when considering word choice. It is rather an intuitive idea that feeding more “literal” translations to a neural MT network should facilitate the training process compared to training with less literal translations (original bi-text). In this paper, we report on the results of experiments where we automatically distill a human translation bi-text which is then used to train neural translation engines. Thus, aiming at …

Content truncated to 500 words