Sequence-Level Knowledge Distillation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Sequence-Level Knowledge Distillation Yoon Kim yoonkim@seas.harvard.edu Alexander M. Rush srush@seas.harvard.edu School of Engineering and Applied Sciences Harvard University Cambridge, MA, USA Abstract Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical ap- proaches. However to reach competitive per- formance, NMT models need to be exceed- ingly large. In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006; Hinton et al., 2015) that have proven successful for reducing the size of neural mod- els in other domains to the problem of NMT. We demonstrate that standard knowledge dis- tillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to elimi- nate the need for beam search (even when ap- plied on the original teacher model). Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in per- formance. It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy de- coding/beam search. Applying weight prun- ing on top of knowledge distillation results in a student model that has 13× fewer param- eters than the original teacher model, with a decrease of 0.4 BLEU. 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015) is a deep learning- based method for translation that has recently shown promising results as an alternative to statistical ap- proaches. NMT systems directly model the proba- bility of the next word in the target sentence sim- ply by conditioning a recurrent neural network on the source sentence and previously generated target words. While both simple and surprisingly accurate, NMT systems typically need to have very high ca- pacity in order to perform well: Sutskever et al. (2014) used a 4-layer LSTM with 1000 hidden units per layer (herein 4×1000) and Zhou et al. (2016) ob- tained state-of-the-art results on English → French with a 16-layer LSTM with 512 units per layer. The sheer size of the models requires cutting-edge hard- ware for training and makes using the models on standard setups very challenging. This issue of excessively large networks has been observed in several other domains, with much fo- cus on fully-connected and convolutional networks for multi-class classification. Researchers have par- ticularly noted that large networks seem to be nec- essary for training, but learn redundant representa- tions in the process (Denil et al., 2013). Therefore compressing deep models into smaller networks has been an active area of research. As deep learning systems obtain better results on NLP tasks, compres- sion also becomes an important practical issue with applications such as running deep learning models for speech and translation locally on cell phones. Existing compression methods generally fall into two …

Content truncated to 500 words