Compression of Neural Machine Translation Models via Pruning

Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 291–301, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Compression of Neural Machine Translation Models via Pruning Abigail See∗ Minh-Thang Luong∗ Christopher D. Manning Computer Science Department, Stanford University, Stanford, CA 94305 {abisee,lmthang,manning}@stanford.edu Abstract Neural Machine Translation (NMT), like many other deep learning domains, typ- ically suffers from over-parameterization, resulting in large storage sizes. This paper examines three simple magnitude-based pruning schemes to compress NMT mod- els, namely class-blind, class-uniform, and class-distribution, which differ in terms of how pruning thresholds are com- puted for the different classes of weights in the NMT architecture. We demonstrate the efficacy of weight pruning as a compres- sion technique for a state-of-the-art NMT system. We show that an NMT model with over 200 million parameters can be pruned by 40% with very little performance loss as measured on the WMT’14 English- German translation task. This sheds light on the distribution of redundancy in the NMT architecture. Our main result is that with retraining, we can recover and even surpass the original performance with an 80%-pruned model. 1 Introduction Neural Machine Translation (NMT) is a simple new architecture for translating texts from one lan- guage into another (Sutskever et al., 2014; Cho et al., 2014). NMT is a single deep neural network that is trained end-to-end, holding several advan- tages such as the ability to capture long-range de- pendencies in sentences, and generalization to un- seen texts. Despite being relatively new, NMT has already achieved state-of-the-art translation re- sults for several language pairs including English- French (Luong et al., 2015b), English-German (Jean et al., 2015a; Luong et al., 2015a; Luong and ∗Both authors contributed equally. student a am I Je Je suis suis étudiant étudiant _ _ source language input target language input target language output Figure 1: A simplified diagram of NMT. Manning, 2015; Sennrich et al., 2016), English- Turkish (Sennrich et al., 2016), and English-Czech (Jean et al., 2015b; Luong and Manning, 2016). Figure 1 gives an example of an NMT system. While NMT has a significantly smaller memory footprint than traditional phrase-based approaches (which need to store gigantic phrase-tables and language models), the model size of NMT is still prohibitively large for mobile devices. For exam- ple, a recent state-of-the-art NMT system requires over 200 million parameters, resulting in a stor- age size of hundreds of megabytes (Luong et al., 2015a). Though the trend for bigger and deeper neural networks has brought great progress, it has also introduced over-parameterization, resulting in long running times, overfitting, and the storage size issue discussed above. A solution to the over- parameterization problem could potentially aid all three issues, though the first (long running times) is outside the scope of this paper. Our contribution. In this paper we investi- gate the efficacy of weight pruning for NMT as a means of compression. We show that despite 291 its simplicity, magnitude-based pruning with re- training is highly effective, and we compare three …

Content truncated to 500 words