Towards Compact and Fast Neural Machine Translation Using a Combined Method

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1475–1481 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Towards Compact and Fast Neural Machine Translation Using a Combined Method Xiaowei Zhang1,2, Wei Chen1∗, Feng Wang1,2, Shuang Xu1 and Bo Xu1 1Institute of Automation, Chinese Academy of Sciences, Beijing, China 2University of Chinese Academy of Sciences, China {zhangxiaowei2015,w.chen,feng.wang,shuang.xu,boxu}@ia.ac.cn Abstract Neural Machine Translation (NMT) lays intensive burden on computation and memory cost. It is a challenge to de- ploy NMT models on the devices with limited computation and memory budgets. This paper presents a four stage pipeline to compress model and speed up the decod- ing for NMT. Our method first introduces a compact architecture based on convo- lutional encoder and weight shared em- beddings. Then weight pruning is ap- plied to obtain a sparse model. Next, we propose a fast sequence interpolation ap- proach which enables the greedy decod- ing to achieve performance on par with the beam search. Hence, the time-consuming beam search can be replaced by simple greedy decoding. Finally, vocabulary se- lection is used to reduce the computation of softmax layer. Our final model achieves 10× speedup, 17× parameters reduction, <35MB storage size and comparable per- formance compared to the baseline model. 1 Introduction Neural Machine Translation (NMT) (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015) has recently gained popu- larity in solving the machine translation problem. Although NMT has achieved state-of-the-art per- formance for several language pairs (Jean et al., 2015; Wu et al., 2016), like many other deep learn- ing domains, it is both computationally intensive and memory intensive. This leads to a challenge of deploying NMT models on the devices with lim- ited computation and memory budgets. ∗*Corresponding author Numerous approaches have been proposed for compression and inference speedup of neural net- works, including but not limited to low-rank ap- proximation (Denton et al., 2014), hash function (Chen et al., 2015), knowledge distillation (Hin- ton et al., 2015), quantization (Courbariaux et al., 2015; Han et al., 2016; Zhou et al., 2017) and spar- sification (Han et al., 2015; Wen et al., 2016). Weight pruning and knowledge distillation have been proved to be able to compress NMT mod- els (See et al., 2016; Kim and Rush, 2016; Fre- itag et al., 2017). The above methods reduce the parameters from a global perspective. However, embeddings dominate the parameters in a rela- tively compact NMT model even if subword (Sen- nrich et al., 2016) (typical about 30K) is used. Character-aware methods (Ling et al., 2015; Lee et al., 2016) have fewer embeddings while suffer from slower decoding speed (Wu et al., 2016). Re- cent work by Li et al. (2016) has shown that weight sharing can be adopted to compress embeddings in language model. We are interested in applying embeddings weight sharing to NMT. As for decoding speedup, Gehring et al. (2016); Kalchbrenner et al. (2016) tried to improve the parallelism in NMT by substituting CNNs for …

Content truncated to 500 words