On the Sparsity of Neural Machine Translation Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1060–1066, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1060 On the Sparsity of Neural Machine Translation Models Yong Wang∗ The University of Hong Kong wangyong@eee.hku.hk Longyue Wang Tencent AI Lab vinnylywang@tencent.com Victor O.K. Li The University of Hong Kong vli@eee.hku.hk Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Modern neural machine translation (NMT) models employ a large number of parameters, which leads to serious over-parameterization and typically causes the underutilization of computational resources. In response to this problem, we empirically investigate whether the redundant parameters can be reused to achieve better performance. Experiments and analyses are systematically conducted on dif- ferent datasets and NMT architectures. We show that: 1) the pruned parameters can be re- juvenated to improve the baseline model by up to +0.8 BLEU points; 2) the rejuvenated pa- rameters are reallocated to enhance the ability of modeling low-level lexical information. 1 Introduction Modern neural machine translation (NMT) (Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) models employ sufficient capacity to fit the massive data well by utilizing a large number of parameters, and suffer from the widely recog- nized issue, namely, over-parameterization. For example, See et al. (2016) showed that over 40% of the parameters in an RNN-based NMT model can be pruned with negligible performance loss. How- ever, the low utilization efficiency of parameters results in a waste of computational resources (Qiao et al., 2019), as well as renders the model stuck in a local optimum (Han et al., 2017; Yu et al., 2019). In response to the over-parameterization issue, network pruning has been widely investigated for both computer vision (CV) (Han et al., 2016; Luo et al., 2017) and natural language processing (NLP) tasks (See et al., 2016; Lan et al., 2020). Recent work has proven that such spare parameters can be reused to maximize the utilization of models in CV tasks such as image classification (Han et al., 2017; ∗Work was done when interning at Tencent AI Lab. Qiao et al., 2019). The leverage of parameter re- juvenation in sequence-to-sequence learning, how- ever, has received relatively little attention from the research community. In this paper, we empirically study the efficiency issue for NMT models. Specifically, we first investigate the effects of weight pruning on advanced Transformer mod- els, showing that 20% parameters can be directly pruned, and by continuously training the sparse networks, we can prune 50% with no performance loss. Starting from this observation, we then exploit whether these redundant parameters are able to be re-utilized for improving the performance of NMT models. Experiments are systematically conducted on different datasets (i.e. Zh⇒En, De⇒En and En⇒Fr) and NMT architectures (i.e. Transformer, RNNSearch and LightConv). Results demonstrate that the rejuvenation approach can significantly and consistently improve the translation quality by up to +0.8 BLEU points. Further analyses reveal that the rejuvenated parameters are reallocated to en- hance the ability to model the source-side low-level information, lacking of …

Content truncated to 500 words