Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3909–3915 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3909 Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture Christopher Brix, Parnia Bahar, Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Abstract Sparse models require less memory for storage and enable a faster inference by reducing the necessary number of FLOPs. This is relevant both for time-critical and on-device computa- tions using neural networks. The stabilized lot- tery ticket hypothesis states that networks can be pruned after none or few training iterations, using a mask computed based on the unpruned converged model. On the transformer archi- tecture and the WMT 2014 English→German and English→French tasks, we show that stabi- lized lottery ticket pruning performs similar to magnitude pruning for sparsity levels of up to 85%, and propose a new combination of prun- ing techniques that outperforms all other tech- niques for even higher levels of sparsity. Fur- thermore, we confirm that the parameter’s ini- tial sign and not its specific value is the pri- mary factor for successful training, and show that magnitude pruning cannot be used to find winning lottery tickets. 1 Introduction Current neural networks are heavily growing in depth, with many fully connected layers. As ev- ery fully connected layer includes large matrices, models often contain millions of parameters. This is commonly seen as an over-parameterization (Dauphin and Bengio, 2013; Denil et al., 2013). Different techniques have been proposed to decide which weights can be pruned. In structured prun- ing techniques (Voita et al., 2019), whole neurons or even complete layers are removed from the net- work. Unstructured pruning only removes individ- ual connections between neurons of succeeding lay- ers, keeping the global network architecture intact. The first technique directly results in smaller model sizes and faster inference, while the second offers more flexibility in the selection of which parame- ters to prune. Although the reduction in necessary storage space can be realized using sparse matrix representations (Stanimirovi and Tasic, 2009), most popular frameworks currently do not have sufficient support for sparse operations. However, there is ac- tive development for possible solutions (Liu et al., 2015; Han et al., 2016; Elsen et al., 2019). This paper compares and improves several unstructured pruning techniques. The main contributions of this paper are to: • verify that the stabilized lottery ticket hypoth- esis (Frankle et al., 2019) performs similar to magnitude pruning (Narang et al., 2017) on the transformer architecture (Vaswani et al., 2017) with 60M parameters up to a sparsity of 85%, while magnitude pruning is superior for higher sparsity levels. • demonstrate significant improvements for high sparsity levels over magnitude pruning by using it in combination with the lottery ticket hypothesis. • confirm that the signs of the initial parameters are more important than the specific values to which they are reset, even for large networks like the transformer. • …

Content truncated to 500 words