RNN Architecture Learning with Sparse Regularization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1179–1184, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1179 RNN Architecture Learning with Sparse Regularization Jesse Dodge♣ Roy Schwartz♠♦ Hao Peng♠ Noah A. Smith♠♦ ♣Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA ♦Allen Institute for Artificial Intelligence, Seattle, WA, USA ♠Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA jessed@cs.cmu.edu roys@allenai.org {hapeng,nasmith}@cs.washington.edu Abstract Neural models for NLP typically use large numbers of parameters to reach state-of-the- art performance, which can lead to excessive memory usage and increased runtime. We present a structure learning method for learn- ing sparse, parameter-efficient NLP models. Our method applies group lasso to rational RNNs (Peng et al., 2018), a family of mod- els that is closely connected to weighted finite- state automata (WFSAs). We take advan- tage of rational RNNs’ natural grouping of the weights, so the group lasso penalty directly removes WFSA states, substantially reducing the number of parameters in the model. Our experiments on a number of sentiment anal- ysis datasets, using both GloVe and BERT embeddings, show that our approach learns neural structures which have fewer parame- ters without sacrificing performance relative to parameter-rich baselines. Our method also highlights the interpretable properties of ra- tional RNNs. We show that sparsifying such models makes them easier to visualize, and we present models that rely exclusively on as few as three WFSAs after pruning more than 90% of the weights. We publicly release our code.1 1 Introduction State-of-the-art neural models for NLP are heav- ily parameterized, requiring hundreds of millions (Devlin et al., 2019) and even billions (Rad- ford et al., 2019) of parameters. While over- parameterized models can sometimes be easier to train (Livni et al., 2014), they may also introduce memory problems on small devices and lead to increased carbon emission (Strubell et al., 2019; Schwartz et al., 2019). In feature-based NLP, structured-sparse regular- ization, in particular the group lasso (Yuan and 1https://github.com/dodgejesse/ sparsifying_regularizers_for_RRNNs w(1) 1 w(1) 2 w(1) 3 ˆw(1) 1 ˆw(1) 2 ˆw(1) 3 w(2) 1 w(2) 2 w(2) 3 ˆw(2) 1 ˆw(2) 2 (0) w(3) 1 w(3) 2 w(3) 3 (0) (0) (0) Base structure Learned structure Figure 1: Our approach learns a sparse structure (right hand side) of a base rational RNN (left hand side) where each hidden unit corresponds to a WFSA (in this example, three hidden units, represented by the three rows). Grayed-out, dashed states are removed from the model, while retained states are marked in bold green. Lin, 2006), has been proposed as a method to reduce model size while preserving performance (Martins et al., 2011). But, in neural NLP, some of the most widely used models—LSTMs (Hochre- iter and Schmidhuber, 1997) and GRUs (Cho et al., 2014)—do not have an obvious, intuitive no- tion of “structure” in their parameters (other than, perhaps, division into layers), so the use of struc- …

Content truncated to 500 words