Predefined Sparseness in Recurrent Sequence Models

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 324–333 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 324 Predefined Sparseness in Recurrent Sequence Models Thomas Demeester, Johannes Deleu, Fr´ederic Godin, Chris Develder Ghent University - imec Ghent, Belgium firstname.lastname@ugent.be Abstract Inducing sparseness while training neural net- works has been shown to yield models with a lower memory footprint but similar effective- ness to dense models. However, sparseness is typically induced starting from a dense model, and thus this advantage does not hold during training. We propose techniques to enforce sparseness upfront in recurrent sequence mod- els for NLP applications, to also benefit train- ing. First, in language modeling, we show how to increase hidden state sizes in recurrent lay- ers without increasing the number of parame- ters, leading to more expressive models. Sec- ond, for sequence labeling, we show that word embeddings with predefined sparseness lead to similar performance as dense embeddings, at a fraction of the number of trainable parameters. 1 Introduction Many supervised learning problems today are solved with deep neural networks exploiting large- scale labeled data. The computational and mem- ory demands associated with the large amount of parameters of deep models can be alleviated by us- ing sparse models. Applying sparseness can be seen as a form of regularization, as it leads to a reduced amount of model parameters1, for given layer widths or representation sizes. Current suc- cessful approaches gradually induce sparseness during training, starting from densely initialized networks, as detailed in Section 2. However, we propose that models can also be built with pre- defined sparseness, i.e., such models are already sparse by design and do not require sparseness in- ducing training schemes. The main benefit of such an approach is mem- ory efficiency, even at the start of training. Espe- cially in the area of natural language processing, in 1The sparseness focused on in this work, occurs on the level of trainable parameters, i.e., we do not consider data sparsity. line with the hypothesis by Yang et al. (2017) that natural language is “high-rank”, it may be useful to train larger sparse representations, even when facing memory restrictions. For example, in order to train word representations for a large vocabu- lary using limited computational resources, prede- fined sparseness would allow training larger em- beddings more effectively compared to strategies inducing sparseness from dense models. The contributions of this paper are (i) a predefined sparseness model for recurrent neu- ral networks, (ii) as well as for word embed- dings, and (iii) proof-of-concept experiments on part-of-speech tagging and language modeling, in- cluding an analysis of the memorization capacity of dense vs. sparse networks. An overview of re- lated work is given in the next Section 2. We sub- sequently present predefined sparseness in recur- rent layers (Section 3), as well as embedding lay- ers (Section 4), each illustrated by experimental results. This is followed by an empirical investi- gation of the memorization capacity of language …

Content truncated to 500 words