Auto-Sizing Neural Networks: With Applications to n-gram Language Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 908–916, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Auto-Sizing Neural Networks: With Applications to n-gram Language Models Kenton Murray and David Chiang Department of Computer Science and Engineering University of Notre Dame {kmurray4,dchiang}@nd.edu Abstract Neural networks have been shown to improve performance across a range of natural-language tasks. However, design- ing and training them can be complicated. Frequently, researchers resort to repeated experimentation to pick optimal settings. In this paper, we address the issue of choosing the correct number of units in hidden layers. We introduce a method for automatically adjusting network size by pruning out hidden units through ℓ∞,1 and ℓ2,1 regularization. We apply this method to language modeling and demonstrate its ability to correctly choose the number of hidden units while maintaining perplexity. We also include these models in a machine translation decoder and show that these smaller neural models maintain the signif- icant improvements of their unpruned ver- sions. 1 Introduction Neural networks have proven to be highly ef- fective at many tasks in natural language. For example, neural language models and joint lan- guage/translation models improve machine trans- lation quality significantly (Vaswani et al., 2013; Devlin et al., 2014). However, neural networks can be complicated to design and train well. Many de- cisions need to be made, and performance can be highly dependent on making them correctly. Yet the optimal settings are non-obvious and can be laborious to find, often requiring an extensive grid search involving numerous experiments. In this paper, we focus on the choice of the sizes of hidden layers. We introduce a method for automatically pruning out hidden layer units, by adding a sparsity-inducing regularizer that en- courages units to deactivate if not needed, so that they can be removed from the network. Thus, af- ter training with more units than necessary, a net- work is produced that has hidden layers correctly sized, saving both time and memory when actually putting the network to use. Using a neural n-gram language model (Bengio et al., 2003), we are able to show that our novel auto-sizing method is able to learn models that are smaller than models trained without the method, while maintaining nearly the same perplexity. The method has only a single hyperparameter to adjust (as opposed to adjusting the sizes of each of the hidden layers), and we find that the same setting works consistently well across different training data sizes, vocabulary sizes, and n-gram sizes. In addition, we show that incorporating these mod- els into a machine translation decoder still results in large BLEU point improvements. The result is that fewer experiments are needed to obtain mod- els that perform well and are correctly sized. 2 Background Language models are often used in natural lan- guage processing tasks involving generation of text. For instance, in machine translation, the lan- guage model helps to output fluent translations, and in speech recognition, the language model helps to disambiguate …

Content truncated to 500 words