Compressing Neural Language Models by Sparse Word Representations

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 226–235, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Compressing Neural Language Models by Sparse Word Representations Yunchuan Chen,1,2 Lili Mou,1,3 Yan Xu,1,3 Ge Li,1,3 Zhi Jin1,3,∗ 1Key Laboratory of High Confidence Software Technologies (Peking University), MoE, China 2University of Chinese Academy of Sciences, chenyunchuan11@mails.ucas.ac.cn 3Institute of Software, Peking University, doublepower.mou@gmail.com, {xuyan14,lige,zhijin}@pku.edu.cn ∗Corresponding author Abstract Neural networks are among the state-of- the-art techniques for language modeling. Existing neural language models typically map discrete words to distributed, dense vector representations. After information processing of the preceding context words by hidden layers, an output layer estimates the probability of the next word. Such ap- proaches are time- and memory-intensive because of the large numbers of parame- ters for word embeddings and the output layer. In this paper, we propose to com- press neural language models by sparse word representations. In the experiments, the number of parameters in our model in- creases very slowly with the growth of the vocabulary size, which is almost imper- ceptible. Moreover, our approach not only reduces the parameter space to a large ex- tent, but also improves the performance in terms of the perplexity measure.1 1 Introduction Language models (LMs) play an important role in a variety of applications in natural language processing (NLP), including speech recognition and document recognition. In recent years, neu- ral network-based LMs have achieved signifi- cant breakthroughs: they can model language more precisely than traditional n-gram statistics (Mikolov et al., 2011); it is even possible to gen- erate new sentences from a neural LM, benefit- ing various downstream tasks like machine trans- lation, summarization, and dialogue systems (De- vlin et al., 2014; Rush et al., 2015; Sordoni et al., 2015; Mou et al., 2015b). 1Code released on https://github.com/chenych11/lm Existing neural LMs typically map a discrete word to a distributed, real-valued vector repre- sentation (called embedding) and use a neural model to predict the probability of each word in a sentence. Such approaches necessitate a large number of parameters to represent the em- beddings and the output layer’s weights, which is unfavorable in many scenarios. First, with a wider application of neural networks in resource- restricted systems (Hinton et al., 2015), such ap- proach is too memory-consuming and may fail to be deployed in mobile phones or embedded sys- tems. Second, as each word is assigned with a dense vector—which is tuned by gradient-based methods—neural LMs are unlikely to learn mean- ingful representations for infrequent words. The reason is that infrequent words’ gradient is only occasionally computed during training; thus their vector representations can hardly been tuned ade- quately. In this paper, we propose a compressed neural language model where we can reduce the number of parameters to a large extent. To accomplish this, we first represent infrequent words’ embeddings with frequent words’ by sparse linear combina- tions. This is inspired by the observation that, in a dictionary, an unfamiliar word is typically defined by common words. We …

Content truncated to 500 words