Binarized LSTM Language Model

Proceedings of NAACL-HLT 2018, pages 2113–2121 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Binarized LSTM Language Model Xuan Liu∗, Di Cao∗, Kai Yu Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, SpeechLab, Department of Computer Science and Engineering, Brain Science and Technology Research Center, Shanghai Jiao Tong University, Shanghai, China {liuxuan0526, caodi0207, ky219.cam}@gmail.com Abstract The long short-term memory (LSTM) lan- guage model (LM) has been widely investi- gated for automatic speech recognition (ASR) and natural language processing (NLP). Al- though excellent performance is obtained for large vocabulary tasks, tremendous memory consumption prohibits the use of LSTM LMs in low-resource devices. The memory con- sumption mainly comes from the word em- bedding layer. In this paper, a novel binarized LSTM LM is proposed to address the problem. Words are encoded into binary vectors and other LSTM parameters are further binarized to achieve high memory compression. This is the first effort to investigate binary LSTMs for large vocabulary language modeling. Ex- periments on both English and Chinese LM and ASR tasks showed that binarization can achieve a compression ratio of 11.3 without any loss of LM and ASR performance and a compression ratio of 31.6 with acceptable mi- nor performance degradation. 1 Introduction Language models (LMs) play an important role in natural language processing (NLP) tasks. N- gram language models used to be the most pop- ular language models. Considering the previous N-1 words, N-gram language models predict the next word. However, this leads to the loss of long- term dependencies. The sample space size in- creases exponentially as N grows, which induces data sparseness (Cao and Yu, 2017). Neural network (NN) based models were first introduced into language modeling in 2003 (Ben- gio et al., 2003). Given contexts with a fixed size, the model can calculate the probability dis- tribution of the next word. However, the prob- lem of long-term dependencies still remained, be- ∗Both authors contributed equally to this work. cause the context window is fixed. Currently, re- current neural network (RNN) based models are widely used on natural language processing (NLP) tasks for excellent performance (Mikolov et al., 2010). Recurrent structures in neural networks can solve the problem of long-term dependencies to a great extent. Some gate based structures, such as long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and gated recurrent unit (GRU) (Chung et al., 2014) improve the recur- rent structures and achieve state-of-the-art perfor- mance on most NLP tasks. However, neural network models occupy tremendous memory space so that it is almost im- possible to put the models into low-resource de- vices. In practice, the vocabulary is usually very large. So the memory consumption mainly comes from the embedding layers. And, the word embed- ding parameters are floating point values, which adds to the memory consumption. The first contribution in this paper is that a novel language model, the binarized embedding language model (BELM) is proposed to reduce the memory consumption. Words are represented in the …

Content truncated to 500 words