Long-Short Range Context Neural Networks for Language Modeling

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1473–1481, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Long-Short Range Context Neural Networks for Language Modeling Youssef Oualil1,2 and Mittul Singh1,3 and Clayton Greenberg1,2,3 and Dietrich Klakow1,2,3 1Spoken Language Systems (LSV) 2Collaborative Research Center on Information Density and Linguistic Encoding 3Graduate School of Computer Science Saarland University, Saarbr¨ucken, Germany {firstname.lastname}@lsv.uni-saarland.de Abstract The goal of language modeling techniques is to capture the statistical and structural prop- erties of natural languages from training cor- pora. This task typically involves the learning of short range dependencies, which generally model the syntactic properties of a language and/or long range dependencies, which are se- mantic in nature. We propose in this paper a new multi-span architecture, which separately models the short and long context informa- tion while it dynamically merges them to per- form the language modeling task. This is done through a novel recurrent Long-Short Range Context (LSRC) network, which explicitly models the local (short) and global (long) context using two separate hidden states that evolve in time. This new architecture is an adaptation of the Long-Short Term Memory network (LSTM) to take into account the lin- guistic properties. Extensive experiments con- ducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art language modeling techniques. 1 Introduction A high quality Language Model (LM) is considered to be an integral component of many systems for speech and language technology applications, such as machine translation (Brown et al., 1990), speech recognition (Katz, 1987), etc. The goal of an LM is to identify and predict probable sequences of pre- defined linguistic units, which are typically words. These predictions are typically guided by the seman- tic and syntactic properties encoded by the LM. In order to capture these properties, classical LMs were typically developed as fixed (short) context techniques such as, the word count-based meth- ods (Rosenfeld, 2000; Kneser and Ney, 1995), com- monly known as N-gram language models, as well as the Feedforward Neural Networks (FFNN) (Ben- gio et al., 2003), which were introduced as an al- ternative to overcome the exponential growth of pa- rameters required for larger context sizes in N-gram models. In order to overcome the short context constraint and capture long range dependencies known to be present in language, Bellegarda (1998a) proposed to use Latent Semantic Analysis (LSA) to capture the global context, and then combine it with the standard N-gram models, which capture the local context. In a similar but more recent approach, Mikolov and Zweig (2012) showed that Recurrent Neural Net- work (RNN)-based LM performance can be signif- icantly improved using an additional global topic information obtained using Latent Dirichlet Allo- cation (LDA). In fact, although recurrent architec- tures theoretically allow the context to indefinitely cycle in the network, Hai Son et al. (2012) have shown that, in practice, this information changes quickly in the classical RNN (Mikolov et al., …

Content truncated to 500 words