Neural Lattice Language Models

Neural Lattice Language Models Jacob Buckman Language Technologies Institute Carnegie Mellon University jacobbuckman@gmail.com Graham Neubig Language Technologies Institute Carnegie Mellon University gneubig@cs.cmu.edu Abstract In this work, we propose a new language mod- eling paradigm that has the ability to perform both prediction and moderation of informa- tion flow at multiple granularities: neural lat- tice language models. These models con- struct a lattice of possible paths through a sen- tence and marginalize across this lattice to cal- culate sequence probabilities or optimize pa- rameters. This approach allows us to seam- lessly incorporate linguistic intuitions – in- cluding polysemy and the existence of multi- word lexical items – into our language model. Experiments on multiple language modeling tasks show that English neural lattice language models that utilize polysemous embeddings are able to improve perplexity by 9.95% rela- tive to a word-level baseline, and that a Chi- nese model that handles multi-character to- kens is able to improve perplexity by 20.94% relative to a character-level baseline. 1 Introduction Neural network models have recently contributed to- wards a great amount of progress in natural language processing. These models typically share a common backbone: recurrent neural networks (RNN), which have proven themselves to be capable of tackling a variety of core natural language processing tasks (Hochreiter and Schmidhuber, 1997; Elman, 1990). One such task is language modeling, in which we estimate a probability distribution over sequences of tokens that corresponds to observed sentences (§2). Neural language models, particularly models con- ditioned on a particular input, have many applica- tions including in machine translation (Bahdanau et al., 2016), abstractive summarization (Chopra et al., 2016), and speech processing (Graves et al., 2013). dogs chased the small cat dogs chased the small cat dogs chased the small dogs chased the the_small the_small_cat small_cat dogs_chased chased chased_the dogs_chased_the chased_the_small Figure 1: Lattice decomposition of a sentence and its cor- responding lattice language model probability calculation Similarly, state-of-the-art language models are al- most universally based on RNNs, particularly long short-term memory (LSTM) networks (Jozefowicz et al., 2016; Inan et al., 2017; Merity et al., 2016). While powerful, LSTM language models usually do not explicitly model many commonly-accepted linguistic phenomena. As a result, standard mod- els lack linguistically informed inductive biases, po- tentially limiting their accuracy, particularly in low- data scenarios (Adams et al., 2017; Koehn and Knowles, 2017). In this work, we present a novel modification to the standard LSTM language mod- eling framework that allows us to incorporate some varieties of these linguistic intuitions seamlessly: neural lattice language models (§3.1). Neural lat- tice language models define a lattice over possi- ble paths through a sentence, and maximize the marginal probability over all paths that lead to gen- erating the reference sentence, as shown in Fig. 1. Depending on how we define these paths, we can in- corporate different assumptions about how language should be modeled. In the particular instantiations of neural lattice language models covered by this paper, we focus on two properties of language that could potentially be …

Content truncated to 500 words