Language Models Based on Semantic Composition

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 430–439, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Language Models Based on Semantic Composition Jeff Mitchell and Mirella Lapata School of Informatics, University of Edinburgh Edinburgh EH8 9LW, UK jeff.mitchell@ed.ac.uk, mlap@inf.ed.ac.uk Abstract In this paper we propose a novel statistical language model to capture long-range se- mantic dependencies. Specifically, we ap- ply the concept of semantic composition to the problem of constructing predictive his- tory representations for upcoming words. We also examine the influence of the un- derlying semantic space on the composi- tion task by comparing spatial semantic representations against topic-based ones. The composition models yield reductions in perplexity when combined with a stan- dard n-gram language model over the n-gram model alone. We also obtain per- plexity reductions when integrating our models with a structured language model. 1 Introduction Statistical language modeling plays an important role in many areas of natural language process- ing including speech recognition, machine trans- lation, and information retrieval. The prototypi- cal use of language models is to assign proba- bilities to sequences of words. By invoking the chain rule, these probabilities are generally es- timated as the product of conditional probabili- ties P(wi|hi) of a word wi given the history of preceding words hi ≡ wi−1 1 . In theory, the history could span any number of words up to wi such as sentences or even a paragraphs. In practice, how- ever, it has proven challenging to deal with the combinatorial growth in the number of possible histories which in turn impacts reliable parame- ter estimation. A simple and effective strategy is to truncate the chain rule to include only the n-1 preceding words (n is often set within the range of 3–5). The simplification reduces the number of free parameters. However, low values of n impose an artificially local horizon to the language model, and compromise its ability to capture long-range dependencies, such as syntactic relationships, se- mantic or thematic constraints. The literature offers many examples of how to overcome this limitation, essentially by allowing the modulation of probabilities by dependencies which extend to words beyond the n-gram horizon. Cache language models (Kuhn and de Mori, 1992) increase the probability of words observed in the history, e.g., by some factor which decays expo- nentially with distance. Trigger models (Rosen- feld, 1996) go a step further by allowing arbi- trary word pairs to be incorporated into the cache. Structured language models (e.g., Roark (2001)) go beyond the representation of history as a lin- ear sequence of words to capture the syntactic con- structions in which these words are embedded. It is also possible to build representations of history which are semantic rather than syntactic (Bellegarda (2000; Coccaro and Jurafsky (1998; Gildea and Hofmann (1999)). In this approach, es- timates for the probabilities of upcoming words are derived from a comparison of their semantic content with the content of the history so far. The semantic representations, in this case, are …

Content truncated to 500 words