Less is More: Significance-Based N-gram Selection for Smaller, Better Language Models

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 746–755, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Less is More: Significance-Based N-gram Selection for Smaller, Better Language Models Robert C. Moore Chris Quirk Microsoft Research Redmond, WA 98052, USA {bobmoore,chrisq}@microsoft.com Abstract The recent availability of large corpora for training N-gram language models has shown the utility of models of higher or- der than just trigrams. In this paper, we investigate methods to control the increase in model size resulting from applying stan- dard methods at higher orders. We in- troduce significance-based N-gram selec- tion, which not only reduces model size, but also improves perplexity for several smoothing methods, including Katz back- off and absolute discounting. We also show that, when combined with a new smoothing method and a novel variant of weighted-difference pruning, our selection method performs better in the trade-off be- tween model size and perplexity than the best pruning method we found for modi- fied Kneser-Ney smoothing. 1 Introduction Statistical language models are potentially useful for any language technology task that produces natural-language text as a final (or intermediate) output. In particular, they are extensively used in speech recognition and machine translation. De- spite the criticism that they ignore the structure of natural language, simple N-gram models, which estimate the probability of each word in a text string based on the N −1 preceding words, remain the most widely-used type of model. Until the late 1990s, N-gram language models of order higher than trigrams were seldom used. This was due, at least in part, to the fact the amounts of training data available did not produce significantly better results from higher-order mod- els. Since that time, however, increasingly large amounts of language model training data have be- come available ranging from approximately one billion words (the Gigaword corpora from the Linguistic Data Consortium) to trillions of words (Brants et al., 2007). With these larger resources, the use of language models based on 5-grams to 7-grams is becoming increasingly common. The problem we address here is that, even when relatively modest amounts of training data are used, high-order N-gram language models esti- mated by standard techniques can be impractically large. Hence, we investigate ways of building high-order N-gram language models without dra- matically increasing model size. This is, of course, the same goal behind much previous work on lan- guage model pruning, including that of Seymore and Rosenfeld (1996), Stolcke (1998), and Good- man and Gao (2000). We take a novel approach, however, which we refer to as significance-based N-gram selection. We reject a higher-order esti- mate of the probability of a particular word in a particular context whenever the distribution of ob- servations for the higher-order estimate provides no evidence that the higher-order estimate is bet- ter than our backoff estimate. Perhaps our most surprising result is that significance-based N-gram selection not only re- duces language model size, but it also improves perplexity when applied to a number of widely- used smoothing methods, …

Content truncated to 500 words