Improvements in Stochastic Language Modeling

IMPROVEMENTS IN STOCHASTIC LANGUAGE MODELING Ronald Rosenfeld and Xuedong Huang School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 ABSTRACT We describe two attempt to improve our stochastic language models. In the first, we identify a systematic overestimation in the traditional backoff model, and use statisticalreasoning to correct it. Our modifi- cation results in up to 6% reduction in the perplexity of various tasks. Although the improvement is modest, it is achieved with hardly any increasein the complexity of the model. Both analysis and empirical data suggestthat the moditieation is most suitable when training data is sparse. In the second attempt, we propose a new type of adaptive language model. Existing adaptive models use a dynamic eacbe, based on the history of the document seen up to that point. But another source of information in the history, within-document word sequence correlations, has not yet been tapped. We describe a model that attempts to capture this information, using a framework where one word sequence laJggers another, eansing its estimated probability to be raised. We discuss various issues in the design of such a model, and describe our first attempt at building one. Our preliminary results include a perplexity reduction of between 10% and 32%, depending on the test set. 1. INTRODUCTION Linguistic constraints are an important factor in human com- prehension of speech. Their effect on automatic speech recog- nition is similar, in that they provide both a pruning method and a means of ordering likely candidates. As vocabularies for speech recognition systems increase in size, more accurate modeling of linguistic constraints becomes essential. Two fundamental issues in language modeling are smooth- ing and adaptation. Smoothing allows a model to assign reasonable probabilities to events that have never been ob- served before. Adaptation takes advantage of recently gained knowledge -- the text seen so far -- to adjust the model's expectations. In what follows, we discuss two attempts at improving our current stochastic language modeling techniques. In the first, we try to improve smoothing by correcting a deficiency in a successful and well known smoothing method, the backoff model. In the second, we propose a novel kind of adapta- tion, one that is based on correlation among word sequences occurring in the same document. 2. CORRECTING OVERESTIMATION IN THE BACKOFF MODEL 2.1. The Problem The backoff n-gram language model[l] estimates the prob- ability of w,, given the immediate past history w~ n--1 = (wl .... w~-0. It is defined recursively as: Pn(w"lw~-t) = / (1 - d)C(w~) / C(~1-1) if C(w~) > 0 o~(C(w~-l)) • en_l(wnlw~ -1) if C(w~) = 0 k (1) where d, the discount ratio, is a function of C(w~), and the a's are the backoff weights, calculated to satisfy the sum-to-1 probability constraints. The backoff language model is a compact yet powerful way of modeling the dependence of the current word on its immediate history. An important factor in the backoff model is its be- havior on the backed-off cases, namely when a given n-gram w~ …

Content truncated to 500 words