Improving Language Models by Clustering Training Sentences

Improving Language Models by Clustering Training Sentences David Carter SRI International 23 Millers Yard Cambridge CB2 1RQ, UK dmc@cam, sri. com Abstract Many of the kinds of language model used in speech understanding suffer from imper- fect modeling of intra-sentential contextual influences. I argue that this problem can be addressed by clustering the sentences in a training corpus automatically into subcor- pora on the criterion of entropy reduction, and calculating separate language model parameters for each cluster. This kind of clustering offers a way to represent impor- tant contextual effects and can therefore significantly improve the performance of a model. It also offers a reasonably auto- matic means to gather evidence on whether a more complex, context-sensitive model using the same general kind of linguistic in- formation is likely to reward the effort that would be required to develop it: if cluster- ing improves the performance of a model, this proves the existence of further context dependencies, not exploited by the unclus- tered model. As evidence for these claims, I present results showing that clustering im- proves some models but not others for the ATIS domain. These results are consistent with other findings for such models, sug- gesting that the existence or otherwise of an improvement brought about by cluster- ing is indeed a good pointer to whether it is worth developing further the unclustered model. 1 Introduction In speech recognition and understanding systems, many kinds of language model may be used to choose between the word and sentence hypotheses for which there is evidence in the acoustic data. Some words, word sequences, syntactic constructions and seman- tic structures are more likely to occur than others, and the presence of more likely objects in a sen- tence hypothesis is evidence for the correctness of that hypothesis. Evidence from different knowledge sources can be combined in an attempt to optimize the selection of correct hypotheses; see e.g. Alshawi and Carter (1994); Rayner et al (1994); Rosenfeld (1994). Many of the knowledge sources used for this pur- pose score a sentence hypothesis by calculating a simple, typically linear, combination of scores asso- ciated with objects, such as N-grams and grammar rules, that characterize the hypothesis or its pre- ferred linguistic analysis. When these scores are viewed as log probabilities, taking a linear sum corre- sponds to making an independence assumption that is known to be at best only approximately true, and that may give rise to inaccuracies that reduce the effectiveness of the knowledge source. The most obvious way to make a knowledge source more accurate is to increase the amount of structure or context that it takes account of. For example, a bigram model may be replaced by a trigram one, and the fact that dependencies exist among the like- lihoods of occurrence of grammar rules at different locations in a parse tree can be modeled by associat- ing probabilities with states in a parsing table rather than simply with the rules themselves (Briscoe and Carroll, 1993). However, such remedies have …

Content truncated to 500 words