On Combining Language Models: Oracle Approach

On Combining Language Models : Oracle Approach � Kadri Hacioglu and Wayne Ward Center for Spoken Language Research University of Colorado at Boulder fhacioglu,whwg@cslr.colorado.edu ABSTRACT In this paper, we address the problem of combining several lan- guage models (LMs). We find that simple interpolation methods, like log-linear and linear interpolation, improve the performance but fall short of the performance of an oracle. The oracle knows the reference word string and selects the word string with the best per- formance (typically, word or semantic error rate) from a list of word strings, where each word string has been obtained by using a dif- ferent LM. Actually, the oracle acts like a dynamic combiner with hard decisions using the reference. We provide experimental results that clearly show the need for a dynamic language model combina- tion to improve the performance further. We suggest a method that mimics the behavior of the oracle using a neural network or a de- cision tree. The method amounts to tagging LMs with confidence measures and picking the best hypothesis corresponding to the LM with the best confidence. 1. INTRODUCTION Statistical language models (LMs) are essential in speech recog- nition and understanding systems for high word and semantic ac- curacy, not to mention robustness and portability. Several language models have been proposed and studied during the past two decades [8]. Although it has turned out to be a rather difficult task to beat the (almost) standard class/word n-grams (typically n = 2 or 3), there has been a great deal of interest in grammar based language models [1]. A promising approach for limited domain applications is the use of semantically motivated phrase level stochastic context free grammars (SCFGs) to parse a sentence into a sequence of se- mantic tags which are further modeled using n-grams [2, 9, 10, 3]. The main motivation behind the grammar based LMs is the inabil- ity of n-grams to model longer-distance constraints in a language. With the advent of fairly fast computers and efficient parsing and search schemes several researchers have focused on incorporating relatively complex language models into speech recognition and understanding systems at different levels. For example, in [3], we �The work is supported by DARPA through SPAWAR under grant #N66001-00-2-8906. report a significant perplexity improvement with a moderate in- crease in word/semantic accuracy, at N-best list (rescoring) level, using a dialog-context dependent, semantically motivated grammar based language model. Statistical language modeling is a ”learning from data” problem. The generic steps to be followed for language modeling are � preparation of training data � selection of a model type � specification of the model structure � estimation of model parameters The training data should consist of large amounts of text, which is hardly satisfied in new applications. In those cases, complex models fit to the training data. On the other hand, simple models can not capture the actual structure. In the Bayes’ (sequence) de- cision framework of speech recognition/understanding we heavily constrain the model structure to come up with a …

Content truncated to 500 words