Statistical Language Processing Using Hidden Understanding Models

STATISTICAL LANGUAGE PROCESSING USING HIDDEN UNDERSTANDING MODELS Scott Miller College of Computer Science Northeastern University Boston, MA 02115 Richard Schwartz BBN Systems and Technologies 70 Fawcett St., Cambridge, MA 02138 Robert Bobrow BBN Systems and Technologies 70 Fawcett St., Cambridge, MA 02138 Robert Ingria BBN Systems and Technologies 70 Fawcett St., Cambridge, MA 02138 ABSTRACT This paper introduces a class of statistical mechanisms, called hidden understanding models, for natural language processing. Much of the framework for hidden understanding models derives from statistical models used in speech recognition, especially the use of hidden Markov models. These techniques are applied to the central problem of determining meaning directly from a sequence of spoken or written words. We present an overall description of the hidden understanding methodology, and discuss some of the critical implementation issues. Finally, we report on experimental results, including results of the December 1993 AR.PA evaluation. 1 INTRODUCTION Hidden understanding models are an innovative application of statistical mechanisms that, given a string of words, determines the most likely meaning for the string. The overall approach represents a substantial departure from traditional approaches by replacing hand-crafted grammars and rules with statistical models that are automatically learned from examples. Advantages of tiffs approach include potential improvements in both robustness and portability of natural language systems. Hidden understanding models were motivated by techniques that have been extremely successful in speech recognition, especially hidden Markov Models [Baum, 72]. Related techniques have previously been applied to the problem of segmenting a sentence into a sequence of concept relations [Pieraccini et aL, 91]. However, because of differences between language understanding and speech recognition, significant changes are required in the speech recognition methodology. Unlike speech, where each phoneme results in a local sequence of spectra, the relation between the meaning of a sentence and the sequence of words is not a simple linear sequential model. Language is inherently nested, with subgroups of concepts within other concepts. A statistical system for understanding language must take this and other differences into account in its overall design. In principle, we have the following requirements for a hidden understanding system: A notational system for expressing meanings. A statistical model that is capable of representing meanings and the association between meanings and words. An automatic training program which, given pairs of meanings and word sequences, can estimate the parameters of a statistical model. An understanding program that can search the statistical model to fred the most likely meaning given a word sequence. "J training [/ m eanin$ sentences 1"- program ~ expressions 4, I" program I I" expressmns J Figure 1: The main components of a hidden understanding system. Below, we describe solutions for each of these requirements, and report on initial experiments with hidden understanding models. -2 EXPRESSING MEANINGS One of the key requirements for a hidden understanding model is that the meaning representation must be both expressive and appropriate for automatic learning techniques. Logical notations, such as the predicate calculus, are generally considered to possess sufficient expressive power. The difficulty …

Content truncated to 500 words