Fertility Models for Statistical Natural Language Understanding

Fertility Models for Statistical Natural Language Understanding Stephen Della Pietra °, Mark Epstein, Salim Roukos, Todd Ward IBM Thomas J. Watson Research Center P.O. Box 218 Yorktown Heights, NY 10598, USA (*Now With Renaissance Technologies, Stonybrook, NY, USA) sdella@rentec, tom [meps/roukos/tward] ©watson. ibm. com Abstract Several recent efforts in statistical nat- ural language understanding (NLU) have focused on generating clumps of English words from semantic meaning concepts (Miller et al., 1995; Levin and Pierac- cini, 1995; Epstein et al., 1996; Epstein, 1996). This paper extends the IBM Ma- chine Translation Group's concept of fertil- ity (Brown et al., 1993) to the generation of clumps for natural language understand- ing. The basic underlying intuition is that a single concept may be expressed in Eng- lish as many disjoint clump of words. We present two fertility models which attempt to capture this phenomenon. The first is a Poisson model which leads to appeal- ing computational simplicity. The second is a general nonparametric fertility model. The general model's parameters are boot- strapped from the Poisson model and up- dated by the EM algorithm. These fertility models can be used to impose clump fertil- ity structure on top of preexisting clump generation models. Here, we present re- sults for adding fertility structure to uni- gram, bigram, and headword clump gener- ation models on ARPA's Air Travel Infor- mation Service (ATIS] domain. 1 Introduction The goal of a natural language understanding (NLU) system is to interpret a user's request and respond with an appropriate action. We view this interpre- tation as translation from a natural language ex- pression, E, into an equivalent expression, F, in an unambigous formal language. Typically, this for- mal language will be hand-crafted to enhance per- formance on some task-specific domain. A statisti- cal NLU system translates a request E as the most likely formal expression ~' according to a probability model p, = are maxp(F[E) --- are maxp(F, E). over all F over all F We have previously built a fully automatic statis- tical NLU system (Epstein et al., 1996) based on the source-channel factorization of the joint distribution p(f , E) p(f , E) = p(f)p(ZlF ). This factorization, which has proven effective in speech recognition (Bahl, Jelinek, and Mercer, 1983), partitions the joint probability into an a pri- ori intention model p(F), and a translation model p(E[F) which models how a user might phrase a re- quest F in English. For the ATIS task, our formal language is a mi- nor variant of the NL-Parse (Hemphill, Godfrey, and Doddington, 1990) used by ARPA to annotate the ATIS corpus. An example of a formal and natural language pair is: • F: List flights from New Orleans to Memphis flying on Monday departing early_morning • E: do you have any flights going to Memphis leaving New Orleans early Monday morning Here, the evidence for the formal language concept 'early_morning' resides in the two disjoint clumps of English 'early' and 'morning'. In this paper, we in- troduce the notion of concept …

Content truncated to 500 words