Exploiting Syntactic Structure for Language Modeling

Exploiting Syntactic Structure for Language Modeling Ciprian Chelba and Frederick Jelinek Center for Language and Speech Processing The Johns Hopkins University, Barton Hall 320 3400 N. Charles St., Baltimore, MD-21218, USA {chelba,jelinek} @jhu.edu Abstract The paper presents a language model that devel- ops syntactic structure and uses it to extract mean- ingful information from the word history, thus en- abling the use of long distance dependencies. The model assigns probability to every joint sequence of words-binary-parse-structure with headword an- notation and operates in a left-to-right manner -- therefore usable for automatic speech recognition. The model, its probabilistic parameterization, and a set of experiments meant to evaluate its predictive power are presented; an improvement over standard trigram modeling is achieved. 1 Introduction The main goal of the present work is to develop a lan- guage model that uses syntactic structure to model long-distance dependencies. During the summer96 DoD Workshop a similar attempt was made by the dependency modeling group. The model we present is closely related to the one investigated in (Chelba et al., 1997), however different in a few important aspects: • our model operates in a left-to-right manner, al- lowing the decoding of word lattices, as opposed to the one referred to previously, where only whole sen- tences could be processed, thus reducing its applica- bility to n-best list re-scoring; the syntactic structure is developed as a model component; • our model is a factored version of the one in (Chelba et al., 1997), thus enabling the calculation of the joint probability of words and parse structure; this was not possible in the previous case due to the huge computational complexity of the model. Our model develops syntactic structure incremen- tally while traversing the sentence from left to right. This is the main difference between our approach and other approaches to statistical natural language parsing. Our parsing strategy is similar to the in- cremental syntax ones proposed relatively recently in the linguistic community (Philips, 1996). The probabilistic model, its parameterization and a few experiments that are meant to evaluate its potential for speech recognition are presented. /// /~tract NP the_DT contract NN ~,dc,l VBI)with INa DT Ioss_NN of_IN 7_CD ~: ~::, X~,'~ after Figure 1: Partial parse 2 The Basic Idea and Terminology Consider predicting the word after in the sentence: the contract ended with a loss of 7 cents after trading as low as 89 cents. A 3-gram approach would predict after from (7, cents) whereas it is intuitively clear that the strongest predictor would be ended which is outside the reach of even 7-grams. Our assumption is that what enables humans to make a good prediction of after is the syntactic structure in the past. The linguistically correct partial parse of the word his- tory when predicting after is shown in Figure 1. The word ended is called the headword of the con- stituent (ended (with (...) )) and ended is an ex- posed headword when predicting after -- topmost headword in the largest constituent that contains …

Content truncated to 500 words