Exploiting Syntactic Structure for Language Modeling

Exploiting Syntactic Structure for Language Modeling Ciprian Chelba and Frederick Jelinek Center for Language and Speech Processing The ,Johns Hopkins University, Barton Hall 320 3400 N. Charles St., Baltimore, MD-21218, USA {chelba,jelinek} @jhu.edu Abstract The paper presents a language model that devel- ops syntactic structure and uses it to extract nman- ingflfl information from the word history, thus en- abling the use of long distance dependencies. The model assigns probability to every joint sequence of words-binary-parse-structure with headword an- notation and operates in a left-to-right manner - therefore usable for automatic speech recognition. The model, its probabilistic parameterization, and a set of experinmnts meant to evaluate its predictive power are presented; an improvement over standard trigram modeling is achieved. 1 Introduction The main goal of the present work is to develop a lan- guage model that uses syntactic structure to model tong-distance dependencies. During the summer96 DoD Workshop a similar attempt was made by the dependency modeling group. The model we present is closely related to tile one investigated in (Chelba et al., 1997), however different in a few important aspects: • our model operates in a left-to-right manner, al- lowing the decoding of word lattices, as opposed to the one referred to previously, where only whole sen- tences could be processed, thus reducing its applica- bility to n-best list re-scoring; the syntactic structure is developed as a model component; • our model is a factored version of the one in (Chelba et al., 1997), thus enabling tile calculation of the joint probability of words and parse structure; this was not possible in the previous case due to the huge computational complexity of the model. Our model develops syntactic structure incremen- tally while traversing the sentence fl'om left to right. This is the main difference between our approach and other approaches to statistical natural language parsing. Our parsing strategy is similar to tile in- cremental syntax ones proposed relatively recently in the linguistic community (Philips, 1996). The probabilistic model, its parameterization and a few experiments that are meant to evaluate its potential for speech recognition m'e presented. Ih¢_DT clmlract NN • i,Jc,! ",M) with IN a_DT loss NN oL[N 7 CD ~. \i.> alter Figure 1: Partial parse 2 The Basic Idea and Terminology Consider predicting the word after in the sentence: the contract ended with a loss of 7 cents after trading as low as 89 cents, A 3-gram approach would predict after from (7, cents) whereas it is intuitively clear that the strongest predictor would be ended which is outside tile reach of even 7-grams. Our assumption is that what enables humans to make a good prediction of after is the syntactic structure in the past. The linguistically correct partial parse of the word his- tory when predicting after is shown in Figure 1. The word ended is called the headword of tile con- stitucnt (ended (with (...)) ) and ended is all ex- posed headword when predicting after -- topmost headword in the largest constituent that contains …

Content truncated to 500 words