A Structured Language Model

A Structured Language Model Ciprian Chelba The Johns Hopkins University CLSP, Barton Hall 320 3400 N. Charles Street, Baltimore, MD-21218 chelba@j hu. edu Abstract The paper presents a language model that develops syntactic structure and uses it to extract meaningful information from the word history, thus enabling the use of long distance dependencies. The model as- signs probability to every joint sequence of words-binary-parse-structure with head- word annotation. The model, its proba- bilistic parametrization, and a set of ex- periments meant to evaluate its predictive power are presented. the dog I heard yesterday barked Figure 1: Partial parse '¢"~.( ~ I h_{-=*l ) ~_{-I [ h_O ~ w_l ... w..p ........ w q...w~r w_lr+ll ...w_k w_lk+l} ..... w_n </s> Figure 2: A word-parse k-prefix 1 Introduction The main goal of the proposed project is to develop a language model(LM) that uses syntactic structure. The principles that guided this propo§al were: • the model will develop syntactic knowledge as a built-in feature; it will assign a probability to every joint sequence of words-binary-parse-structure; • the model should operate in a left-to-right man- ner so that it would be possible to decode word lat- tices provided by an automatic speech recognizer. The model consists of two modules: a next word predictor which makes use of syntactic structure as developed by a parser. The operations of these two modules are intertwined. 2 The Basic Idea and Terminology Consider predicting the word barked in the sen- tence: the dog I heard yesterday barked again. A 3-gram approach would predict barked from (heard, yesterday) whereas it is clear that the predictor should use the word dog which is out- side the reach of even 4-grams. Our assumption is that what enables us to make a good predic- tion of barked is the syntactic structure in the past. The correct partial parse of the word his- tory when predicting barked is shown in Figure 1. The word dog is called the headword of the con- stituent ( the (dog (...) )) and dog is an exposed headword when predicting barked -- topmost head- word in the largest constituent that contains it. The syntactic structure in the past filters out irrelevant words and points to the important ones, thus en- abling the use of long distance information when predicting the next word. Our model will assign a probability P(W, T) to every sentence W with ev- ery possible binary branching parse T and every possible headword annotation for every constituent of T. Let W be a sentence of length I words to which we have prepended <s> and appended </s> so that wo =<s> and wl+l =</s>. Let Wk be the word k-prefix w0... wk of the sentence and WkT~ the word-parse k-prefix. To stress this point, a word-parse k-prefix contains only those binary trees whose span is completely included in the word k- prefix, excluding wo =<s>. Single words can be re- garded as root-only trees. Figure 2 shows a word- parse k-prefix; h_0 .. h_{-m} …

Content truncated to 500 words