Attention Shifting for Parsing Speech

Attention Shifting for Parsing Speech ∗ Keith Hall Department of Computer Science Brown University Providence, RI 02912 kh@cs.brown.edu Mark Johnson Department of Cognitive and Linguistic Science Brown University Providence, RI 02912 Mark Johnson@Brown.edu Abstract We present a technique that improves the efficiency of word-lattice parsing as used in speech recogni- tion language modeling. Our technique applies a probabilistic parser iteratively where on each iter- ation it focuses on a different subset of the word- lattice. The parser’s attention is shifted towards word-lattice subsets for which there are few or no syntactic analyses posited. This attention-shifting technique provides a six-times increase in speed (measured as the number of parser analyses evalu- ated) while performing equivalently when used as the first-stage of a multi-stage parsing-based lan- guage model. 1 Introduction Success in language modeling has been dominated by the linear n-gram for the past few decades. A number of syntactic language models have proven to be competitive with the n-gram and better than the most popular n-gram, the trigram (Roark, 2001; Xu et al., 2002; Charniak, 2001; Hall and Johnson, 2003). Language modeling for speech could well be the first real problem for which syntactic techniques are useful. John ate the pizza on a plate with a fork . NP:plate NP:fork PP:with PP:on IN IN VB NP VP:ate Figure 1: An incomplete parse tree with head-word an- notations. One reason that we expect syntactic models to perform well is that they are capable of model- ing long-distance dependencies that simple n-gram ∗ This research was supported in part by NSF grants 9870676 and 0085940. models cannot. For example, the model presented by Chelba and Jelinek (Chelba and Jelinek, 1998; Xu et al., 2002) uses syntactic structure to identify lexical items in the left-context which are then mod- eled as an n-gram process. The model presented by Charniak (Charniak, 2001) identifies both syn- tactic structural and lexical dependencies that aid in language modeling. While there are n-gram mod- els that attempt to extend the left-context window through the use of caching and skip models (Good- man, 2001), we believe that linguistically motivated models, such as these lexical-syntactic models, are more robust. Figure 1 presents a simple example to illustrate the nature of long-distance dependencies. Using a syntactic model such as the the Structured Lan- guage Model (Chelba and Jelinek, 1998), we pre- dict the word fork given the context {ate, with} where a trigram model uses the context {with, a}. Consider the problem of disambiguating between . . . plate with a fork and . . . plate with effort. The syntactic model captures the semantic relationship between the words ate and fork. The syntactic struc- ture allows us to find lexical contexts for which there is some semantic relationship (e.g., predicate- argument). Unfortunately, syntactic language modeling tech- niques have proven to be extremely expensive in terms of computational effort. Many employ the use of string parsers; in order to utilize such tech- niques for language modeling one must preselect a set of …

Content truncated to 500 words