A Study on Richer Syntactic Dependencies for Structured Language Modeling

A Study on Richer Syntactic Dependencies for Structured Language Modeling Peng Xu Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218 xp@clsp.jhu.edu Ciprian Chelba Microsoft Research One Microsoft Way Redmond, WA 98052 chelba@microsoft.com Frederick Jelinek Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218 jelinek@clsp.jhu.edu Abstract We study the impact of richer syntac- tic dependencies on the performance of the structured language model (SLM) along three dimensions: parsing accu- racy (LP/LR), perplexity (PPL) and word- error-rate (WER, N-best re-scoring). We show that our models achieve an im- provement in LP/LR, PPL and/or WER over the reported baseline results us- ing the SLM on the UPenn Treebank and Wall Street Journal (WSJ) corpora, respectively. Analysis of parsing per- formance shows correlation between the quality of the parser (as measured by pre- cision/recall) and the language model per- formance (PPL and WER). A remarkable fact is that the enriched SLM outperforms the baseline 3-gram model in terms of WER by 10% when used in isolation as a second pass (N-best re-scoring) language model. 1 Introduction The structured language model uses hidden parse trees to assign conditional word-level language model probabilities. As explained in (Chelba and Jelinek, 2000), Section 4.4.1, if the final best parse is used to be the only parse, the reduction in PPL —relative to a 3-gram baseline— using the SLM’s headword parametrization for word prediction is about 40%. The key to achieving this reduction is a good guess of the final best parse for a given sen- tence as it is being traversed left-to-right, which is much harder than finding the final best parse for the entire sentence, as it is sought by a regular statistical parser. Nevertheless, it is expected that techniques developed in the statistical parsing community that aim at recovering the best parse for an entire sen- tence, i.e. as judged by a human annotator, should also be productive in enhancing the performance of a language model that uses syntactic structure. The statistical parsing community has used var- ious ways of enriching the dependency structure underlying the parametrization of the probabilistic model used for scoring a given parse tree (Charniak, 2000) (Collins, 1999). Recently, such models (Char- niak, 2001) (Roark, 2001) have been shown to out- perform the SLM in terms of both PPL and WER on the UPenn Treebank and WSJ corpora, respectively. In (Chelba and Xu, 2001), a simple way of enriching the probabilistic dependencies in the CONSTRUC- TOR component of the SLM also showed better PPL and WER performance; the simple modifica- tion to the training procedure brought the WER per- formance of the SLM to the same level with the best as reported in (Roark, 2001). In this paper, we present three simple ways of enriching the syntactic dependency structure in the SLM, extending the work in (Chelba and Xu, 2001). The results show that an improved parser (as mea- sured by LP/LR) is indeed helpful in reducing the PPL and WER. Another remarkable fact is …

Content truncated to 500 words