Statistical Language Modeling with Performance Benchmarks using Various Levels of Syntactic-Semantic Information

Statistical Language Modeling with Performance Benchmarks using Various Levels of Syntactic-Semantic Information Dharmendra KANEJIYA∗, Arun KUMAR†, Surendra PRASAD∗ ∗Department of Electrical Engineering †Centre for Applied Research in Electronics Indian Institute of Technology New Delhi 110016 INDIA kanejiya@hotmail.com, arunkm@care.iitd.ernet.in, sprasad@ee.iitd.ernet.in Abstract Statistical language models using n-gram approach have been under the criticism of neglecting large-span syntactic-semantic in- formation that influences the choice of the next word in a language. One of the ap- proaches that helped recently is the use of latent semantic analysis to capture the se- mantic fabric of the document and enhance the n-gram model. Similarly there have been some approaches that used syntactic analysis to enhance the n-gram models. In this paper, we explain a framework called syntactically enhanced latent semantic anal- ysis and its application in statistical lan- guage modeling. This approach augments each word with its syntactic descriptor in terms of the part-of-speech tag, phrase type or the supertag. We observe that given this syntactic knowledge, the model outperforms LSA based models significantly in terms of perplexity measure. We also present some observations on the effect of the knowledge of content or function word type in language modeling. This paper also poses the prob- lem of better syntax prediction to achieve the benchmarks. 1 Introduction Statistical language models consist of estimat- ing the probability distributions of a word given the history of words so far used. The standard n-gram language model considers two histories to be equivalent if they end in the same n − 1 words. Due to the tradeoff between predictive power and reliability of estimation, n is typically chosen to be 2 (bi-gram) or 3 (tri-gram). Even tri-gram model suffers from sparse-data estima- tion problem, but various smoothing techniques (Goodman, 2001) have led to significant im- provements in many applications. But still the criticism that n-grams are unable to capture the long distance dependencies that exist in a lan- guage, remains largely valid. In order to model the linguistic structure that spans a whole sentence or a paragraph or even more, various approaches have been taken re- cently. These can be categorized into two main types : syntactically motivated and semanti- cally motivated large span consideration. In the first type, probability of a word is decided based on a parse-tree information like grammat- ical headwords in a sentence (Charniak, 2001) (Chelba and Jelinek, 1998), or based on part- of-speech (POS) tag information (Galescu and Ringger, 1999). Examples of the second type are (Bellegarda, 2000) (Coccaro and Jurafsky, 1998), where latent semantic analysis (LSA) (Landauer et al., 1998) is used to derive large- span semantic dependencies. LSA uses word- document co-occurrence statistics and a matrix factorization technique called singular value de- composition to derive semantic similarity mea- sure between any two text units - words or doc- uments. Each of these approaches, when inte- grated with n-gram language model, has led to improved performance in terms of perplexity as well as speech recognition accuracy. While each of these approaches has been stud- ied …

Content truncated to 500 words