Training Connectionist Models for the Structured Language Model

Training Connectionist Models for the Structured Language Model � Peng Xu, Ahmad Emami and Frederick Jelinek Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218 � xp,emami,jelinek � @jhu.edu Abstract We investigate the performance of the Structured Language Model (SLM) in terms of perplexity (PPL) when its compo- nents are modeled by connectionist mod- els. The connectionist models use a dis- tributed representation of the items in the history and make much better use of contexts than currently used interpolated or back-off models, not only because of the inherent capability of the connection- ist model in fighting the data sparseness problem, but also because of the sub- linear growth in the model size when the context length is increased. The connec- tionist models can be further trained by an EM procedure, similar to the previously used procedure for training the SLM. Our experiments show that the connectionist models can significantly improve the PPL over the interpolated and back-off mod- els on the UPENN Treebank corpora, after interpolating with a baseline trigram lan- guage model. The EM training procedure can improve the connectionist models fur- ther, by using hidden events obtained by the SLM parser. 1 Introduction In many systems dealing with natural speech or lan- guage such as Automatic Speech Recognition and � This work was supported by the National Science Founda- tion under grants No.IIS-9982329 and No.IIS-0085940. Statistical Machine Translation, a language model is a crucial component for searching in the often prohibitively large hypothesis space. Most of the state-of-the-art systems use n-gram language mod- els, which are simple and effective most of the time. Many smoothing techniques that improve lan- guage model probability estimation have been pro- posed and studied in the n-gram literature (Chen and Goodman, 1998). Recent efforts have studied various ways of us- ing information from a longer context span than that usually captured by normal n-gram language mod- els, as well as ways of using syntactical informa- tion that is not available to the word-based n-gram models (Chelba and Jelinek, 2000; Charniak, 2001; Roark, 2001; Uystel et al., 2001). All these language models are based on stochastic parsing techniques that build up parse trees for the input word sequence and condition the generation of words on syntactical and lexical information available in the parse trees. Since these language models capture useful hierar- chical characteristics of language, they can improve the PPL significantly for various tasks. Although more improvement can be achieved by enriching the syntactical dependencies in the structured language model (SLM) (Xu et al., 2002), a severe data sparse- ness problem was observed in (Xu et al., 2002) when the number of conditioning features was increased. There has been recent promising work in us- ing distributional representation of words and neu- ral networks for language modeling (Bengio et al., 2001) and parsing (Henderson, 2003). One great ad- vantage of this approach is its ability to fight data sparseness. The model size grows only sub-linearly with the number of predicting features …

Content truncated to 500 words