Modeling of Long Distance Context Dependency

Modeling of Long Distance Context Dependency ZHOU GuoDong Institute for Infocomm Research 21 Heng Mui Keng Terrace Singapore 119613 Email: zhougd@i2r.a-star.edu.sg Abstract Ngram models are simple in language modeling and have been successfully used in speech recognition and other tasks. However, they can only capture the short distance context dependency within an n-words window where currently the largest practical n for a natural language is three while much of the context dependency in a natural language occurs beyond a three words window. In order to incorporate this kind of long distance context dependency in the ngram model of our Mandarin speech recognition system, this paper proposes a novel MI-Ngram modeling approach. This new MI-Ngram model consists of two components: a normal ngram model and a novel MI model. The ngram model captures the short distance context dependency within an n-words window while the MI model captures the context dependency between the word pairs over a long distance by using the concept of mutual information. That is, the MI-Ngram model incorporates the word occurrences beyond the scope of the normal ngram model. It is found that MI- Ngram modeling has much better performance than the normal word ngram modeling. Experimentation shows that about 20% of errors can be corrected by using a MI-Trigram model compared with the pure word trigram model. 1 Introduction Language modeling is the attempt to characterize, capture and exploit the regularities and constraints in a natural language and has been successfully applied to many domains. Among all the language modeling approaches, ngram models have been most widely used in speech recognition (Jelinek 1990; Gale and Church 1990; Brown et al. 1992; Yang et al. 1996) and other applications. While ngram models are simple in language modeling and have been successfully used in speech recognition and other tasks, they have obvious deficiencies. For instance, ngram models can only capture the short-distance dependency within an n- words window where currently the largest practical N for a natural language is three. In the meantime, it is found that there always exist many preferred relationships between words. Two highly associated word pairs are “not only/but also” and “doctor/nurse”. Psychological experiments in Meyer D. et al. (1975) indicated that the human’s reaction to a highly associated word pair was stronger and faster than that to a poorly associated word pair. Such preference information is very useful for natural language processing (Church K.W. et al. 1990; Hiddle D. et al. 1993; Rosenfeld R. 1994 and Zhou G.D. et al.1998). Obviously, the preference relationships between words can expand from a short to long distance. While we can use conventional ngram models to capture the short distance dependency, the long distance dependency should also be exploited properly. The purpose of this paper is to propose a new modeling approach to capture the context dependency over both a short distance and a long distance and apply it in Mandarin speech recognition. This paper is organized as follows. In Section 2, we present the normal ngram modeling …

Content truncated to 500 words