The Use of Clustering Techniques for Language Modeling V Application to Asian Language

27 Computational Linguistics and Chinese Language Processing Vol. 6, No. 1, February 2001, pp. 27-60 © Computational Linguistics Society of R.O.C. The Use of Clustering Techniques for Language Modeling – Application to Asian Language Jianfeng Gao*, Joshua T. Goodman+, Jiangbo Miao** Abstract Cluster-based n-gram modeling is a variant of normal word-based n-gram modeling. It attempts to make use of the similarities between words. In this paper, we present an empirical study of clustering techniques for Asian language modeling. Clustering is used to improve the performance (i.e. perplexity) of language models as well as to compress language models. Experimental tests are presented for cluster-based trigram models on a Japanese newspaper corpus and on a Chinese heterogeneous corpus. While the majority of previous research on word clustering has focused on how to get the best clusters, we have concentrated our research on the best way to use the clusters. Experimental results show that some novel techniques we present work much better than previous methods, and achieve more than 40% size reduction at the same level of perplexity. 1. Introduction Statistical language modelling (SLM) has been successfully applied in many domains, such as speech recognition, optical character recognition, machine translation, spelling correction, information retrieval, and spoken language understanding [Jelinek, 1990; Church, 1988; Brown et al., 1990; Kernighan et al., 1990; Miller et al., 1999; Zue, 1995]. The dominant technology in SLM is n-gram models. Typically, n-gram models are trained on very large corpora. In constructing n-gram models, we always face two problems. First, for a general domain model, large amounts of training data can lead to models that are too large for realistic applications. On the other hand, for specific domains, n-gram models usually suffer from the data sparseness problem because large amounts of domain-specific data are usually not available. When n-gram models are used, we can define clusters for similar words in a corpus. We * Microsoft Research, Asia, Beijing, 100080, P.R.C. E-mail: jfgao@microsoft.com + Microsoft Research, Redmond Washington 98052, USA. E-mail: joshuago@microsoft.com ** Department of Computer & Information Sciences University of Delaware, USA. This work was done while the author was visiting Microsoft Research Asia. 28 J. Gao et al. thus extend word-based n-gram models to cluster-based n-gram models. This has been demonstrated as an effective way to handle the data sparseness problem. Recent research also shows that cluster-based n-gram models are effective for rapid domain adaptation, training on small data sets, and reducing the memory requirements for realistic applications. Extending our previous work in [Goodman, 2001; Gao et al., 2001; Goodman and Gao, 2000], this paper presents an empirical study of clustering techniques for Asian language modeling. Clustering is used to improve the performance (i.e. perplexity) of language models as well as to compress language models. Experimental tests will be presented for cluster-based trigram models on a Japanese newspaper corpus of more than 10 million words, and on a Chinese heterogeneous corpus of more than 11 million characters. The majority of the previous research on word clustering has focused on …

Content truncated to 500 words