Language Model Based on Word Clustering

Language Model Based on Word Clustering Lichi Yuan School of Information Technology, Jiangxi University of Finance & Economics，Nanchang 330013，China yuan_lichi@hotmail.com Abstract. Category-based statistic language model is an important method to solve the problem of sparse data. But there are two bottlenecks about this model: (1) the problem of word clustering, it is hard to find a suitable clustering method that has good performance and not large amount of computation. (2) class-based method always loses some prediction ability to adapt the text of different domain. The authors try to solve above problems in this paper. This paper presents a definition of word similarity by utilizing mutual information. Based on word similarity, this paper gives the definition of word set similarity. Experiments show that word clustering algorithm based on similarity is better than conventional greedy clustering method in speed and performance. At the same time, this paper presents a new method to create the vari-gram model. Keywords: Word clustering, statistical language model, Vari-gram language model 1 Introduction Statistical language modeling[5] (SLM) has been successfully applied to many domains such as speech recognition, machine translation, information retrieval, and spoken language understanding. The dominant technology in SLM is n-gram models. Typically n-gram models are trained on very large corpora. In constructing n-gram models, we always face two problems. First, for a general domain model, large amounts of training data can lead to models that are too large for realistic applications. On the other hand, for specific domains, n-gram models usually suffer from the data sparseness problem, because large amounts of domain-specific data are usually not available. When n-gram models are used, we can define clusters for similar words in a corpus. We thus augment word-based n-gram models to cluster-based n-gram models. This has been demonstrated as an effective way to handle the data sparseness problem. There are many different clustering algorithms[1], but they can be classified into a few basic types. There are two types of structures produced by clustering algorithms, hierarchical clustering and flat or non- hierarchical clustering. Flat clustering simply consists of a certain number of clusters and the relation between clusters is often undetermined. Most algorithms produce flat clusters and improve them by iterating a reallocation operation that reassigns objects. The tree of a hierarchical clustering can be produced either bottom-up, by starting with the individual objects and grouping the most similar ones, or top-down, whereby one starts with all the objects and divides them into groups so as to maximize within-group similarity. This paper proposes a bottom-up hierarchical clustering algorithm based on similarity. This paper also discusses the problem of the design of vari–gram[3]. The cluster n-gram model has been demonstrated as an effective way to deal with the data sparseness problem which exists in the word n-gram model, but this method sacrifice some predictable ability. We can improve the system predict performance by enhancing n, because the number of clusters is much less the number of words. However, it also has a few drawbacks, including: exponential growth in parameters …

Content truncated to 500 words