A Hierarchical Word Sequence Language Model

PACLIC 28 !489 A Hierarchical Word Sequence Language Model Xiaoyi Wu Nara Institute of Science and Technology Computational Linguistics Laboratory 8916-5 Takayama, Ikoma, Nara Japan xiaoyi-w@is.naist.jp Yuji Matsumoto Nara Institute of Science and Technology Computational Linguistics Laboratory 8916-5 Takayama, Ikoma, Nara Japan matsu@is.naist.jp Abstract Most language models used for natural lan- guage processing are continuous. However, the assumption of such kind of models is too simple to cope with data sparsity problem. Al- though many useful smoothing techniques are developed to estimate these unseen sequences, it is still important to make full use of contex- tual information in training data. In this paper, we propose a hierarchical word sequence lan- guage model to relieve the data sparsity prob- lem. Experiments verified the effectiveness of our model. 1 Introduction Most language models used for natural language processing, such as n-gram approach proposed by Shannon (1948), are continuous. However, the as- sumption that a word depends upon the preceding n-1 words is too simple to cope with data sparsity problem. Thus, a number of useful smoothing techniques such as back-off (Katz,1987), Kneser-Ney (Kneser & Ney,1995), modified Kneser-Ney (Chen & Good- man,1999) have been developed to estimate the probabilities of unseen sequences. Yet even with 30 years worth of newswire text, more than one third of all trigrams are unseen (Allison et al., 2005). It is still important to make full use of contextual infor- mation hidden in training data. D. Guthrie. et. al. (2006) proposed using skip- gram (Huang et. al., 1993) to overcome the data sparsity problem. The skip-gram model using dis- continuous sequences to model languages has truly helped to decrease the unseen sequences, but we should not neglect the fact that it also brings the greatly increase of processing time and redundant contexts. D. Guthrie. et. al. (2006) examined the coverage of skip-gram, but didn’t analyze the effi- ciency of them, which will be discussed in section 4.3 and section 4.4 in this paper. Taking into account of the balance between cov- erage and usage, we present a hierarchical word se- quence model to relieve the data sparsity problem. Differing from other hierarchical language models, such as hierarchical phrase-based model (Chiang, 2007) used in SMT systems, our model is essentially a n-gram language model whose modeling assump- tion is determined by tree structures. We introduce our main idea in Section 2. In Sec- tion 3, we propose the hierarchical word sequence model. We show the effectiveness of this model by several experiments in Section 4 and conclude in Section 5. 2 Basic Ideas Data sparsity is caused by the low frequency word combinations and unknown word combinations, which are inevitably increased by the assumption that a word depends upon the preceding n-1 words. For instance, given two sentences A = ’I hit the tennis ball’ and B = ’I hit the ball’, suppose that A is in the training data and B is in the test data, then the bigram (the, ball) and trigram (hit, the, ball) will not …

Content truncated to 500 words