An Improved Hierarchical Word Sequence Language Model Using Directional Information

An Improved Hierarchical Word Sequence Language Model Using Directional Information Xiaoyi Wu Nara Institute of Science and Technology Computational Linguistics Laboratory 8916-5 Takayama, Ikoma, Nara Japan xiaoyi-w@is.naist.jp Yuji Matsumoto Nara Institute of Science and Technology Computational Linguistics Laboratory 8916-5 Takayama, Ikoma, Nara Japan matsu@is.naist.jp Abstract For relieving data sparsity problem, Hierarchi- cal Word Sequence (abbreviated as HWS) lan- guage model, which uses word frequency in- formation to convert raw sentences into spe- cial n-gram sequences, can be viewed as an effective alternative to normal n-gram method. In this paper, we use directional information to make HWS models more syntactically ap- propriate so that higher performance can be achieved. For evaluation, we perform intrin- sic and extrinsic experiments, both verify the effectiveness of our improved model. 1 Introduction Probabilistic Language Modeling is a fundamental research direction of Natural Language Processing. It is widely used in many applications such as ma- chine translation (Brown et al., 1990), spelling cor- rection (Mays et al., 1991), speech recognition (Ra- biner and Juang, 1993), word prediction (Bickel et al., 2005) and so on. Most research about Probabilistic Language Mod- eling, such as back-off (Katz,1987), Kneser-Ney (Kneser and Ney, 1995), and modified Kneser-Ney (Chen and Goodman, 1999), only focus on smooth- ing methods because they all take n-gram approach (Shannon, 1948) as a default setting for extracting word sequences from a sentence. Yet even with 30 years worth of newswire text, more than one third of all trigrams are still unseen (Allison et al., 2005), which cannot be distinguished accurately even us- ing a high-performance smoothing method such as modified Kneser-Ney (abbreviated as MKN). It is better to make these unseen sequences actually be observed rather than to leave them to smoothing method directly. For the purpose of extracting more valid word se- quences and relieving data sparsity problem, Wu and Matsumoto (2014) proposed a heuristic approach to convert a sentence into a hierarchical word sequence (abbreviated as HWS) structure, by which special n- grams can be achieved. In this paper, we improve HWS models by adding directional information for achieving higher performance. This paper is organized as follows. In Section 2, we give a complete review of the HWS language model. We present our improved HWS model in Section 3. In Section 4, we show the effectiveness of our model by several experiments. Finally, we summarize our findings in Section 5. 2 Review of HWS Language Model The HWS language model is defined as follows. Suppose that we have a frequency-sorted vocab- ulary list V = {v1, v2, ..., vm}, where C(v1) ≥ C(v2) ≥ ... ≥ C(vm)1. According to V , given any sentence S = w1, w2, ..., wn, the most frequently used word wi ∈ S(1 ≤ i ≤ n) can be se- lected2 for splitting S into two substrings SL = w1, ..., wi−1 and SR = wi+1, ..., wn. Sim- ilarly, for SL and SR, wj ∈ SL(1 ≤ j ≤ i − 1) and wk ∈ SR(i + 1 …

Content truncated to 500 words