Nonlocal Language Modeling based on Context Co-occurrence Vectors

Nonlocal Language Modeling based on Context Co-occurrence Vectors Sadao Kurohashi and Manabu Ori Graduate School of Informatics Kyoto University Yoshida-honmachi, Sakyo; Kyoto, 606-8501 Japan kuro@±, ky0to-u, ac. j p, oriOpine, kuee. kyoto-u, ac. j p Abstract This paper presents a novel nonlocal lmlguage model which utilizes contextual information. A reduced vector space model calculated from co-occurrences of word pairs provides word co-occurrence vectors. The sum of word co- occurrence vectors represents tile context of a document, and the cosine similarity between the context vector and the word co-occurrence vectors represents the ]ong-distmlce lexical de- pendencies. Experiments on the Mainichi Newspaper corpus show significant improve- ment in perplexity (5.070 overall and 27.2% on target vocabulary) 1 Introduction Human pattern recognition rarely handles iso- lated or independent objects. We recog- nize objects in various spatiotemporal circum- stances such as an object in a scene, a word in an uttermlce. These circumstances work as conditions, eliminating ambiguities and en- abling robust recognition. The most challeng- ing topics in machine pattern recognition are in what representation and to what extent those circumstances are utilized. In laalguage processing, a context--that is; a portion of the utterance or the text before the object--is ml important circumstmlce. One way of representing a context is statis- tical language nmdels which provide a word sequence probability, P(w~), where w~ de- notes the sequence wi...wj. In other words, they provide the conditional probability of a word given with the previous word sequence, P( wilw~-l ), which shows the prediction of a word in a given context. The most conmmn laalguage models used nowadays are N-granl models based on a (N- 1)-th order Markov process: event pre- dictions depend on at most (N- 1) previous events. Therefore, they offer the following ap- proximation: P(w.ilw -1) wiJwi_N+l) (I) A common value for N is 2 (bigram language model) or 3 (trigram language model); only a short local context of one or two words is considered. Even such a local context is effective in some cases. For example, in Japanese, after the word kokumu 'state affairs', words such as daijin 'minister' mad shou 'department' likely follow; kaijin 'monster' and shou 'priZe' do not. After dake de 'only at', you cml often find wa (topic-marker), but you hardly find ga (nominative-marker) or wo (accusative- marker). These examples show behaviors of compound nouns and function word sequences are well handled by bigram mad trigraan mod- els. These models are exploited in several ap- plications such as speech recognition, optical character recognition and nmrphological anal- ysis. Local language models, however, cannot predict nmch in some cases. For instance, the word probability distribution after de wa 'at (topic-marker)' is very flat. However, even if the probability distribution is flat in local lan- guage models, the probability of daijin 'min- ister' and kaijin 'monster' must be very differ- ent in documents concenfing politics. Bigram and trigram models are obviously powerless to such kind of nonlocal, long-distmlce lexical dependencies. This paper presents a nonlocal language model. The important information …

Content truncated to 500 words