A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis

International Joint Conference on Natural Language Processing, pages 162–170, Nagoya, Japan, 14-18 October 2013. A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis Ryohei Sasano1 Sadao Kurohashi2 Manabu Okumura1 1 Precision and Intelligence Laboratory, Tokyo Institute of Technology 2 Graduate School of Informatics, Kyoto University {sasano,oku}@pi.titech.ac.jp, kuro@i.kyoto-u.ac.jp Abstract This paper presents a simple but effec- tive approach to unknown word processing in Japanese morphological analysis, which handles 1) unknown words that are de- rived from words in a pre-defined lexicon and 2) unknown onomatopoeias. Our ap- proach leverages derivation rules and ono- matopoeia patterns, and correctly recog- nizes certain types of unknown words. Ex- periments revealed that our approach rec- ognized about 4,500 unknown words in 100,000 Web sentences with only 80 harm- ful side effects and a 6% loss in speed. 1 Introduction Morphological analysis is the first step in many natural language applications. Since words are not segmented by explicit delimiters in Japanese, Japanese morphological analysis consists of two subtasks: word segmentation and part-of-speech (POS) tagging. Japanese morphological anal- ysis has successfully adopted lexicon-based ap- proaches for newspaper articles (Kurohashi et al., 1994; Asahara and Matsumoto, 2000; Kudo et al., 2004), in which an input sentence is trans- formed into a lattice of candidate words using a pre-defined lexicon, and an optimal path in the lat- tice is then selected. Figure 1 shows an example of a word lattice for morphological analysis and an optimal path. Since the transformation from a sentence into a word lattice basically depends on the pre-defined lexicon, the existence of unknown words, i.e., words that are not included in the pre- defined lexicon, is a major problem in Japanese morphological analysis. There are two major approaches to this prob- lem: one is to augment the lexicon by acquiring unknown words from a corpus in advance (Mori and Nagao, 1996; Murawaki and Kurohashi, 2008) and the other is to introduce better un- known word processing to the morphological ana- Input : “�����” (My father is a Japanese.) Lattice : � (father) [Noun] � (is) [Particle] � (tooth) [Noun] � (day) [Noun] �� (Japanese) [Noun] � (book) [Noun] � (man) [Noun] �� (the identical person) [Noun] EOS BOS Figure 1: Example of word lattice. The bold lines indicate the optimal path. lyzer (Nagata, 1999; Uchimoto et al., 2001; Asa- hara and Matsumoto, 2004; Azuma et al., 2006; Nakagawa and Uchimoto, 2007). Although both approaches have their own advantages and should be exploited cooperatively, this paper focuses only on the latter approach. Most previous work on this approach has aimed at developing a single general-purpose unknown word model. However, there are several types of unknown words, some of which can be easily dealt with by introducing simple derivation rules and unknown word patterns. In addition, as we will discuss in Section 2.3, the importance of un- known word processing varies across unknown word types. In this paper, we aim to deal with unknown words that are considered important and can be dealt with using …

Content truncated to 500 words