Japanese Unknown Word Identification by Character-based Chunking

Japanese Unknown Word Identification by Character-based Chunking Masayuki Asahara and Yuji Matsumoto Graduate School of Information Science Nara Institute of Science and Technology, Japan {masayu-a,matsu}@is.naist.jp Abstract We introduce a character-based chunking for un- known word identification in Japanese text. A major advantage of our method is an ability to detect low frequency unknown words of unrestricted character type patterns. The method is built upon SVM-based chunking, by use of character n-gram and surround- ing context of n-best word segmentation candidates from statistical morphological analysis as features. It is applied to newspapers and patent texts, achiev- ing 95% precision and 55-70% recall for newspa- pers and more than 85% precision for patent texts. 1 Introduction Japanese and Chinese sentences are written without spaces between words. A word segmentation pro- cess is a prerequisite for natural language process- ing (NLP) of non-segmented language family. Sta- tistical morphological analyzers are often used for word segmentation in Japanese NLP, which achieve over 96% precision. However, unknown word pro- cessing still remains an issue to be addressed in those morphological analyzers. Unknown word processing in non-segmented languages are more challenging, as it first needs to identify boundaries of unknown words in texts, prior to assignment of correspoinding part-of-speech. Unknown word processing in morphological analysis of non-segmented language can follow one of either approaches: modular or embedded. In the modular approach, a separate off-line module is used to extract unknown words from text (Mori 1996; Ikeya 2000). They are checked and added to the lexicon of morphological analyzers. In the em- bedded approach, an on-line module which statisti- cally induces the likelihood of a particular string be- ing a word is embedded in a morphological analyzer (Nagata, 1999; Uchimoto et al., 2001). A modular approach is generally preferable in practice, since it allows developers to maintain a high quality lex- icon which is crucial for good performance. Previ- ous work of the modular approach was either un- able to detect low frequency unknown words (Mori 1996) or limited to predefined character patterns for low frequency unknown words (Ikeya 2000). We propose a general-purpose unknown word identification based on character-based chunking in order to address these shortcomings. A cascade model of a morphological analyzer (trained with Markov Model) and a chunker (trained with Sup- port Vector Machines) is applied. The morpho- logical analyzer produces n-best word segmenta- tion candidates, from which candidate segmentation boundaries, character n-gram and surrounding con- texts are extracted as features for each character. The chunker determines the boundaries of unknown words based on the features. The rest of this paper is as follows. We describe our method in Section 2, and present experimental results on newspaper articles and patent text in Sec- tion 3. Related work is provided in Section 4, and a summary and future directions are given in Section 5. 2 Method We describe our method for unknown word identifi- cation. The method is based on the following three steps: 1. A statistical morphological analyzer is applied to the input sentence …

Content truncated to 500 words