Long Short-Term Memory for Japanese Word Segmentation

Long Short-Term Memory for Japanese Word Segmentation Yoshiaki Kitagawa∗and Mamoru Komachi Tokyo Metropolitan University 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan ace1235813@gmail.com, komachi@tmu.ac.jp Abstract This study presents a long short-term memory (LSTM) neural network approach to Japanese word segmentation (JWS). Previous studies on Chinese word segmentation have succeeded in using recurrent neural networks such as LSTM and gated recurrent units. However, in contrast to Chinese, Japanese includes several character types such as hiragana, katakana, and kanji, which produce orthographic varia- tions and increase the difficulty of word seg- mentation. Additionally, while it is important to consider a global context, traditional JWS approaches still rely on local features. To ad- dress this problem, this study proposes em- ploying an LSTM-based approach to JWS. 1 Introduction Word segmentation is a fundamental task of Japanese language processing. Moreover, word segmentation errors in East Asian languages (e.g., Japanese and Chinese), which lack a trivial word segmentation process, can cause problems for down- stream NLP applications. Thus, it is crucial to per- form accurate word segmentation for the Japanese language. To achieve high accuracy, modern Japanese word segmentation (JWS) methods utilize discriminative models relying on extensive feature engineering. However, machine-learning-based methods tend to require hand-crafted feature templates. Thus, they suffer from data sparseness. Neural network mod- els have, therefore, been investigated for various NLP tasks to address the problem of feature engi- neering (Liu et al., 2015; Sutskever et al., 2014; ∗Now at Yahoo Japan Corporation. Socher et al., 2013; Turian et al., 2010; Mikolov et al., 2013). Neural network models enable the use of dense feature vectors (i.e., embeddings) that are learned via representation learning. Another important problem in JWS corresponds to context modeling. Traditional JWS methods em- ploy feature templates to expand local features in a fixed window. However, global information beyond the window is not considered. Conversely, recur- rent neural network (RNN) models grasp long dis- tance information owing to the use of long short– term memory (LSTM), achieving state-of-the-art ac- curacy in Chinese word segmentation (Chen et al., 2015b). However, it is uncertain whether the LSTM approach is also effective for JWS because there are many types of character sets in Japanese that pro- duce orthographic variations. Therefore, we propose an LSTM network for JWS that incorporates character-level embeddings and long-distance dependency. The main contribu- tions of this study are as follows. • We propose an LSTM model for JWS and investigate methods to utilize sparse features, such as character type, character n-gram, and dictionary features. • The experimental results indicate that the proposed word segmentation model achieves comparable performance to conventional ap- proaches in both token- and sentence-level ac- curacy with respect to various datasets. • Our souce code is available at GitHub1. 1https://github.com/ace12358/ WordSegmentation PACLIC 32 279 32nd Pacific Asia Conference on Language, Information and Computation Hong Kong, 1-3 December 2018 Copyright 2018 by the authors 2 LSTM for Japanese Word Segmentation Machine-learning-based approaches for word seg- mentation build a classifier from an annotated cor- pus to classify the existence …

Content truncated to 500 words