Using Various Features in Machine Learning to Obtain High Levels of Performance for Recognition of Japanese Notational Variants

Using Various Features in Machine Learning to Obtain High Levels of Performance for Recognition of Japanese Notational Variants Masahiro Kojima a, Masaki Murata b, Jun’ichi Kazama c, Kow Kuroda c, Atsushi Fujita d,c, Eiji Aramaki e, Masaaki Tsuchida c, Yasuhiko Watanabe a, and Kentaro Torisawa c aDept. of Media Informatics, Ryukoku University, Seta, Otsu-shi, Shiga, 520-2194, Japan t10m101@mail.ryukoku.ac.jp,watanabe@rins.ryukoku.ac.jp bDept. of Information and Electronics, Tottori University, Koyama-Minami, Tottori, 680-8550, Japan murata@ike.tottori-u.ac.jp cNational Institute of Information and Communications Technology, Hikaridai, Seika-cho, Kyoto, Japan {kazama, kuroda, m-tsuchida, torisawa}@nict.go.jp dFaculty of Systems Information Science, Future University Hakodate, Hakodate, 041-8655, Japan fujita@fun.ac.jp eCenter for Knowledge Structuring, University of Tokyo, Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan eiji.aramaki@gmail.com Abstract. We proposed a method of using machine learning with various features for the recognition of Japanese notational variants. We increased 0.06 at the F-measure by specific features using existing dictionaries and character pairs useful for recognizing notational vari- ants and obtained 0.91 at the F-measure for the recognition of notational variants. By using the method, we could extract 160 thousand word pairs with a precision rate of 0.9. We also constructed a method using patterns in addition to machine learning and observed that we could extract 4.2 million notational variant pairs with a precision rate of 0.78. We confirmed that our method was much better than an existing method through experiments. Keywords: machine learning, Japanese notational variant, various features, variant pair, edit distance. 1 Introduction In Japanese, there exist several different and similar notational variants of a word. For example, “ ” (su pa ge te i) and “ ” (su pa ge tsu te i) can be used for “spaghetti”. A dictionary including such notational variants is extremely useful in many cases such as normal- ization of word expressions in information retrieval and normalization of word expressions in text mining or information extraction. Therefore, we studied the automatic extraction of the notational variants of a Japanese word. In terms of related studies, Brill et al. extracted pairs of Japanese Katakana expressions and their corresponding English expressions (Brill et al., 2001), McCallum et al. detected matching or mismatching in restaurant names or paper citation data (McCallum et al., 2005), and Tsuruoka et al., Aramaki et al., and Okazaki et al. extracted synonyms or notational variants in a restricted do- main such as biology and medical science (Tsuruoka et al., 2007; Aramaki et al., 2008; Okazaki et al., 2008). In contrast to these studies, we handled the automatic extraction of Japanese notational variants in all the domains. We used supervised machine learning approaches, in which we can easily use various types of information in the process of extracting Japanese notational variants. By supervised machine learning approaches (Suzaki et al., 2002; Masuyama and Nakagawa, 2004; Shimada et al., 2005), we used the information that existing dictionaries, including nota- tional variants, provide. We also used the information that similar characters such as “A/a” and “ Copyright 2010 by Masahiro Kojima, Masaki Murata, Jun’ichi Kazama, Kow Kuroda, Atsushi Fujita, Eiji Aramaki, Masaaki Tsuchida, …

Content truncated to 500 words