Morphological Analysis for Japanese Noisy Text based on Character-level and Word-level Normalization

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1773–1782, Dublin, Ireland, August 23-29 2014. Morphological Analysis for Japanese Noisy Text Based on Character-level and Word-level Normalization SAITO Itsumi, SADAMITSU Kugatsu, ASANO Hisako and MATSUO Yoshihiro NTT Media Intelligence Laboratories {saito.itsumi, sadamitsu.kugatsu, asano.hisako, matsuo.yoshihiro}@lab.ntt.co.jp Abstract Social media texts are often written in a non-standard style and include many lexical variants such as insertions, phonetic substitutions, abbreviations that mimic spoken language. The nor- malization of such a variety of non-standard tokens is one promising solution for handling noisy text. A normalization task is very difficult to conduct in Japanese morphological analysis because there are no explicit boundaries between words. To address this issue, in this paper we propose a novel method for normalizing and morphologically analyzing Japanese noisy text. We generate both character-level and word-level normalization candidates and use discriminative methods to formulate a cost function. Experimental results show that the proposed method achieves accept- able levels in both accuracy and recall for word segmentation, POS tagging, and normalization. These levels exceed those achieved with the conventional rule-based system. 1 Introduction Social media texts attract a lot of attention in the fields of information extraction and text mining. Al- though texts of this type contain a lot of information, such as one’s reputation or emotions, they often contain non-standard tokens (lexical variants) that are considered out-of-Vocabulary (OOV) terms. We define an OOV as a word that does not exist in the dictionary. Texts in micro-blogging services such as Twitter are particularly apt to contain words written in a non-standard style, e.g., by lengthening them (“goooood” for “good”) or abbreviating them (“thinkin’ ” for “thinking”). This is also seen in the Japanese language, which has standard word forms and variants of them that are often used in social media texts. To take one word as an example, the standard form is ���� (oishii, “It is delicious”) and its variants include ���� ���(oishiiiii), ��� �(oishii), and ��� �(oishii), where the un- derlined characters are the differences from the standard form. Such non-standard tokens often degrade the accuracy of existing language processing systems, which are trained using a clean corpus. Almost all text normalization tasks for languages other than Japanese (e.g., English), aim to replace the non-standard tokens that are explicitly segmented using the context-appropriate standard words (Han et al. (2012), Han and Baldwin (2011), Hassan and Menezes (2013), Li and Liu (2012), Liu et al. (2012), Liu et al. (2011), Pennell and Liu (2011), Cook and Stevenson (2009), Aw et al. (2006)). On the other hand, the problem is more complicated in Japanese morphological analysis because Japanese words are not segmented by explicit delimiters. In traditional Japanese morphological analysis, word segmentation and part-of-speech (POS) tagging are simultaneously estimated. Therefore, we have to simultaneously analyze normalization, word segmentation, and POS tagging to estimate the normalized form using the context information. For example, the input ����������(pan-keiki oishiiii, “This pancake tastes good”) written in the standard form is ��������� (pan-keiki oishii). …

Content truncated to 500 words