Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 99–109, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization Nobuhiro Kaji∗† and Masaru Kitsuregawa†‡ ∗National Institute of Information and Communications Technology †Institute of Industrial Science, The University of Tokyo ‡National Institute of Informatics {kaji, kitsure}@tkl.iis.u-tokyo.ac.jp Abstract Microblogs have recently received widespread interest from NLP re- searchers. However, current tools for Japanese word segmentation and POS tagging still perform poorly on microblog texts. We developed an annotated corpus and proposed a joint model for over- coming this situation. Our annotated corpus of microblog texts enables not only training of accurate statistical models but also quantitative evaluation of their performance. Our joint model with lexical normalization handles the orthographic diversity of microblog texts. We con- ducted an experiment to demonstrate that the corpus and model substantially contribute to boosting accuracy. 1 Introduction Microblogs, such as Twitter1 and Weibo2, have re- cently become an important target of NLP tech- nology. Since microblogs offer an instant way of posting textual messages, they have been given increasing attention as valuable sources for such actions as mining opinions (Jiang et al., 2011) and detecting sudden events such as earthquake (Sakaki et al., 2010). However, many studies have reported that cur- rent NLP tools do not perform well on microblog texts (Foster et al., 2011; Gimpel et al., 2011). In the case of Japanese text processing, the most se- rious problem is poor accuracy of word segmen- tation and POS tagging. Since these two tasks are positioned as the fundamental step in the text processing pipeline, their accuracy is vital for all downstream applications. 1https://twitter.com 2https://www.weibo.com 1.1 Development of annotated corpus The main obstacle that makes word segmentation and POS tagging in the microblog domain chal- lenging is the lack of annotated corpora. Because current annotated corpora are from other domains, such as news articles, it is difficult to train models that perform well on microblog texts. Moreover, system performance cannot be evaluated quantita- tively. We remedied this situation by developing an an- notated corpus of Japanese microblogs. We col- lected 1831 sentences from Twitter and manually annotated these sentences with word boundaries, POS tags, and normalized forms of words (c.f., Section 1.2). We, for the first time, present a comprehen- sive empirical study of Japanese word segmenta- tion and POS tagging on microblog texts by us- ing this corpus. Specifically, we investigated how well current models trained on existing corpora perform in the microblog domain. We also ex- plored performance gains achieved by using our corpus for training, and by jointly performing lex- ical normalization (c.f., Section 1.2). 1.2 Joint modeling with lexical normalization Orthographic diversity in microblog texts causes a problem when training a statistical model for word segmentation and POS tagging. Microblog texts frequently contain informal words that are spelled in a non-standard manner, e.g., “oredi (already)”, “b4 (before)”, and “talkin …

Content truncated to 500 words