Mining Informal Language from Chinese Microtext: Joint Word Recognition and Segmentation

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 731–741, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Mining Informal Language from Chinese Microtext: Joint Word Recognition and Segmentation Aobo Wang1 Min-Yen Kan1,2∗ 1 Web IR / NLP Group (WING) 2 Interactive and Digital Media Institute (IDMI) National University of Singapore 13 Computing Link, Singapore 117590 {wangaobo,kanmy}@comp.nus.edu.sg Abstract We address the problem of informal word recognition in Chinese microblogs. A key problem is the lack of word delimiters in Chinese. We exploit this reliance as an opportunity: recognizing the relation be- tween informal word recognition and Chi- nese word segmentation, we propose to model the two tasks jointly. Our joint in- ference method significantly outperforms baseline systems that conduct the tasks in- dividually or sequentially. 1 Introduction User generated content (UGC) – including mi- croblogs, comments, SMS, chat and instant mes- saging – collectively referred to as microtext by Gouwset et al. (2011) or network informal lan- guage by Xia et al. (2005), is the hallmark of the participatory Web. While a rich source that many applications are interested in mining for knowledge, microtext pro- cessing is difficult to process. One key reason for this difficulty is the ubiquitous presence of informal words – anomalous terms that manifest as ad hoc abbreviations, neologisms, unconven- tional spellings and phonetic substitutions. Such informality is often present in oral conversation, and user-generated microblogs reflect this infor- mality. Natural language processing (NLP) tools largely fail to work properly on microtext, as they have largely been trained on formally written text (i.e., newswire). Recent work has started to ad- dress these shortcomings (Xia and Wong, 2006; Kobus et al., 2008; Han and Baldwin, 2011). In- formal words and their usage in microtext evolves quickly, following social trends and news events. ∗This research is supported by the Singapore National Research Foundation under its International Research Centre @ Singapore Funding Initiative and administered by the IDM Programme Office. These characteristics make it difficult for lexicog- raphers to compile lexica to keep with the pace of language change. We focus on this problem in the Chinese lan- guage. Through our analysis of a gathered Chinese microblog corpus, we observe that Chinese infor- mal words originate from three primary sources, as given in Table 1. But unlike noisy words in English, Chinese in- formal words are more difficult to mechanically recognize for two critical reasons: first, Chinese does not employ word delimiters; second, Chinese informal words combine numbers, alphabetic let- ters and Chinese characters. Techniques for En- glish informal word detection that rely on word boundaries and informal word orthography need to be adapted for Chinese. Consider the micro- text “�•g�†” (meaning “Don’t tell me the spoilers (to a movie or joke)”, also in Table 1). If “�•” (“don’t”) and “ †” (past tense marker) are correctly recognized as two words, we may predict the previously unseen characters “g�” (“tell spoilers”) as an informal word, based on the learned Chinese language patterns. However, …

Content truncated to 500 words