Chinese Informal Word Normalization: an Experimental Study

Chinese Informal Word Normalization: an Experimental Study Aobo Wang1∗, Min-Yen Kan1,2†, Daniel Andrade3, Takashi Onishi3, Kai Ishikawa3 1 Web IR / NLP Group (WING) 2 Interactive and Digital Media Institute (IDMI) National University of Singapore {wangaobo,kanmy}@comp.nus.edu.sg 3 Knowledge Discovery Research Laboratories NEC Corporation, Nara, Japan {s-andrade@cj,t-onishi@bq,k-ishikawa@dq}.jp.nec.com Abstract We study the linguistic phenomenon of informal words in the domain of Chi- nese microtext and present a novel method for normalizing Chinese informal words to their formal equivalents. We formal- ize the task as a classification problem and propose rule-based and statistical fea- tures to model three plausible channels that explain the connection between for- mal and informal pairs. Our two-stage selection-classification model is evaluated on a crowdsourced corpus and achieves a normalization precision of 89.5% across the different channels, significantly im- proving the state-of-the-art. 1 Introduction Microtext – including microblogs, comments, SMS, chat and instant messaging (collectively re- ferred to as microtext by Gouwset et al. (2011) or network informal language by Xia et al. (2005)) – is receiving a larger research focus from the com- putational linguistic community. A key challenge is the presence of informal words – terms that manifest as ad hoc abbreviations, neologisms, un- conventional spellings and phonetic substitutions. This phenomenon is so prevalent a challenge in Chinese microtext that the dual problems of infor- mal word recognition and normalization deserve research. Given the close connection between an informal word and its formal equivalent, the restoration (normalization) of an informal word to its formal one is an important pre-processing step for NLP tasks that rely on string matching or word frequency statistics (Han et al., 2012). ∗This research is done in part during Aobo Wang’s in- ternship in NEC Corporation. †This research is supported by the Singapore National Research Foundation under its International Research Centre @ Singapore Funding Initiative and administered by the IDM Programme Office. It is important to note that simply re-training models trained on formal text or annotated mi- crotext is insufficient: user-generated micro- texts exhibit markedly different orthographic and syntactic constraints compared to their formal equivalents. For example, consider the infor- mal microtext “³ù>�” (formally, “Œ�> �”;“harmonious society”). A machine translation system may mistranslate it literally as “crab com- munity” based on the meaning of its component words, if it lacks knowledge of the informal word “³ù” (“Œ�” ; “harmonious”). It is thus desir- able to normalize informal words to their standard formal equivalents before proceeding with stan- dard text processing workflows. In this work, we present a novel method for normalizing informal word to their formal equiv- alents. Specifically, given an informal word with its context as input, we generate hypotheses for its formal equivalents by searching the Google Web 1T corpus (Brants and Franz, 2006). Prospec- tive informal–formal pairs are further classified by a supervised binary classifier to identify cor- rect pairs. In the classification model, we incor- porate both rule-based and statistical feature func- tions that are learned from both gold-standard an- notation and formal domain synonym dictionaries. Also importantly, …

Content truncated to 500 words