A Phonetic-Based Approach to Chinese Chat Text Normalization

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 993–1000, Sydney, July 2006. c⃝2006 Association for Computational Linguistics A Phonetic-Based Approach to Chinese Chat Text Normalization Yunqing Xia, Kam-Fai Wong Department of S.E.E.M. The Chinese University of Hong Kong Shatin, Hong Kong {yqxia, kfwong}@se.cuhk.edu.hk Wenjie Li Department of Computing The Hong Kong Polytechnic University Kowloon, Hong Kong cswjli@comp.polyu.edu.hk Abstract Chatting is a popular communication media on the Internet via ICQ, chat rooms, etc. Chat language is different from natural language due to its anoma- lous and dynamic natures, which renders conventional NLP tools inapplicable. The dynamic problem is enormously trouble- some because it makes static chat lan- guage corpus outdated quickly in repre- senting contemporary chat language. To address the dynamic problem, we pro- pose the phonetic mapping models to present mappings between chat terms and standard words via phonetic transcrip- tion, i.e. Chinese Pinyin in our case. Dif- ferent from character mappings, the pho- netic mappings can be constructed from available standard Chinese corpus. To perform the task of dynamic chat lan- guage term normalization, we extend the source channel model by incorporating the phonetic mapping models. Experi- mental results show that this method is effective and stable in normalizing dy- namic chat language terms. 1 Introduction Internet facilitates online chatting by providing ICQ, chat rooms, BBS, email, blogs, etc. Chat language becomes ubiquitous due to the rapid proliferation of Internet applications. Chat lan- guage text appears frequently in chat logs of online education (Heard-White, 2004), customer relationship management (Gianforte, 2003), etc. On the other hand, wed-based chat rooms and BBS systems are often abused by solicitors of terrorism, pornography and crime (McCullagh, 2004). Thus there is a social urgency to under- stand online chat language text. Chat language is anomalous and dynamic. Many words in chat text are anomalous to natural language. Chat text comprises of ill-edited terms and anomalous writing styles. We refer chat terms to the anomalous words in chat text. The dynamic nature reflects that chat language changes more frequently than natural languages. For example, many popular chat terms used in last year have been discarded and replaced by new ones in this year. Details on these two fea- tures are provided in Section 2. The anomalous nature of Chinese chat lan- guage is investigated in (Xia et al., 2005). Pattern matching and SVM are proposed to recognize the ambiguous chat terms. Experiments show that F-1 measure of recognition reaches 87.1% with the biggest training set. However, it is also disclosed that quality of both methods drops sig- nificantly when training set is older. The dy- namic nature is investigated in (Xia et al., 2006a), in which an error-driven approach is pro- posed to detect chat terms in dynamic Chinese chat terms by combining standard Chinese cor- pora and NIL corpus (Xia et al., 2006b). Lan- guage texts in standard Chinese corpora are used as negative samples and chat text pieces in the NIL corpus as positive …

Content truncated to 500 words