A Transition-based Model for Joint Segmentation, POS-tagging and Normalization

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1837–1846, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Transition-based Model for Joint Segmentation, POS-tagging and Normalization Tao Qian1,3, Yue Zhang2, Meishan Zhang2∗, Yafeng Ren1 and Donghong Ji1 1Computer School, Wuhan University, Wuhan, China 2Singapore University of Technology and Design 3College of Computer Science and Technology, Hubei University of Science and Technology, XianNing, China {taoqian, renyafeng, dhji}@whu.edu.cn {yue zhang, meishan zhang}@sutd.edu.sg Abstract We propose a transition-based model for joint word segmentation, POS tagging and text normalization. Different from pre- vious methods, the model can be trained on standard text corpora, overcoming the lack of annotated microblog corpora. To evaluate our model, we develop an anno- tated corpus based on microblogs. Exper- imental results show that our joint model can help improve the performance of word segmentation on microblogs, giving an er- ror reduction in segmentation accuracy of 12.02%, compared to the traditional ap- proach. 1 Introduction Microblogs, such as Twitter, SMS and Weibo, has become an important research topic in NLP. Pre- vious work has shown that off-the-shelf NLP tools can perform poorly on microblogs (Foster et al., 2011; Gimpel et al., 2011; Han and Baldwin, 2011). One of the major challenges for microblog processing is the issue of informal words. For ex- ample, “tmrw” has been frequently used in tweets for “tomorrow”, causing OOV problems. Text normalization has been introduced as a pre-processing step for microblog processing, which transforms informal words into their stan- dard forms. Most work in the literature focuses on English microblog normalization, treating it as a noisy channel problem (Pennell and Liu, 2014; Cook and Stevenson, 2009; Yang and Eisenstein, 2013) or a translation problem (Aw et al., 2006; Contractor et al., 2010; Li and Liu, 2012; Zhang et al., 2014c), and training models based on words. Lack of annotated corpora, text normalization is more challenging for Chinese. Unlike En- glish, Chinese informal words are more difficult ∗corresponding author to mechanically normalize for two main reasons. First, Chinese does not have word delimiters. Second, Chinese informal words manifest diver- sity, such as abbreviations, neologisms, unconven- tional spellings and phonetic substitutions. Intu- itively, there is mutual dependency between Chi- nese word segmentation and normalization, and therefore two tasks should be solved jointly. Wang and Kan (2013) proposed a joint model to process word segmentation and informal word detection. However, text normalization was not included in the joint model. Kaji et al (2014) proposed a joint model for word segmentation, POS tagging and normalization for Japanese Mi- croblogs, which was trained on a partially anno- tated microblog corpus. Their method requires special annotation for text normalization, which can be expensive. In this paper, we propose a joint model for Chi- nese text normalization, word-segmentation and POS tagging, which can be trained using standard segmentation and POS tagging annotation, over- coming the lack of an annotated corpus on Chi- nese microblogs. Our model is based on Zhang and Clark (2010), with an …

Content truncated to 500 words