A Novel Schema-Oriented Approach for Chinese New Word Identification

A Novel Schema-Oriented Approach for Chinese New Word Identification Zhao Lu Dept. of Computer Science and Technology, East China Normal University zlu@cs.ecnu.edu.cn Zhixian Yan Samsung Research America, Silicon Valley, USA zhixian.yan@samsung.com Junzhong Gu Dept. of Computer Science and Technology, East China Normal University jzgu@cs.ecnu.edu.cn Abstract With the popularity of network application- s, new words become more common and bring the poor performance of natural lan- guage processing related applications in- cluding web search. Identifying new words automatically from texts is still a very chal- lenging problem, especially for Chinese. In this paper, we propose a novel schema- oriented approach for Chinese new word i- dentification (named “ChNWI”). This ap- proach has three main steps: (1) we suggest three composition schemas that cover nearly all two-character up to four-character Chi- nese word surfaces; (2) we employ support vector machine (SVM) to classify Chinese new words of three schemas using their u- nique linguistic characteristics; and (3) we design various rules to filter identified Chi- nese new words of three schemas. Our ex- tensive evaluations with two corpora (Chi- nese news titles and CIPS-SIGHAN 2012 CSMB) show ChNWI’s efficiency on Chi- nese new word identification. 1 Introduction With the rapid development of information tech- nology, as well as the growth of social networks (e.g., Chinese Microblog, WeChat), Chinese new words are constantly being created and their us- ages have become an inevitable phenomenon. Au- tomatic identification of new words plays an im- portant role in a number of areas in Chinese lan- guage processing, such as automatic segmenta- tion, information retrieval and machine transla- tion (Zhang et al., 2010; Duan et al., 2012). In the Chinese new word identification (NWI) task, new words refer to new composition words that are not registered in the dictionary of a Chinese segmenter. Statistical approaches are the most widely used methods in NWI. The previous methods extrac- t some linguistic features of new word compo- sitions, i.e., word composition probability, co- occurrence probability, mutual information, and word frequency, while they assume above linguis- tic features playing the same impact on various word surfaces (Chang and Lee, 2003; Li et al., 2008; Zhang et al., 2010). Some methods also have binary decision, either ”new words” or ”not new words”. A SVM-based method (Li et al., 2008) aims at two word surfaces, NW11 and N- W21, and the method uses same linguistic features for the two surfaces. Other statistical models, for instance, a latent discriminative model (Pang et al., 2009), a linear-time incremental model (Zhang et al., 2012) and conditional random fields (CRFs) model (Wang et al., 2012), are designed for NWI. Recently, some hybrid methods have been sug- gested. These hybrid methods employ more or fewer rules for statistical methods to obtain an op- timal efficiency of identification. However, the rules these methods used are created by the peo- ple, which cause these methods are not suitable for other new word composition schemas (Zhang et al., 2006; Jiang et al., 2011; Xi et al., 2012). Despite the …

Content truncated to 500 words