Chinese Segmentation and New Word Detection using Conditional Random Fields

Chinese Segmentation and New Word Detection using Conditional Random Fields Fuchun Peng, Fangfang Feng, Andrew McCallum Computer Science Department, University of Massachusetts Amherst 140 Governors Drive, Amherst, MA, U.S.A. 01003 {fuchun, feng, mccallum}@cs.umass.edu Abstract Chinese word segmentation is a difficult, im- portant and widely-studied sequence modeling problem. This paper demonstrates the abil- ity of linear-chain conditional random fields (CRFs) to perform robust and accurate Chi- nese word segmentation by providing a prin- cipled framework that easily supports the in- tegration of domain knowledge in the form of multiple lexicons of characters and words. We also present a probabilistic new word detection method, which further improves performance. Our system is evaluated on four datasets used in a recent comprehensive Chinese word seg- mentation competition. State-of-the-art perfor- mance is obtained. 1 Introduction Unlike English and other western languages, many Asian languages such as Chinese, Japanese, and Thai, do not delimit words by white-space. Word segmentation is therefore a key precursor for lan- guage processing tasks in these languages. For Chi- nese, there has been significant research on find- ing word boundaries in unsegmented sequences (see (Sproat and Shih, 2002) for a review). Un- fortunately, building a Chinese word segmentation system is complicated by the fact that there is no standard definition of word boundaries in Chinese. Approaches to Chinese segmentation fall roughly into two categories: heuristic dictionary-based methods and statistical machine learning methods. In dictionary-based methods, a predefined dictio- nary is used along with hand-generated rules for segmenting input sequence (Wu, 1999). However these approaches have been limited by the impos- sibility of creating a lexicon that includes all pos- sible Chinese words and by the lack of robust sta- tistical inference in the rules. Machine learning ap- proaches are more desirable and have been success- ful in both unsupervised learning (Peng and Schuur- mans, 2001) and supervised learning (Teahan et al., 2000). Many current approaches suffer from either lack of exact inference over sequences or difficulty in in- corporating domain knowledge effectively into seg- mentation. Domain knowledge is either not used, used in a limited way, or used in a complicated way spread across different components. For example, the N-gram generative language modeling based ap- proach of Teahan et al (2000) does not use domain knowledge. Gao et al (2003) uses class-based lan- guage for word segmentation where some word cat- egory information can be incorporated. Zhang et al (2003) use a hierarchical hidden Markov Model to incorporate lexical knowledge. A recent advance in this area is Xue (2003), in which the author uses a sliding-window maximum entropy classifier to tag Chinese characters into one of four position tags, and then covert these tags into a segmentation using rules. Maximum entropy models give tremendous flexibility to incorporate arbitrary features. How- ever, a traditional maximum entropy tagger, as used in Xue (2003), labels characters without considering dependencies among the predicted segmentation la- bels that is inherent in the state transitions of finite- state sequence models. Linear-chain conditional random fields (CRFs) (Lafferty et …

Content truncated to 500 words