A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 56–64, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information Xu Sun Department of Computer Science University of Tokyo sunxu@is.s.u-tokyo.ac.jp Yaozhong Zhang Department of Computer Science University of Tokyo yaozhong.zhang@is.s.u-tokyo.ac.jp Takuya Matsuzaki Department of Computer Science University of Tokyo matuzaki@is.s.u-tokyo.ac.jp Yoshimasa Tsuruoka School of Computer Science University of Manchester yoshimasa.tsuruoka@manchester.ac.uk Jun’ichi Tsujii Department of Computer Science, University of Tokyo, Japan School of Computer Science, University of Manchester, UK National Centre for Text Mining, UK tsujii@is.s.u-tokyo.ac.jp Abstract Conventional approaches to Chinese word segmentation treat the problem as a character- based tagging task. Recently, semi-Markov models have been applied to the problem, in- corporating features based on complete words. In this paper, we propose an alternative, a latent variable model, which uses hybrid in- formation based on both word sequences and character sequences. We argue that the use of latent variables can help capture long range dependencies and improve the recall on seg- menting long words, e.g., named-entities. Ex- perimental results show that this is indeed the case. With this improvement, evaluations on the data of the second SIGHAN CWS bakeoff show that our system is competitive with the best ones in the literature. 1 Introduction For most natural language processing tasks, words are the basic units to process. Since Chinese sen- tences are written as continuous sequences of char- acters, segmenting a character sequence into a word sequence is the first step for most Chinese process- ing applications. In this paper, we study the prob- lem of Chinese word segmentation (CWS), which aims to find these basic units (words1) for a given sentence in Chinese. Chinese character sequences are normally am- biguous, and out-of-vocabulary (OOV) words are a major source of the ambiguity. Typical examples of OOV words include named entities (e.g., orga- nization names, person names, and location names). Those named entities may be very long, and a dif- ficult case occurs when a long word W (|W| ≥ 4) consists of some words which can be separate words on their own; in such cases an automatic segmenter may split the OOV word into individual words. For example, (Computer Committee of International Federation of Automatic Control) is one of the organization names in the Microsoft Research corpus. Its length is 13 and it contains more than 6 individual words, but it should be treated as a single word. Proper recogni- tion of long OOV words are meaningful not only for word segmentation, but also for a variety of other purposes, e.g., full-text indexing. However, as is il- lustrated, recognizing long words (without sacrific- ing the performance on short words) is challenging. Conventional approaches to Chinese word seg- mentation treat the problem as a character-based la- 1Following previous work, in this paper, words can also refer to multi-word expressions, including proper names, long named entities, idioms, etc. 56 beling task (Xue, 2003). Labels are assigned …

Content truncated to 500 words