Adaptive Chinese Word Segmentation

Adaptive Chinese Word Segmentation 1 Jianfeng Gao*, Andi Wu*, Mu Li*, Chang-Ning Huang*, Hongqiao Li**, Xinsong Xia$, Haowei Qin& * Microsoft Research. {jfgao, andiwu, muli, cnhuang}@microsoft.com ** Beijing Institute of Technology, Beijing. lhqtxm@bit.edu.cn $ Peking University, Beijing. xia_xinsong@founder.com & Shanghai Jiaotong university, Shanghai. haoweiqin@sjtu.edu.cn 1 This work was done while Hongqiao Li, Xinsong Xia and Haowei Qin were visiting Microsoft Research (MSR) Asia. We thank Xiaodan Zhu for his early contribution, and the three reviewers, one of whom alerted us the related work of (Uchimoto et al., 2001). Abstract This paper presents a Chinese word segmen- tation system which can adapt to different domains and standards. We first present a sta- tistical framework where domain-specific words are identified in a unified approach to word segmentation based on linear models. We explore several features and describe how to create training data by sampling. We then describe a transformation-based learning method used to adapt our system to different word segmentation standards. Evaluation of the proposed system on five test sets with dif- ferent standards shows that the system achieves state- of-the-art performance on all of them. 1 Introduction Chinese word segmentation has been a long- standing research topic in Chinese language proc- essing. Recent development in this field shows that, in addition to ambiguity resolution and unknown word detection, the usefulness of a Chinese word segmenter also depends crucially on its ability to adapt to different domains of texts and different segmentation standards. The need of adaptation involves two research issues that we will address in this paper. The first is new word detection. Different domains/applications may have different vocabularies which contain new words/terms that are not available in a general dictionary. In this paper, new words refer to OOV words other than named entities, factoids and mor- phologically derived words. These words are mostly domain specific terms (e.g. 蜂窝式 ‘cellular’) and time-sensitive political, social or cultural terms (e.g. 三通‘Three Links’, 非典 ‘SARS’). The second issue concerns the customizable display of word segmentation. Different Chinese NLP-enabled applications may have different re- quirements that call for different granularities of word segmentation. For example, speech recogni- tion systems prefer “longer words” to achieve higher accuracy whereas information retrieval systems prefer “shorter words” to obtain higher recall rates, etc. (Wu, 2003). Given a word seg- mentation specification (or standard) and/or some application data used as training data, a segmenter with customizable display should be able to provide alternative segmentation units according to the specification which is either pre-defined or implied in the data. In this paper, we first present a statistical framework for Chinese word segmentation, where various problems of word segmentation are solved simultaneously in a unified approach. Our ap- proach is based on linear models where component models are inspired by the source-channel models of Chinese sentence generation. We then describe in detail how the new word identification (NWI) problem is handled in this framework. We explore several features and describe how to create training data by sampling. We evaluate the performance of …

Content truncated to 500 words