TBL-Improved Non-Deterministic Segmentation and POS Tagging for a Chinese Parser

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 264–272, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics TBL-Improved Non-Deterministic Segmentation and POS Tagging for a Chinese Parser Martin Forst & Ji Fang Intelligent Systems Laboratory Palo Alto Research Center Palo Alto, CA 94304, USA {mforst|fang}@parc.com Abstract Although a lot of progress has been made recently in word segmentation and POS tagging for Chinese, the output of cur- rent state-of-the-art systems is too inaccu- rate to allow for syntactic analysis based on it. We present an experiment in im- proving the output of an off-the-shelf mod- ule that performs segmentation and tag- ging, the tokenizer-tagger from Beijing University (PKU). Our approach is based on transformation-based learning (TBL). Unlike in other TBL-based approaches to the problem, however, both obligatory and optional transformation rules are learned, so that the final system can output multi- ple segmentation and POS tagging anal- yses for a given input. By allowing for a small amount of ambiguity in the out- put of the tokenizer-tagger, we achieve a very considerable improvement in accu- racy. Compared to the PKU tokenizer- tagger, we improve segmentation F-score from 94.18% to 96.74%, tagged word F-score from 84.63% to 92.44%, seg- mented sentence accuracy from 47.15% to 65.06% and tagged sentence accuracy from 14.07% to 31.47%. 1 Introduction Word segmentation and tagging are the neces- sary initial steps for almost any language process- ing system, and Chinese parsers are no exception. However, automatic Chinese word segmentation and tagging has been recognized as a very difficult task (Sproat and Emerson, 2003), for the follow- ing reasons: First, Chinese text provides few cues for word boundaries (Xia, 2000; Wu, 2003) and part-of- speech (POS) information. With the exception of punctuation marks, Chinese does not have word delimiters such as the whitespace used in English text, and unlike other languages without whites- paces such as Japanese, Chinese lacks morpholog- ical inflections that could provide cues for word boundaries and POS information. In fact, the lack of word boundary marks and morphological in- flection contributes not only to mistakes in ma- chine processing of Chinese; it has also been iden- tified as a factor for parsing miscues in Chinese children’s reading behavior (Chang et al., 1992). Second, in addition to the two problems de- scribed above, segmentation and tagging also suf- fer from the fact that the notion of a word is very unclear in Chinese (Xu, 1997; Packard, 2000; Hsu, 2002). While the word is an intuitive and salient notion in English, it is by no means a clear notion in Chinese. Instead, for historical reasons, the intuitive and clear notion in Chinese language and culture is the character rather than the word. Classical Chinese is in general mono- syllabic, with each syllable corresponding to an independent morpheme that can be visually ren- dered with a written character. In other words, characters did represent the basic syntactic unit in Classical Chinese, and thus became the sociolog- …

Content truncated to 500 words