A Maximum Entropy Chinese Character-Based Parser

A Maximum Entropy Chinese Character-Based Parser Xiaoqiang Luo 1101 Kitchawan Road, 23-121 IBM T.J. Watson Research Center Yorktown Heights, NY 10598 xiaoluo@us.ibm.com Abstract The paper presents a maximum entropy Chinese character-based parser trained on the Chinese Treebank (“CTB” hence- forth). Word-based parse trees CTB are first converted into character-in based trees, where word-level part-of- speech (POS) tags become constituent labels and character-level tags are de- rived from word-level POS tags. maximum entropy parser is then trainedA on the character-based corpus. The parser does word-segmentation, POS- tagging and parsing in a unified frame- work. An average label F-measure ����� and word-segmentation F-measure � � � are achieved by the parser. Our re- sults show that word-level POS tags can improve significantly word-segmentation, but higher-level syntactic strutures are of little use to word segmentation in the max- imum entropy parser. A word-dictionary helps to improve both word-segmentation and parsing accuracy. 1 Introduction: Why Parsing Characters? After Linguistic Data Consortium (LDC) re- leased the Chinese Treebank (CTB) developed at UPenn (Xia et al., 2000), various statistical Chinese parsers (Bikel and Chiang, 2000; Xu et al., 2002) have been built. Techniques used in parsing En- glish have been shown working fairly well when ap- plied to parsing Chinese text. As there is no word boundary in written Chinese text, CTB is manually segmented into words and then labeled. Parsers de- scribed in (Bikel and Chiang, 2000) and (Xu et al., 2002) operate at word-level with the assumption that input sentences are pre-segmented. The paper studies the problem of parsing Chi- nese unsegmented sentences. The first motivation is that a character-based parser can be used directly in natural language applications that operate at char- acter level, whereas a word-based parser requires a separate word-segmenter. The second and more important reason is that the availability of CTB, a large corpus with high quality syntactic annota- tions, provides us with an opportunity to create a highly-accurate word-segmenter. It is widely known that Chinese word-segmentation is a hard problem. There are multiple studies (Wu and Fung, 1994; Sproat et al., 1996; Luo and Roukos, 1996) show- ing that the agreement between two (untrained) na- tive speakers is about upper  � to lower �� � . The agreement between multiple human subjects is even lower (Wu and Fung, 1994). The rea- son is that human subjects may differ in segment- ing things like personal names (whether family and given names should be one or two words), num- ber and measure units and compound words, al- though these ambiguities do not change a human being’s understanding of a sentence. Low agree- ment between humans affects directly evaluation of machines’ performance (Wu and Fung, 1994) as it is hard to define a gold standard. It does not nec- essarily imply that machines cannot do better than humans. Indeed, if we train a model with consis- tently segmented data, a machine may do a bet- ter job in “remembering” word segmentations. As will be shown shortly, it is straightforward to en- …

Content truncated to 500 words