Error-Driven Learning of Chinese Word Segmentation

Language, Information and Computation (PACLIC12), 18-20 Feb, 1998, 218-229 Error-Driven Learning of Chinese Word Segmentation Julia Hockenmaier* Centre for Cognitive Science University of Edinburgh and Institut fur maschinelle Sprachverarbeitung Universitat Stuttgart julia@cogsci . ed. ac .uk Chris Brewt Language Technology Group Human Communication Research Centre University of Edinburgh Chris.Brew@ed.ac.uk Abstract Palmer ([4]) demonstrated how Brill's Transformation-based Error-Driven Learning can be applied to word segmentation in various languages. We present experimental results which show that such al- gorithms can achieve satisfactory performance even with a a very dimple initial state annotator We also present two preliminary studies, which suggest that even higher performance might be achieved if simple morphological information is available to the system, and that segmentation performance might actually be improved by combining segmentation with rudimentary part-of-speech tagging. 1 Introduction Chinese word segmentation is an interesting, but difficult problem. The difficulties include the following: - "word" is not a very well-defined concept in the context of Chinese: linguists do not have generally accepted guidelines, and in experiments native speakers show only about 75 % agreement on the "correct" segmentation. - Even if we have guidelines, the problem does not become trivial. The biggest stumbling-blocks for any automatic segmenter are unseen words which occur neither in the lexicon nor in the training data. Previous work in this area tended to use some combination of stochastic methods and lexical heuristics. Stochastic methods provide robustness and trainability at the expense of compactness and perspicuity in the learnt models, while lexical heuristics tend to be labour intensive and highly corpus-specific But Palmer ([4]) has recently applied Brill's Transformation-based Error-Driven Learning to the segmentation problem, with promising results. The models which are produced are both more compact and easier to interpret than stochastic models, yet produce a performance which is comparable to that of the best hand-crafted systems. *The first author was supported by a grant from the Studienstiftung des deutschen Volkes. t The second author was partly supported by a grant from ESRC to the Human Communication Research Centre. Correspondence about this paper should be sent to Chris . Brew@ed. ac . uk. 218 Language, Information and Computation (PACLIC12), 18-20 Feb, 1998, 218-229 Our work also uses Brill's technique. We extend Palmer's work by showing some effects of variation in corpus size and rule complexity. We present experiments which achieve satisfactory performance even with a very simple initial state tagger. A major reason for this success is the large size of the training corpus which was employed. Segmentation-driven retagging Two preliminary studies, reported at the end of this paper, indicate that segmentation performance might actually be improved by simultaneously addressing the task of assigning tags to the text. The benefit of this apparently paradoxical move comes because rules learnt by the system can use the tag layer to describe the context in a richer way than is available in the simpler system. This information can then be exploited by later rules. But while our system assigns part-of-speech tags, it does so only in …

Content truncated to 500 words