Automatic Construction of a Chinese Electronic Dictionary

Automatic Construction of a Chinese Electronic Dictionary +Jing-Shin Chang, ÷Yi-Chung Lin and ÷*Keh-Yih Su { shin,lyc } @hermes.ee.nthu.edu.tw, kysu@bdc.com.tw +Department of Electrical Engineering Natural Language Processing Lab National Tsing-Hua University Hsinchu, Taiwan 30043, ROC *Behavior Design Corporation 2F, No.5, Industrial East Road IV Science-Based Industrial Park Hsinchu, Taiwan 30077, ROC ABSTRACT In this paper, an unsupervised approach for constructing a large-scale Chinese electronic dictionary is surveyed. The main purpose is to enable cheap and quick acquisition of a large-scale dictionary from a large untagged text corpus with the aid of the information in a small tagged seed corpus. The basic model is based on a Viterbi reestimation technique. During the dictionary construction process, it tries to optimize the automatic segmentation and tagging process by repeatedly refining the set of parameters of the underlying language model. The refined parameters are then used to furtherget a better tagging result. In addition, a two-class classifier, which is capable of classifying an n-gram either as a word or a non-word, is used in combination with the Viterbi training module to improve the system performance. Two different system configurations had been developed to construct the dictionary. The configurations include (1) a Viterbi word identification module followed by a Viterbi POS tagging module and (2) a two-class classification module as the postfilter for the above Viterbi word identification module. With a seed of 1,000 sentences and an untagged corpus of 311,591 sentences, the performance for bigram word identification is 56.88% in precision and 77.37% in recall when the two-class classifier is applied to the word list suggested by the Viterbi word identification module. The Viterbi part of speech tag reestimation stage gives the figures of 71.16% and 71.81% weighted precision rates and 73.42% and 73.83% weighted recall rates for the 2 different configurations when using a seed corpus of 9676 sentences. 1. Introduction and System Overview A large-scale electronic dictionary is the fundamental component to many natural language and spoken language :processing applications such as spelling correction, grammar checking, text-speech conversion, intelligent Chinese input methods and machine translation. However, a large electronic dic- tionary for natural language processing may not be available. This is true for the current Chinese language processing community. One possible way is to convert a general dictionary into its electronic form. Firstly, however, a general dictionary may not be updated frequently to reflect the current status of language uses, and many new lexicon entries may not be available. Secondly, a general dictionary may be lack of certain field-specific terms such that the language processing system cannot make use of such unregistered words for a domain-specific application. Even with the term registered, it may not have the special syntactic or semantic annotations for a particular domain. For instance, a machine translation system for translating computer manuals may need to update its lexicon frequently to catch up with the constantly changing computer technologies. In this case, a general dictionary may not provide significant help in the particular domain. Furthermore, the number of lexical entries …

Content truncated to 500 words