Applying Meaningful Word-Pair Identifier to the Chinese Syllable-to-Word Conversion Problem

Applying Meaningful Word-Pair Identifier to the Chinese Syllable-to-Word Conversion Problem Jia-Lin Tsai, Tien-Jien Chiang and Wen-Lian Hsu Institute of Information Science, Academia Sinica, Nankang, Taipei, Taiwan, R.O.C. {tsaijl,tmjiang, hsu}@iis.sinica.edu.tw Abstract. Syllable-to-word (STW) conversion is a frequently used Chinese input method that is fundamental to syllable/speech understanding. The two major problems with STW conversion are the segmentation of syllable input and the ambiguities caused by homonyms. This paper describes a meaningful word-pair (MWP) identifier that can be used to resolve homonym/segmentation ambiguities and perform STW conversion effectively for Chinese language texts. It is designed as a support system with Chinese input systems. In this paper, five types of meaning- ful word-pairs are investigated, namely: noun-verb (NV), noun-noun (NN), verb-verb (VV), adjective-noun (AN) and adverb-verb (DV). The pre-collected datasets of meaningful word-pairs are based on our previous work auto-generation of NVEF knowledge in Chinese (AUTO-NVEF) [30, 32], where NVEF stands for noun-verb event frame. The main purpose of this study is to illustrate that a hybrid approach of combining statistical language model- ing (SLM) with contextual information, such as meaningful word-pairs, is effective for improving syllable-to-word systems and is important for syllable/speech understanding. Our experiments show the following: (1) the MWP identifier achieves tonal (syllables with four tones) and toneless (syllables without four tones) STW accuracies of 98.69% and 90.7%, respectively, among the identified word-pairs for the test syllables; (2) by STW error analysis, we find that the major critical problem of tonal STW systems is the failure of homonym disambiguation (52%), while that of toneless STW systems is inadequate syllable segmentation (48%); (3) by applying the MWP identifier, together with the Microsoft input method editor (MSIME 2003) and an optimized bigram model (BiGram), the to- nal and toneless STW improvements of the two STW systems are 25.25%/21.82% and 12.87%/15.62%, respec- tively. Keywords: syllable-to-word, contextual information, top-down identifier, n-gram model. 1. Introduction More than 100 Chinese input methods have been developed in the past [1, 17, 12, 5, 18, 10, 19, 16, 4, 28, 11, 20]. Their underlying approaches can be classified into four types: (a) Optical character recognition (OCR) based [5], (b) On-line handwriting based [19], (c) Speech based [4, 10], and (d) Keyboard based, such as syllabic-input-to-character [27, 16, 2, 14, 15, 22]; arbitrary codes based [8]; and structure scheme based [11]. The major goal of these syllable input systems is to achieve high STW accu- racy, but syllable understanding is rarely considered [16]. Currently, the most popular method for Chinese input is syllable based (or phonetic/pinyin based), because Chinese people are taught to write the corresponding phonetic/pinyin syllable of each Chinese character in primary school. Basically, each Chinese character corresponds to at least one syllable. Although there are more than 13,000 distinct Chinese characters (of which 5,400 are commonly used), there are only 1,300 distinct syllables. The homonym (homophone) problem is, therefore, quite severe when using a Chinese phonetic input method [5]. As per [26], each Chinese syllable can be mapped from 3 to over 100 Chinese characters, with …

Content truncated to 500 words