A Novel Approach for Handling Unknown Word Problem in Chinese-Vietnamese Machine Translation

Computational Linguistics and Chinese Language Processing Vol. 19, No. 1, March 2014, pp. 1-10 1  The Association for Computational Linguistics and Chinese Language Processing A Novel Approach for Handling Unknown Word Problem in Chinese-Vietnamese Machine Translation Phuoc Tran, and Dien Dinh Abstract For languages where space cannot be a boundary of a word, such as Chinese and Vietnamese, word segmentation is always the task to be done first in a statistical machine translation system (SMT). The word segmentation increases the translation quality, but it causes many unknown words (UKW) in the target translation. In this paper, we will present a novel approach to translate UKW. Based on the meaning relationship between Chinese and Vietnamese, we built a model which based on the meaning of the characters forming the UKW before translating the UKW through the model. Experiments show that our method significantly improved the performance of SMT. Keywords: Chinese-Vietnamese SMT, Unknown Word, Sino-Vietnamese, Pure-Vietnamese, SVBUT Model, PVBUT Model. 1. Introduction Unlike Western languages (typically English), Chinese and Vietnamese words are not separated by a space. A Chinese sentence consists of a series of characters, including punctuation, and no spaces between the characters. In Vietnamese, the spelled words (one-syllabled word) are separated by only one space, and the punctuation is located after the spelled words. Therefore, word segmentation is always solved first in Chinese or Vietnamese statistical machine translation (SMT) into other languages. The word segmentation increases the translation quality but generates many unknown words (UKW). A Chinese word usually includes many meaningful characters; when translating it into Vietnamese, its meaning is usually divided into three cases. The first case is where the meanings of Chinese characters are their Sino-Vietnamese meanings, usually a 1-1 correspondence. The second case is where the meanings of the Chinese characters are similar  Faculty of Information Technology, University of Food Industry, Ho Chi Minh City, Vietnam E-mail: phuoctt@cntp.edu.vn  Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam E-mail: ddien@fit.hcmus.edu.vn 2 Phuoc Tran & Dien Dinh or related to the meaning of the Chinese word containing those characters. The final case is where the meanings of Chinese characters are not relevant to the meaning of the Chinese word containing them. In the first case, Vietnamese words largely are borrowed from Chinese words (often called Sino-Vietnamese, which make up about 65% of the total number of Vietnamese words). Thus, the Sino-Vietnamese words generally appear in Vietnamese text. This very important feature is the basis for our handling UKW approach. In the second case, the meaning of the Chinese word is a combination of Pure-Vietnamese meanings of Chinese characters that form the Chinese word. For these two cases, we re-split a Chinese UKW into characters and translate the characters into Sino-Vietnamese or Pure-Vietnamese. Then, we proceed to incorporate the meanings of the characters and filter their meanings to be suitable to Vietnamese meaning. In the final case, the meaning of Chinese word is not related to the meanings of the characters forming them. …

Content truncated to 500 words