Chinese Unknown Word Translation by Subword Re-segmentation

Chinese Unknown Word Translation by Subword Re-segmentation Ruiqiang Zhang1,2 and Eiichiro Sumita1,2 1National Institute of Information and Communications Technology 2ATR Spoken Language Communication Research Laboratories 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0288, Japan {ruiqiang.zhang, eiichiro.sumita}@{nict.go.jp, atr.jp} Abstract We propose a general approach for trans- lating Chinese unknown words (UNK) for SMT. This approach takes advantage of the properties of Chinese word composition rules, i.e., all Chinese words are formed by sequential characters. According to the proposed approach, the unknown word is re-split into a subword sequence followed by subword translation with a subword- based translation model. “Subword” is a unit between character and long word. We found the proposed approach significantly improved translation quality on the test data of NIST MT04 and MT05. We also found that the translation quality was further im- proved if we applied named entity transla- tion to translate parts of unknown words be- fore using the subword-based translation. 1 Introduction The use of phrase-based translation has led to great progress in statistical machine translation (SMT). Basically, the mechanism of this approach is re- alized by two steps:training and decoding. In the training phase, bilingual parallel sentences are pre- processed and aligned using alignment algorithms or tools such as GIZA++ (Och and Ney, 2003). Phrase pairs are then extracted to be a phrase translation ta- ble. Probabilities of a few pre-defined features are computed and assigned to the phrase pairs. The fi- nal outcome of the training is a translation table con- sisting of source phrases, target phrases, and lists of probabilities of features. In the decoding phase, the translation of a test source sentence is made by reordering the target phrases corresponding to the source phrases, and searching for the best hypothesis that yields the highest scores defined by the search criterion. However, this mechanism cannot solve unknown word translation problems. Unknown words (UNK) point to those unseen words in the training or non- existing words in the translation table. One strat- egy to deal with translating unknown words is to re- move them from the target sentence without transla- tion on assumption of fewer UNKs in the test data. Of course, this simple way produces a lower quality of translations if there are a lot of UNKs in the test data, especially for using a Chinese word segmenter that produces many UNKs. The translation of UNKs need to be solved by a special method. The translation of Chinese unknown words seems more difficult than other languages because Chinese language is a non-inflected language. Unlike other languages (Yang and Kirchhoff, 2006; Nießlen and Ney, 2000; Goldwater and McClosky, 2005), Chi- nese UNK translation cannot use information from stem and inflection analysis. Using machine translit- eration can resolve part of UNK translation (Knight and Graehl, 1997). But this approach is effective for translating phonetically related unknown words, not for other types. No unified approach for translating Chinese unknown words has been proposed. In this paper we propose a novel statistics-based approach for unknown word translation. This ap- …

Content truncated to 500 words