Exploiting Parallel Corpus for Handling Out-of-Vocabulary Words

Exploiting Parallel Corpus for Handling Out-of-vocabulary Words Juan Luo IPS Waseda University Fukuoka, Japan juan.luo@suou.waseda.jp John Tinsley CNGL Dublin City University Dublin, Ireland jtinsley@computing.dcu.ie Yves Lepage IPS Waseda University Fukuoka, Japan yves.lepage@waseda.jp Abstract This paper presents a hybrid model for han- dling out-of-vocabulary words in Japanese- to-English statistical machine translation output by exploiting parallel corpus. As the Japanese writing system makes use of four different script sets (kanji, hira- gana, katakana, and romaji), we treat these scripts differently. A machine translitera- tion model is built to transliterate out-of- vocabulary Japanese katakana words into English words. A Japanese dependency structure analyzer is employed to tackle out- of-vocabulary kanji and hiragana words. The evaluation results demonstrate that it is an effective approach for addressing out-of- vocabulary word problems and decreasing the OOVs rate in the Japanese-to-English machine translation tasks. 1 Introduction Phrase-based statistical machine translation sys- tems rely on parallel corpora for learning trans- lation rules and phrases, which are stored in “phrase tables”. Words that cannot be found in phrase tables thus result in out-of-vocabulary words (OOVs) for a machine translation system. The large number of loanwords and orthographic variants in Japanese makes the OOVs problem more severe than in other languages. As stated in (Oh et al., 2006), most of out-of-vocabulary words in translations from Japanese are made up of proper nouns and technical terms, which are phonetically transliterated from other languages. In addition, the highly irregular Japanese orthog- raphy as is analyzed in (Halpern, 2002) poses a challenge for machine translation tasks. Japanese is written in four different sets of scripts: kanji, hiragana, katakana, and ro- maji (Halpern, 2002). Kanji is a logographic system consisting of characters borrowed from the Chinese characters. Hiragana is a syl- labary system used mainly for functional ele- ments. Katakana is also a syllabary system. Along with hiragana, they are generally referred as kana. Katakana is used to write new words or loan words, i.e., words that are borrowed and translit- erated from foreign languages. Romaji is just the Latin alphabet. In this paper, we present a method of tack- ling out-of-vocabulary words to improve the per- formance of machine translation. This method consists of two components. The first compo- nent relies on a machine transliteration model for katakana words that is based on the phrase-based machine translation framework. Furthermore, by making use of limited resources, i.e., the same parallel corpus used to build the machine trans- lation system, a method of automatically acquir- ing bilingual word pairs for transliteration training data from this parallel corpus is used. With these enriched bilingual pairs, the transliteration model is further improved. In the second component, a Japanese dependency structure analyzer is used to build a kanji-hiragana system for handling ortho- graphic variants. The structure of the paper is as follows. Sec- tion 2 reviews related works. In Section 3, we present a back-transliteration model which is based on the SMT framework for handling katakana OOV words. Section 4 describes a method of tackling kanji and hiragana …

Content truncated to 500 words