Very Large-Scale Lexical Resources to Enhance Chinese and Japanese Machine Translation

Very Large-Scale Lexical Resources to Enhance Chinese and Japanese Machine Translation Jack Halpern The CJK Dictionary Institute, Inc. 34-14, 2-chome, Tohoku, Niiza-shi, Saitama 352-0001 JAPAN jack@cjki.org Abstract A major issue in machine translation (MT) applications is the recognition and translation of named entities. This is especially true for Chinese and Japanese, whose scripts present linguistic and algorithmic challenges not found in other languages. This paper discusses some of the major issues in Japanese and Chinese MT, such as the difficulties of translating proper nouns and technical terms, and the complexities of orthographic variation in Japanese. Of special interest are neural machine translation (NMT) systems, which suffer from a serious out-of-vocabulary problem. However, the current architecture of these systems makes it technically challenging for them to alleviate this problem by supporting lexicons. This paper introduces some Very Large-Scale Lexical Resources (VLSLR) consisting of millions of named entities, and argues that the quality of MT in general, and NMT systems in particular, can be significantly enhanced through the integration of lexicons. Keywords: machine translation, lexicon, Chinese, Japanese 1. Introduction A major issue in MT applications is the translation of named entities and technical terms. This is especially true for Chinese and Japanese, whose scripts present linguistic and algorithmic challenges not found in other languages. Some factors that contribute to these difficulties include: 1. The Japanese orthography is highly irregular, requiring advanced capabilities such as cross-script normalization (Halpern, 2008). 2. The morphological complexity of Japanese requires the use of a robust morphological analyzer for segmentation and lemmatization (Brill et al., 2001; Yu et al., 2000). 3. The accurate conversion between Simplified Chinese (SC) and Traditional Chinese (TC) (Halpern and Kerman, 1999). 4. The difficulty of accurately translating POIs (points of interest). These are extremely numerous, difficult to detect, and have an unstable orthography. 5. The large number of technical terms. 6. The lack of comprehensive lexical resources. This paper discusses some of these issues, and introduces Very Large-Scale Lexical Resources (VLSLR) consisting of millions of named entities. It argues that lexicons can enhance the translation accuracy of NMT systems, which currently don't use lexicons. 2. Japanese Orthographic Variants The Japanese orthography is highly irregular. The numerous orthographic variants, which are common and unpredictable, negatively impact recall and pose a major challenge to MT. The variation results from the unpredictable interaction between four scripts: kanji, hiragana, katakana, and Latin. For example, in 金の卵を 産む鶏 'A hen that lays golden eggs,' tamago 'egg' has four variants (卵,玉子,たまご,タマゴ), niwatori 'chicken' has three (鶏,にわとり,ニワトリ), and umu 'give birth to' has two (産む, 生む), which expands to 24 permutations. Algorithmic solutions have no hope of identifying these as instances of the same underlying sentence without support for orthographic disambiguation/normalization. The most important types of orthographic variation in Japanese (Halpern, 2008) are: (1) Okurigana, which are kana endings such as 当たり外れ and 当り外れ for atarihazure. Normalizing okurigana variants, which are numerous and unpredictable, is a major issue. An effective solution is to use an orthographic variants lexicon. (2) Cross-script …

Content truncated to 500 words