The Open Lexicon Interchange Format (OLIF) comes of age

Using Multiple Edit Distances to Automatically Rank Machine Translation Output Yasuhiro Akiba, Kenji Imamura, and Eiichiro Sumita ATR Spoken Language Translation Research Laboratories 2-2-2 Hikaridai Seikacho Soraku-gun Kyoto 619-0288 Japan {yakiba, kimamura, sumita}@slt.atr.co.jp Abstract This paper addresses the challenging problem of automatically evaluating output from machine translation (MT) systems in order to support the developers of these systems. Conventional approaches to the problem include methods that automatically assign a rank such as A, B, C, or D to MT output according to a single edit distance between this output and a correct translation example. The single edit distance can be differently designed, but changing its design makes assigning a certain rank more accurate, but another rank less accurate. This inhibits improving accuracy of rank assignment. To overcome this obstacle, this paper proposes an automatic ranking method that, by using multiple edit distances, encodes machine-translated sentences with a rank assigned by humans into multi-dimensional vectors from which a classifier of ranks is learned in the form of a decision tree (DT). The proposed method assigns a rank to MT output through the learned DT. The proposed method is evaluated using transcribed texts of real conversations in the travel arrangement domain. Experimental results show that the proposed method is more accurate than the single-edit-distance-based ranking methods, in both closed and open tests. Moreover, the proposed method could estimate MT quality within 3% error in some cases. Keywords: automatic evaluation, machine translation system, multiple edit distances, decision trees, machine learning 1. Introduction ATR has been developing the transfer-driven machine translation system, TDMT (Furuse et al., 1995; Furuse & Iida 1996; Sumita et al., 1999), which is used as a subsystem in a multi-lingual speech translation system called ATR’s multilingual automatic translation system for information exchange, ATR-MATRIX (Takezawa et al., 1999; Yamamoto, 2000). Both TDMT and ATR- MATRIX are designed for the travel arrangement domain. TDMT have been subjectively evaluated with four ranks: A, B, C, and D (Sumita et al., 1999). The four ranks are defined as follows: (A) Perfect: no problems in both information and grammar; (B) Fair: easy-to- understand, with either some unimportant information missing or flawed grammar; (C) Acceptable: broken, but understandable with effort; (D) Nonsense: important information has been translated incorrectly. Machine- translated sentences have been manually ranked by native speakers of target languages. Such subjective evaluation by ranking, however, is taxing on both time and resources (King, 1996). The developers of TDMT would like to evaluate their MT system under development more frequently; therefore if automatic evaluation methods are inexpensive, fast, sufficiently accurate for them to assess whether or not the current version of their MT system is improved, then these automatic evaluation methods will prove beneficial. Conventional approaches to automatic evaluation include methods (Thompson, 1991; Su, 1992; Takezawa et al., 1999; Sugaya et al., 1999; Yasuda et al., 2000; Yasuda et al., 2001) that automatically assign one of several ranks (Sumita et al., 1999; Nagao & Tsujii, 1985) such as A, B, C, and D to MT …

Content truncated to 500 words