Using a Mixture of N-Best Lists from Multiple MT Systems in Rank-Sum-Based Confidence Measure for MT Outputs

Using a Mixture of N-Best Lists from Multiple MT Systems in Rank-Sum-Based Confidence Measure for MT Outputs ∗ Yasuhiro Akiba†,‡, Eiichiro Sumita†, Hiromi Nakaiwa†, Seiichi Yamamoto†, and Hiroshi G. Okuno‡ † ATR Spoken Language Translation Research Laboratories 2-2-2 Hikaridai, Keihana Science City, Kyoto 619-0288, Japan ‡ Graduate School of Informatics, Kyoto University Yoshida-Honmachi, Sakyo-ku, Kyoto 606-8501, Japan {yasuhiro.akiba, eiichiro.sumita, hiromi.nakaiwa seiichi.yamamoto}@atr.jp, and okuno@i.kyoto-u.ac.jp Abstract This paper addressees the problem of eliminat- ing unsatisfactory outputs from machine trans- lation (MT) systems. The authors intend to eliminate unsatisfactory MT outputs by using confidence measures. Confidence measures for MT outputs include the rank-sum-based confi- dence measure (RSCM) for statistical machine translation (SMT) systems. RSCM can be ap- plied to non-SMT systems but does not always work well on them. This paper proposes an alternative RSCM that adopts a mixture of the N-best lists from multiple MT systems instead of a single-system’s N-best list in the exist- ing RSCM. In most cases, the proposed RSCM proved to work better than the existing RSCM on two non-SMT systems and to work as well as the existing RSCM on an SMT system. 1 Introduction This paper addresses the challenging problem of eliminating unsatisfactory outputs from machine translation (MT) systems, which are subsystems of a speech-to-speech machine translation (S2SMT) system. The permissible range of translation quality by MT/S2SMT systems depends on the user. Some users permit only perfect translations, while other users permit even translations with flawed grammar. Unsatisfactory MT outputs are those whose transla- tion quality is worse than the level the user can per- mit. In this paper, the authors intend to eliminate un- satisfactory outputs by using confidence measures for MT outputs. The confidence measures1 indicate how perfect/satisfactory the MT outputs are. In the ∗ This research was supported in part by the Ministry of Public Management, Home Affairs, Posts and Telecommunications, Japan. 1These confidence measures are a kind of automatic evalu- ator such as mWER (Niessen et al., 2000) and BLEU (Papineni et al., 2001). While mWER and BLEU cannot be used online, these confidence measures can. This is because the former are based on reference translations, while the latter is not. discipline of MT, confidence measures for MT out- puts have rarely been investigated. The few existing confidence measures include the rank-sum-based confidence measure (RSCM) for statistical machine translation (SMT) systems, Crank in (Ueffing et al., 2003). The basic idea of this confidence measure is to roughly calculate the word posterior probability by using ranks of MT outputs in an N-best list from an SMT system. In the discipline of non-parametric statistical test, ranks of numerical values are commonly used in- stead of the numerical values themselves for statis- tical tests. In the case of the existing RSCM, the ranks of probabilities of MT outputs in the N-best list were used instead of the probabilities of the out- puts themselves. The existing RSCM scores each word in an MT output by summing the comple- mented ranks of candidates in the N-best list …

Content truncated to 500 words