Translation Errors and Incomprehensibility: a Case Study using Machine-Translated Second Language Proficiency Tests

Translation Errors and Incomprehensibility: A Case Study using Machine-Translated Second Language Proficiency Tests Takuya Matsuzaki†‡, Akira Fujita‡, Naoya Todo‡, Noriko H. Arai‡ †Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8603, JAPAN ‡National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8634, JAPAN matuzaki@nuee.nagoya-u.ac.jp, {a-fujita, ntodo, arai}@nii.ac.jp Abstract This paper reports on an experiment where 795 human participants answered to the questions taken from second language proficiency tests that were translated to their native language. The output of three machine translation systems and two different human translations were used as the test material. We classified the translation errors in the questions according to an error taxonomy and analyzed the participants’ response on the basis of the type and frequency of the translation errors. Through the analysis, we identified several types of errors that deteriorated most the accuracy of the participants’ answers, their confidence on the answers, and their overall evaluation of the translation quality. Keywords: Machine Translation, Evaluation, Error Analysis 1. Introduction What level of “quality” is required for machine translation (MT) systems should change depending on who uses them in what situation. If it is used as a web page translator, even fragmentary information will be helpful. However, if it is used as an automatic interpreter on smartphones, it is a totally different type of matter. It is expected to con- vey correct information between two people who do not understand each other’s language, sometimes in critical sit- uations. In other words the measurement of the quality of MT systems should include not only the intrinsic metrics such as accuracy and fluency but also how often a user can complete his/her purpose in using it. In our previous paper, we proposed a light weight, human- in-the-loop extrinsic evaluation scheme (Matsuzaki et al., 2015), where MT systems are evaluated by human subjects’ scores on the second language ability tests translated to the subjects’ native language by MT. Specifically, we used dialogue completion questions (Figure 1) as the test ma- terial. Dialogues often involve linguistic phenomena that are not frequent in written text, such as interrogatory sen- tences, imperative sentences, and ellipsis. It turned out that our evaluation captured a different dimension of translation quality than that captured by manual and automatic intrin- sic evaluation. In this paper, we report on another experiment involving 795 human participants, and scrutinize what kinds of errors hinder the users’ comprehension of the dialogues more of- ten than others. Although MT-mediated conversation is al- ready in the scope of development, automatic metrics such as BLEU (Papineni et al., 2002) and NIST (Doddington, 2002) do not tell us which errors are more harmful than oth- ers. The translation errors in the test materials were man- ually classified according to an error taxonomy. Then, the subjects’ responses on the translated questions were ana- lyzed on the basis of the type and frequency of the trans- lation errors. Through the analysis, we identified several types of errors that deteriorated most the subjects’ perfor- mance, their confidence on the answers, and the overall …

Content truncated to 500 words