Automatic Error Analysis Based on Grammatical Questions

Automatic Error Analysis Based on Grammatical Questions* Tomoki Nagase a , Hajime Tsukada b , Katsunori Kotani c , Nobutoshi Hatanaka d , Yoshiyuki Sakamoto e aFujitsu Laboratories, 1-1, Kamikodanaka, 4-chome, Nakahara-ku, Kawasaki, 211-8588, Japan nagase.tomoki@jp.fujitsu.com bNTT Communication Science Laboratories, 2-4, Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan tsukada.hajime@lab.ntt.co.jp cKansai Gaidai University, 16-1 Nakamiyahigashino-cho, Hirakata, Osaka 573-1001, Japan kkotani@kansaigaidai.ac.jp dTokyo University of Information Sciences, 4-1 Onaridai,Wakaba-ku,Chiba,265-8501 Japan hatanaka@rsch.tuis.ac.jp e5-38-7, Akatsutsumi, Setagaya-ku, Tokyo, 156-0044, Japan ysakanlp@ka2.so-net.ne.jp Abstract. The present paper proposes automatic error analysis methods that use patterns representing grammatical check points. Our method is comparable to or slightly outperforms conventional methods for automatic evaluation metrics. Different from the conventional methods, our method enables error analysis for each grammatical check point. While our method does not depend on languages, we experimentally show its validity by using a Japanese-to-Chinese test set. Errors in existing Japanese-to-Chinese translation systems are also analyzed. Keywords: automatic evaluation, machine translation, error analysis 1 Introduction In the machine translation (MT) field, human evaluators subjectively evaluate the results of MT systems from the viewpoints of adequacy and fluency. However, two problems in subjective evaluations have been noted. The first is that the time and cost consumption of such evaluations is high. The second is that the evaluations have poor reproducibility due to the difficulty of reaching agreement on the scoring criteria among the evaluators. Thus, in the past decade, evaluation methods that automatically evaluate MT quality based on its similarity to a reference translation have become common (Papineni et al., 2002; Doddington, 2002; Banerjee and Lavie, 2005; Snover, et al., 2006; Melamed et al., 2007; Isozaki et al., 2010; Birch and Osborne, 2011). Error analysis that investigates the strengths and weaknesses of a translation system is required for developing more accurate translation systems. According to the error analysis, system developers improve the translation rules/dictionaries of rule-based MT systems or add bilingual corpora/dictionaries to statistical MT systems. However, all the evaluation methods mentioned that include human subjective evaluation provide less information for error analysis because they give only one score for each document or sentence. Thus, evaluation methods have been proposed that present questions to human evaluators asking whether a sentence contains * This work was conducted as an activity of AAMT (Asia-Pacific Association for Machine Translation) working group Copyright 2011 by T. Nagase, H. Tsukada, K. Kotani, N. Hatanaka, and Y. Sakamoto 25th Pacific Asia Conference on Language, Information and Computation, pages 206–215 206 grammatical error(s) or other type of errors (Isahara, 1995; Joans et al., 2007; Uchimoto et al., 2007; Nagase et al., 2009). These question-based evaluation methods have several features that enable the provision of new viewpoints for evaluation, the evaluation of grammatical coverage, and the provision of better metrics for evaluation combined with conventional automatic evaluation metrics. Joans et al. (2007) used the Defense Language Proficiency Test (DLPT), which was originally developed to measure human language skills in their method. Their method enables MT quality to be evaluated from the viewpoint of “comprehension”. Isahara (1995) developed the …

Content truncated to 500 words