Revisit Automatic Error Detection for Wrong and Missing Translation – A Supervised Approach

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 942–952, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 942 Revisit Automatic Error Detection for Wrong and Missing Translation – A Supervised Approach Wenqiang Lei1, 2, Weiwen Xu2, Ai Ti Aw2, Yuanxin Xiang2, Tat-Seng Chua1 1National University of Singapore, Singapore 2Institute for Infocomm Research, A*STAR, Singapore wenqianglei@gmail.com {xuw1, aaiti, xiang yuanxin}@i2r.a-star.edu.sg dcscts@nus.edu.sg Abstract While achieving great fluency, current ma- chine translation (MT) techniques are bottle- necked by adequacy issues. To have a closer study of these issues and accelerate model development, we propose automatic detect- ing adequacy errors in MT hypothesis for MT model evaluation. To do that, we anno- tate missing and wrong translations, the two most prevalent issues for current neural ma- chine translation model, in 15000 Chinese- English translation pairs. We build a super- vised alignment model for translation error de- tection (AlignDet) based on a simple Align- ment Triangle strategy to set the benchmark for automatic error detection task. We also dis- cuss the difficulties of this task and the benefits of this task for existing evaluation metrics. 1 Introduction Different translation errors impact translation comprehensibility and adequacy differently. For example, wrong product or terminology transla- tion can be considered more severe than missing translation in the eCommerce domain. Though many machine translation evaluation (MTE) met- rics have been proposed, most of them are not able to provide a direct connection between the score and the error class to emphasize the differ- ent impact on translation comprehension and ad- equacy. Most MTE scores measure the similarity between the MT hypothesis and the reference us- ing n-gram and are useful in providing immediate feedback on model performance for system devel- opment and objective evaluation for system com- parison. To further analyse the MT hypothesis, fine-grained translation analysis is normally con- ducted manually which involves expensive human effort (Popovi´c and Ney, 2011; Popovi´c, 2011b; Fishel et al., 2012; Vilar et al., 2006). This motivates early unsupervised error detec- tion methods (Popovi´c and Ney, 2011). Such methods find the erroneous words by calculating the edit distance between the MT hypothesis and its corresponding reference. However, such un- supervised methods suffer from low accuracy and inability to provide and distinguish certain error classes like Missing words and Wrong translation1 (c.f. Sec 3). On the other hand, the works on supervised methods are few due to limited train- ing resources available. The annotated corpus are mainly collected from outputs of Phrase-based Machine Translation systems and usually suffer from small quantity (around 200 instances per lan- guage pair) (Fishel et al., 2012), noisy contents (derived from student assignments) (Wisniewski et al., 2014) or overabundant efforts (additional human efforts are required on postediting the MT hypothesis) (Popovi´c and Arcan, 2016), making the progress and development of this task chal- lenging. We believe a corpus annotated with differ- ent translation error classes will facilitate the re- search on …

Content truncated to 500 words