MoBiL: A Hybrid Feature Set for Automatic Human Translation Quality Assessment

MoBiL: A Hybrid Feature Set for Automatic Human Translation Quality Assessment Yu Yuan1, Serge Sharoff2, Bogdan Babych3 Nanjing University of Information Science &Technology1 University of Leeds2,3 LS2 9JT E-mail: hittle.yuan@gmail.com; {s.sharoff; b.babych}@leeds.ac.uk Abstract In this paper we introduce MoBiL, a hybrid Monolingual, Bilingual and Language modelling feature set and feature selection and evaluation framework. The set includes translation quality indicators that can be utilized to automatically predict the quality of human translations in terms of content adequacy and language fluency. We compare MoBiL with the QuEst baseline set by using them in classifiers trained with support vector machine and relevance vector machine learning algorithms on the same data set. We also report an experiment on feature selection to opt for fewer but more informative features from MoBiL. Our experiments show that classifiers trained on our feature set perform consistently better in predicting both adequacy and fluency than the classifiers trained on the baseline feature set. MoBiL also performs well when used with both support vector machine and relevance vector machine algorithms. Keywords: Translation Quality, Feature Selection, Text Classification, Machine Learning 1. Introduction Progress in a number of Natural Language Processing and Machine Learning technologies in the past 40 years led to the development of applications for automated scoring of essays and language testing. Systems of automated essay scoring have been sufficiently reliable to be commercialized and deployed for large international language test (Dodigovic, 2005: 104). Some well-known systems include Project Essay Grader, E2rater, Intellimetric and PaperRater, to just name a few, but it is not the case with human translation assessment, particularly student translations. A different set of technologies have been applied for automated translation quality evaluation, where a lot of attention has been paid to automatic evaluation of Machine Translation (MT), in the form of methods based on parallel corpora, for example, BLEU (Papineni et al., 2002), or MT Quality Estimation (QE), which estimates the suitability of MT output without a reference translation, QuEst (Shah et al., 2013) selects high quality MT translations (Ma & McKeown, 2013) or detects machine translation errors (Xiong, Zhang, & Li, 2010). However, automatic Translation Quality Assessment (TQA) for human translations is a more complicated problem compared to the automated evaluation of MT. One of the reasons is that machine translations, generally inferior to human translation, usually contain a much smaller range of translation errors in comparison with human translations. Human translation errors only partially overlap with errors made by MT and display greater variability, which makes human translations less predictable, see section 4.2 our pilot experiment for reference. As a result, this non-uniformity of the creative human choices and translations further complicates the task of automated estimation of human translation quality, including automated scoring of trainee translators’ work, and so far this area remains under-researched. Useful research in this area comes from the language learning and translator training scenarios, where some attempts have been made to automatically evaluate students’ translations on a large scale (Jiang, 2013；Wang & Chang, 2009; Wen, Qin, & Jiang, …

Content truncated to 500 words