Selecting Feature Sets for Comparative and Time-Oriented Quality Estimation of Machine Translation Output

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 329–336, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Machine learning methods for comparative and time-oriented Quality Estimation of Machine Translation output Eleftherios Avramidis and Maja Popovi´c German Research Center for Artificial Intelligence (DFKI GmbH) Language Technology Lab Alt Moabit 91c, 10559 Berlin eleftherios.avramidis@dfki.de and maja.popovic@dfki.de Abstract This paper describes a set of experi- ments on two sub-tasks of Quality Esti- mation of Machine Translation (MT) out- put. Sentence-level ranking of alternative MT outputs is done with pairwise classi- fiers using Logistic Regression with black- box features originating from PCFG Pars- ing, language models and various counts. Post-editing time prediction uses regres- sion models, additionally fed with new elaborate features from the Statistical MT decoding process. These seem to be better indicators of post-editing time than black- box features. Prior to training the models, feature scoring with ReliefF and Informa- tion Gain is used to choose feature sets of decent size and avoid computational com- plexity. 1 Introduction During the recent years, Machine Translation (MT) has reached levels of performance which al- low for its integration into real-world translation workflows. Despite the high speed and various ad- vantages of this technology, the fact that the MT results are rarely perfect and often require man- ual corrections has raised a need to assess their quality, predict the required post-editing effort and compare outputs from various systems on applica- tion time. This has been the aim of current re- search on Quality Estimation, which investigates solutions for several variations of such problems. We describe possible solutions for two prob- lems of MT Quality Estimation, as part of the 8th Shared Task on Machine Translation: (a) sentence-level quality ranking (1.2) of multi- ple translations of the same source sentence and (b) prediction of post-editing time (1.3). We present our approach on acquiring (section 2.1) and selecting features (section 2.2), we explain the generation of the statistical estimation systems (section 2.3) and we evaluate the developed solu- tions with some of the standard metrics (section 3). 2 Methods: Quality Estimation as machine learning These two Quality Estimation solutions have been seen as typical supervised machine learning prob- lems. MT output has been given to humans, so that they perform either (a) ranking of the multiple MT system outputs in terms of meaning or (b) post- editing of single MT system output, where time needed per sentence is measured. The output of these tasks has been provided by the shared task organizers as a training material, whereas a small keep-out set has been reserved for testing pur- poses. Our task is therefore to perform automatic qual- ity analysis of the translation output and the trans- lation process in order to provide features for the supervised machine learning mechanism, which is then trained over the corresponding to the respec- tive human behaviour. The task is first optimized in a development phase in order to produce the two best shared task submissions for each task. …

Content truncated to 500 words