Comparative Quality Estimation: Automatic Sentence-Level Ranking of Multiple Machine Translation Outputs

Proceedings of COLING 2012: Technical Papers, pages 115–132, COLING 2012, Mumbai, December 2012. Comparative quality estimation: Automatic sentence-level ranking of multiple Machine Translation outputs Eleftherios Avramidis German Research Center for Artificial Intelligence (DFKI) Language Technology Lab Alt Moabit 91c, 10559 Berlin, Germany eleftherios.avramidis@dfki.de ABSTRACT A machine learning mechanism is learned from human annotations in order to perform preference ranking. The mechanism operates on a sentence level and ranks the alternative machine translations of each source sentence. Rankings are decomposed into pairwise comparisons so that binary classifiers can be trained using black-box features of automatic linguistic analysis. In order to re-compose the pairwise decisions of the classifier, this work introduces weighing the decisions with their classification probabilities, which eliminates ranking ties and increases the coefficient of the correlation with the human rankings up to 80%. The authors also demonstrate several configurations of successful automatic ranking models; the best configuration achieves acceptable correlation with human judgments (tau=0.30), which is higher than that of state-of-the-art reference-aware automatic MT evaluation metrics such as METEOR and Levenshtein distance. KEYWORDS: Machine Translation, Quality Estimation, Ranking. 115 1 Introduction As machine translation (MT) comes closer to the everyday use, the need to predict the quality of machine-translated text is being of intense interest. For this reason, research has been focusing more and more on developing methods of assessing the translated content and deriving indications of translation performance in a real-time translation environment, without access to the correct (reference) translations. Here we are focusing on a specific scenario: we need an automatic system able to rank several machine translation outputs for one given source sentence, according to their comparative quality. This kind of ranking, performed by human annotators, has been established as a practice for evaluating MT-output. Therefore, we attempt to perform “machine ranking”, by employing Machine Learning approaches in order to imitate the human behaviour. This is done through a statistical classifier, which is trained given existing human ranks and several qualitative criteria on the text. This idea can serve several existing applications in MT, as it touches the fields of Hybrid MT, System Combination and MT Evaluation. The applicability is even broader, as the approach presented is system-independent and relies on generic automatic analysis applied on any input containing sets of one source and several translation outputs. 2 Previous work Quality Estimation is a rather recent aspect in research on Machine Translation. As a field, it tries to provide quality assessment on the translation output without the availability of reference translations. Previous work includes statistical methods on predicting word-level confidence (Ueffing and Ney, 2005; Raybaud et al., 2009b), correctness of a sentence (Blatz et al., 2004) and has been recently evolved into a regression problem (Specia et al., 2009; Raybaud et al., 2009a) for estimating correctness scores or correctness probabilities. Whereas the aforementioned work has been focusing on estimating absolute measures of quality for a single output, our focus is on the comparative estimation of quality among several system outputs. In this direction, Rosti et al. …

Content truncated to 500 words