Combining Quality Prediction and System Selection for Improved Automatic Translation Output

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 163–170, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Combining Quality Prediction and System Selection for Improved Automatic Translation Output Radu Soricut SDL Language Weaver 6060 Center Drive, Suite 150 Los Angeles, CA 90045 rsoricut@sdl.com Sushant Narsale∗ Google Inc 1600 Amphitheatre Parkway Mountain View, CA 94043 snarsale@google.com Abstract This paper presents techniques for reference- free, automatic prediction of Machine Trans- lation output quality at both sentence- and document-level. In addition to helping with document-level quality estimation, sentence- level predictions are used for system selection, improving the quality of the output transla- tions. We present three system selection tech- niques and perform evaluations that quantify the gains across multiple domains and lan- guage pairs. 1 Introduction Aside from improving the performance of core- translation models, there additionally exist two orthogonal approaches via which fully-automatic translations can achieve increased acceptance and better integration in real-world use cases. These two approaches are: improved translation accuracy via system combination (Rosti et al., 2008; Karakos et al., 2008; Hildebrand and Vogel, 2008), and auto- matic quality-estimation techniques used as an ad- ditional layer on top of MT systems, which present the user only with translations that are predicted as being accurate (Soricut and Echihabi, 2010; Specia, 2011). In this paper, we describe new contributions to both these approaches. First, we present a novel and superior technique for performing quality esti- mation at document level. We achieve this by chang- ∗Research was completed before the author started in his current role at Google Inc. The opinions stated are his own and not of Google Inc. ing the granularity of the prediction mechanism from document-level (Soricut and Echihabi, 2010) to sentence-level, and predicting BLEU scores via directly modeling the sufficient statistics for BLEU computation. A document-level score is then recre- ated based on the predicted sentence-level sufficient statistics. A second contribution is related to system combination (or, to be more precise, system selec- tion). This is an intended side-effect of the granular- ity change: since the sentence-level statistics allow us to make quality predictions at sentence level, we can use these predictions to perform system com- bination by selecting among various sentence-level translations produced by different MT systems. That is, instead of presenting the user with a document with sentences translated entirely by a single system, we can present documents for which, say, 60% of the sentences were translated by system A, and 40% were translated by system B. We contribute a novel set of features and several techniques for choos- ing between competing machine translation outputs. The evaluation results show better output quality, across multiple domains and language pairs. 2 Related Work Several approaches to reference-free automatic MT quality assessment have been proposed, using classi- fication (Kulesza and Shieber, 2004), regression (Al- brecht and Hwa, 2007), and ranking (Ye et al., 2007; Duh, 2008). The focus of these approaches is on sys- tem performance evaluation, as they use a constant test set and measure …

Content truncated to 500 words