Ranking Translations using Error Analysis and Quality Estimation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 405–407, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Ranking Translations using Error Analysis and Quality Estimation Mark Fishel Institute of Computational Linguistics University of Zurich, Switzerland fishel@cl.uzh.ch Abstract We describe TerrorCat, a submission to this year’s metrics shared task. It is a ma- chine learning-based metric that is trained on manual ranking data from WMT shared tasks 2008–2012. Input features are generated by applying automatic transla- tion error analysis to the translation hy- potheses and calculating the error cate- gory frequency differences. We addition- ally experiment with adding quality es- timation features in addition to the er- ror analysis-based ones. When evaluated against WMT’2012 rankings, the system- level agreement is rather high for several language pairs. 1 Introduction Recently a couple of methods of automatic analy- sis of translation errors have been described (Ze- man et al., 2011; Popovi´c and Ney, 2011). Both of these compare a hypothesis translation to a refer- ence and draw detailed conclusions from the dif- ferences between the two. TerrorCat, a metric submitted to the metrics shared task of WMT’2012 (Callison-Burch et al., 2012) used the output of those two error analysis methods as input features, which yielded mildly promising results (Fishel et al., 2012). How- ever the submitted version of TerrorCat was lan- guage pair-specific, which means that the classifier model used by the metric has to be retrained on new manual pairwise ranking data for every new language pair. This in turn complicates its usage. Our main aim in this work is to make Terror- Cat usable out-of-the-box. We compare models specific to the language pair (baseline), target lan- guage and a universal model for all languages. The updated metric is applied to the WMT’13 metrics shared task. An additional modification to the metric uses input features from quality estimation. Using the resources of the quality estimation shared task of WMT’13 the modified model is applied to the English–Spanish language pair. We start by briefly re-introducing the TerrorCat metric. 2 Baseline The baseline TerrorCat metric is a machine learning-based metric: it uses manually ranked translation hypothesis pairs to train a classifier model. The trained model is then used to predict a ranking for new sentence pairs that have not been ranked yet. To convert the binary comparisons between translation hypothesis pairs into a numeric score per translation hypothesis the wins per hypothe- sis are summed together. Previous year’s work has shown (Fishel et al., 2012) that weighting the wins with the classifier’s confidence for the summed score improves agreement with human judgements. The input features for learning and classifica- tion are obtained by 1. applying translation error analysis software to the compared hypotheses, 2. getting the frequencies of every error type, i.e. the ratio of words marked with that er- ror type to the hypothesis sentence length, 3. and using each error type’s frequency differ- ences between the two hypotheses as input features. Relative frequencies are used on both system …

Content truncated to 500 words