Document-level translation quality estimation: exploring discourse and pseudo-references

Document-level translation quality estimation: exploring discourse and pseudo-references Carolina Scarton and Lucia Specia Department of Computer Science University of Sheffield S1 4DP, UK {c.scarton,l.specia}@sheffield.ac.uk Abstract Predicting the quality of machine trans- lations is a challenging topic. Quality estimation (QE) of translations is based on features of the source and target texts (without the need for human references), and on supervised machine learning meth- ods to build prediction models. Engineer- ing well-performing features is therefore crucial in QE modelling. Several fea- tures have been used so far, but they tend to explore very short contexts within sen- tence boundaries. In addition, most work has targeted sentence-level quality predic- tion. In this paper, we focus on document- level QE using novel discursive features, as well as exploiting pseudo-reference trans- lations. Experiments with features ex- tracted from pseudo-references led to the best results, but the discursive features also proved promising. 1 Introduction The purpose of machine translation (MT) qual- ity estimation (QE) is to provide a quality pre- diction for new, unseen machine translated texts, without relying on reference translations (Blatz et al., 2004; Specia et al., 2009; Bojar et al., 2013). This task is usually addressed with machine learn- ing models trained on datasets composed of source texts, their machine translations, and a quality la- bel assigned by humans or by an automatic metric (e.g.: BLEU (Papineni et al., 2002)). A common use of quality predictions is the estimation of post- c⃝ 2014 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. editing effort in order to decide whether to trans- late a text from scratch or post-edit its machine translation. Another use is the ranking of transla- tions in order to select the best text from multiple MT systems. Feature engineering is an important compo- nent in QE. Although several feature sets have al- ready been explored, most approaches focus on sentence-level quality prediction, with sentence- level features. These disregard document struc- ture or wider contexts beyond sentence bound- aries. To the best of our knowledge, only Rubino et al. (2013) considered discourse-related informa- tion by studying topic model features for sentence- level prediction. Soricut and Echihabi (2010) ex- plored document-level quality prediction, but they did not use explicit discourse information, e.g. in- formation to capture text cohesion or coherence. In this paper we focus on document-level fea- tures and document-level prediction. We be- lieve that judgements on translation quality de- pend on units longer than just a given sentence, taking into account discourse phenomena for lex- ical choice, consistency, style and connectives, among others (Carpuat and Simard, 2012). This is particularly important in MT evaluation contexts, since most MT systems, and in particular statisti- cal MT (SMT) systems, process sentences one by one, in isolation. Our hypothesis is that features that capture discourse phenomena can improve document-level prediction. We consider two fami- lies of features that have been successfully applied in reference-based MT evaluation (Wong and Kit, 2012) and readability assessment …

Content truncated to 500 words