Coping with the Subjectivity of Human Judgements in MT Quality Estimation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 240–251, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Coping with the Subjectivity of Human Judgements in MT Quality Estimation Marco Turchi Matteo Negri Marcello Federico Fondazione Bruno Kessler, FBK-irst Trento , Italy {turchi|negri|federico}@fbk.eu Abstract Supervised approaches to NLP tasks rely on high-quality data annotations, which typically result from expensive manual la- belling procedures. For some tasks, how- ever, the subjectivity of human judgements might reduce the usefulness of the an- notation for real-world applications. In Machine Translation (MT) Quality Esti- mation (QE), for instance, using human- annotated data to train a binary classifier that discriminates between good (useful for a post-editor) and bad translations is not trivial. Focusing on this binary task, we show that subjective human judge- ments can be effectively replaced with an automatic annotation procedure. To this aim, we compare binary classifiers trained on different data: the human-annotated dataset from the 7th Workshop on Statis- tical Machine Translation (WMT-12), and an automatically labelled version of the same corpus. Our results show that human labels are less suitable for the task. 1 Introduction With the steady progress in the field of Statistical Machine Translation (SMT), the translation indus- try is now faced with the possibility of significant productivity increases (i.e. amount of publishable output per unit of time). One way to achieve this goal, in Computer Assisted Translation (CAT) en- vironments, is the integration of (precise, but of- ten partial) suggestions obtained through “fuzzy matches” from a Translation Memory (TM), with (complete, but potentially less precise) translations produced by an MT system. Such integration can loosely consist in presenting translators with un- ranked suggestions obtained from the MT and the TM, or rely on tighter combination strategies. For instance, MT and TM translations can be automat- ically ranked to ease the selection of the most suit- able one for post-editing (He et al., 2010), or the TM can be used to constrain and improve MT sug- gestions (Ma et al., 2011). In all cases, the ef- fectiveness of the integration is conditioned by: i) the quality of MT, and ii) the accuracy in au- tomatically predicting such quality. Higher pro- ductivity increases depend on the capability of the MT system to output useful material that is close to be publishable “as is” (Denkowski and Lavie, 2012), and the capability to automatically identify and present to human translators only such sug- gestions. Recognizing good translations falls in the scope of research on automatic MT Quality Estimation (QE), which addresses the problem of estimating the quality of a translated sentence at run-time, without access to reference translations (Specia et al., 2009; Soricut and Echihabi, 2010; Bach et al., 2011; Specia, 2011; Mehdad et al., 2012b). In recent years QE gained increasing interest in the MT community, resulting in several datasets avail- able for training and evaluation (Callison-Burch et al., 2012), the definition of features showing good correlation with human judgements (Soricut et al., 2012), and the release …

Content truncated to 500 words