Large-scale Multitask Learning for Machine Translation Quality Estimation

Proceedings of NAACL-HLT 2016, pages 558–567, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Large-scale Multitask Learning for Machine Translation Quality Estimation Kashif Shah and Lucia Specia Department of Computer Science University of Sheffield, UK {kashif.shah,l.specia}@sheffield.ac.uk Abstract Multitask learning has been proven a useful technique in a number of Natural Language Processing applications where data is scarce and naturally diverse. Examples include learn- ing from data of different domains and learn- ing from labels provided by multiple annota- tors. Tasks in these scenarios would be the domains or the annotators. When faced with limited data for each task, a framework for the learning of tasks in parallel while using a shared representation is clearly helpful: what is learned for a given task can be transferred to other tasks while the peculiarities of each task are still modelled. Focusing on machine translation quality estimation as application, in this paper we show that multitask learning is also useful in cases where data is abundant. Based on two large-scale datasets, we explore models with multiple annotators and multiple languages and show that state-of-the-art mul- titask learning algorithms lead to improved re- sults in all settings. 1 Introduction Quality Estimation (QE) models predict the qual- ity of Machine Translation (MT) output based on the source and target texts only, without reference translations. This task is often framed as a super- vised machine learning problem using various fea- tures indicating fluency, adequacy and complexity of the source-target text pair, and annotations on translation quality given by human translators. Var- ious kernel-based regression and classification algo- rithms have been explored to learn prediction mod- els. The application of QE we focus on here is that of guiding professional translators during the post- editing of MT output. QE models can provide trans- lators with information on how much editing/time will be necessary to fix a given segment, or on whether it is worth editing it at all, as opposed to translating it from scratch. For this application, models are learnt from quality annotations that re- flect post-editing effort, for instance, 1-5 judgements on estimated post-editing effort (Callison-Burch et al., 2012) or actual post-editing effort measured as post-editing time (Bojar et al., 2013) or edit distance between the MT output and its post-edited version (Bojar et al., 2014; Bojar et al., 2015). One of the biggest challenges in this field is to deal with the inherent subjectivity of quality labels given by humans. Explicit judgements (e.g. the 1-5 point scale) are affected the most, with pre- vious work showing that translators’ perception of post-editing effort differs from actual effort (Kopo- nen, 2012). However, even objective annotations of actual post-editing effort are subject to natural variance. Take, for example, post-editing time as a label: Different annotators have different typing speeds and may require more or less time to deal with the same edits depending on their level of expe- rience, familiarity with the domain, etc. Post-editing distance also varies across translators as there are of- ten …

Content truncated to 500 words