Modelling Annotator Bias with Multi-task Gaussian Processes: An Application to Machine Translation Quality Estimation

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 32–42, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Modelling Annotator Bias with Multi-task Gaussian Processes: An Application to Machine Translation Quality Estimation Trevor Cohn and Lucia Specia Department of Computer Science University of Sheffield Sheffield, United Kingdom {t.cohn,l.specia}@sheffield.ac.uk Abstract Annotating linguistic data is often a com- plex, time consuming and expensive en- deavour. Even with strict annotation guidelines, human subjects often deviate in their analyses, each bringing different biases, interpretations of the task and lev- els of consistency. We present novel tech- niques for learning from the outputs of multiple annotators while accounting for annotator specific behaviour. These tech- niques use multi-task Gaussian Processes to learn jointly a series of annotator and metadata specific models, while explicitly representing correlations between models which can be learned directly from data. Our experiments on two machine trans- lation quality estimation datasets show uniform significant accuracy gains from multi-task learning, and consistently out- perform strong baselines. 1 Introduction Most empirical work in Natural Language Pro- cessing (NLP) is based on supervised machine learning techniques which rely on human anno- tated data of some form or another. The annota- tion process is often time consuming, expensive, and prone to errors; moreover there is often con- siderable disagreement amongst annotators. In general, the predominant perspective to deal with these data annotation issues in previous work has been that there is a single underlying ground truth, and that the annotations collected are noisy and/or biased samples of this. The challenge is then one of quality control, in order to process the data by filtering, averaging or similar to dis- til the truth. We posit that this perspective is too limiting, especially with respect to linguis- tic data, where each individual’s idiolect and lin- guistic background can give rise to many different – and yet equally valid – truths. Particularly in highly subjective annotation tasks, the differences between annotators cannot be captured by simple models such as scaling all instances of a certain annotator by a factor. They can originate from a number of nuanced aspects. This is the case, for example, of annotations on the quality of sen- tences generated using machine translation (MT) systems, which are often used to build quality es- timation models (Blatz et al., 2004; Specia et al., 2009) – our application of interest. In addition to annotators’ own perceptions and expectations with respect to translation quality, a number of factors can affect their judgements on specific sentences. For example, certain anno- tators may prefer translations produced by rule- based systems as these tend to be more grammati- cal, while others would prefer sentences produced by statistical systems with more adequate lexical choices. Likewise, some annotators can be biased by the complexity of the source sentence: lengthy sentences are often (subconsciously) assumed to be of low quality by some annotators. An ex- treme case is the judgement of quality through post-editing time: annotators have different typing speeds, …

Content truncated to 500 words