Rethinking the Agreement in Human Evaluation Tasks

Proceedings of the 27th International Conference on Computational Linguistics, pages 3318–3329 Santa Fe, New Mexico, USA, August 20-26, 2018. 3318 Rethinking the Agreement in Human Evaluation Tasks (Position Paper) Jacopo Amidei and Paul Piwek and Alistair Willis School of Computing and Communications The Open University Milton Keynes, UK Abstract Human evaluations are broadly thought to be more valuable the higher the inter-annotator agree- ment. In this paper we examine this idea. We will describe our experiments and analysis within the area of Automatic Question Generation. Our experiments show how annotators diverge in language annotation tasks due to a range of ineliminable factors. For this reason, we believe that annotation schemes for natural language generation tasks that are aimed at evaluating language quality need to be treated with great care. In particular, an unchecked focus on reduction of dis- agreement among annotators runs the danger of creating generation goals that reward output that is more distant from, rather than closer to, natural human-like language. We conclude the paper by suggesting a new approach to the use of the agreement metrics in natural language generation evaluation tasks. 1 Introduction Human evaluations are broadly thought to be more valuable the higher the inter-annotator agreement (IAA). In natural language processing (NLP), IAA is often viewed as a means of assessing the quality of data on a task, in particular, the reliability. Also, it helps to identify poor annotations, provide information on where annotation guidelines can be improved, and establish an upper bound on system performance. Nevertheless some studies, for example (Sampson and Babarczy, 2008), (Kov´aˇr 2016) and (Kov´aˇr et al., 2016)1, propose IAA limitations when used in the annotation of natural language generation (NLG)2, where annotation usually concerns the quality of a generated sentence – including a range of features such as syntax, semantic and pragmatics. Indeed, natural language brings in itself a variability which cannot be reduced, except by weakening its expressive power. For instance, the perception of sentence idiomaticity can change from person to person because of styles, educational and regional difference (for example differences between British and American English for English sentences). Far from being a problem this shows the richness, variability and beauty of human languages. In this paper we are going to study IAA in the area of Automatic Question Generation (AQG)3. We performed experiments which show that there are factors inherent to question annotation that place an upper-bound on IAA. We found subjective bias that cannot be removed by improving the annotation This work is licenced under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/ 1It is also worth mentioning Martinez Alonso et al. (2016), where the authors use the systematic disagreement in order to train their model. Martinez Alonso et al. raise the discussion about how to distinguish and use potentially relevant from irrelevant disagreement. Although this problem is very interesting and deserves a deep analysis, it goes beyond the scope of the present paper. 2In what follows, when we refer to NLG tasks, we …

Content truncated to 500 words