Identifying Annotator Bias: A new IRT-based method for bias identification

Proceedings of the 28th International Conference on Computational Linguistics, pages 4787–4797 Barcelona, Spain (Online), December 8-13, 2020 4787 Identifying Annotator Bias: A new IRT-based method for bias identification Jacopo Amidei and Paul Piwek and Alistair Willis School of Computing and Communications The Open University Milton Keynes, UK Abstract A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement. 1 Introduction In computational linguistics (CL), since Carletta’s influential paper (Carletta, 1996), the standard ap- proach to check the reliability in annotation efforts is the Inter Annotator Agreement (IAA) – that is the extent to which different annotators independently make the same annotation decisions. For the purpose of calculating the IAA, Carletta proposed the use of a family of statistics related to Cohen’s κ coefficient of agreement (Cohen, 1960), which she collectively refers to as the Kappa statistic.1 An important factor that can affect the IAA is the presence of annotator bias – that is differences between annotator preferences for subjective reasons. Recent work – for example Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018) – show that annotators diverge in language annotation tasks due to a range of ineliminable factors such as background knowledge, preconceptions about language and general educational level. Although clear annotation schemes with effective guidelines and annotator training aim to reduce annotation bias, some individual differences persist. A method that allows for a better identification and understanding of individual bias could be valuable in annotation efforts. For example, it could be used to: • display differences in annotators’ behaviour. Annotations that show a markedly different pattern from the other annotations can either be removed or further analysed. This can help to spot outlying annotators and help with improving annotation guidelines and reduce annotation disagreement.2 • provide a better picture of the annotation reliability. Indeed, once identified, the annotator bias could be used to explain and understand annotation disagreement and accordingly the IAA values. For instance, as we will see in Section 2.3, such a method could be used to show in which respect an annotator shows more strict annotation behaviour than another annotator. This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 1Some examples of the Kappa statistic are Cohen’s κ (Cohen, 1960), Fleiss’ κ (Fleiss, 1971) …

Content truncated to 500 words