Student’s t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce

Proceedings of Recent Advances in Natural Language Processing, pages 419–428 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_047 419 Student’s t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce Serge Gladkoff 1∗, Lifeng Han2∗, and Goran Nenadic2 1 Logrus Global, Translation & Localization 2 The University of Manchester, UK lifeng.han, g.nenadic @ manchester.ac.uk serge.gladkoff @ logrusglobal.com ∗ co-first authors Abstract In natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. How- ever, there has been an ongoing debate on how to better evaluate inter-rater re- liability (IRR) levels for certain evalua- tion tasks, such as translation quality eval- uation (TQE), especially when the data samples (observations) are very scarce. In reality, practitioners need to be able to assess the reliability of human MT quality evaluation based on one, two, or maximum three human linguists’ judge- ments. In this work, we first introduce the little-known method to estimate the confidence interval for the measurement value when only one data (evaluation) point is available. This leads to our ex- ample with two human-generated observa- tional scores, for which we describe “Stu- dent’s t-Distribution”, and explain how to use it to measure the IRR score us- ing only these two data points, and cal- culate the confidence interval (CI) of the quality evaluation. We give a quantitative analysis of how the evaluation confidence can be greatly improved by introducing more observations, even if only one extra observation. We encourage practitioners and researchers to report their IRR scores and confidence intervals in all evaluations, e.g. using Student’s t-Distribution method whenever possible; thus making the NLP evaluation more meaningful, transparent, and trustworthy. 1 Introduction Human evaluations have been always the gold standard to judge the quality of natural language processing (NLP) system’s outputs (Han et al., 2021; Freitag et al., 2021; Gladkoff and Han, 2022). This applies to many sub-tasks including machine translation (MT) (Han et al., 2020; Han, 2022a; Charalampidou and Gladkoff, 2022; MI- LAD, 2022), text summarisation (Bhandari et al., 2020; Latif et al., 2009), question answering (Al- rdahi et al., 2020), information extraction (Wu et al., 2022; Nenadic et al., 2004), and prediction (Yang et al., 2009), as well as domain applica- tions such as social media, biomedical and clini- cal domains knowledge representation (Miloˇsevi´c et al., 2019; Yang et al., 2021; Krauthammer and Nenadic, 2004). Nonetheless, human evaluations have been subject to criticisms and debates about their reliability, particularly when conducted with- out strictly defined procedures. (Han, 2022b; Han and Gladkoff, 2022; Graham et al., 2017). Despite the inclusion of factors such as quality controls and clear guidelines, human evaluation results can vary greatly among different individuals due to subjective judgements influenced by factors such as backgrounds, personalities, cultures, and so on. Naturally, the confidence levels of human eval- uation become the key point to the validity of such work. There have been some efforts made on how to measure the confidence level of human evalua- tions from a statistical point of view, such as …

Content truncated to 500 words