What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8932–8939 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8932 What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think David M. Howcroft∗ School of Computing Edinburgh Napier University Edinburgh, Scotland, UK d.howcroft@napier.ac.uk Verena Rieser The Interaction Lab, MACS Heriot-Watt University Edinburgh, Scotland, UK v.t.rieser@hw.ac.uk Abstract Previous work has shown that human evalua- tions in NLP are notoriously under-powered. Here, we argue that there are two common factors which make this problem even worse: NLP studies usually (a) treat ordinal data as in- terval data and (b) operate under high variance settings while the differences they are hoping to detect are often subtle. We demonstrate through simulation that ordinal mixed effects models are better able to detect small differ- ences between models, especially in high vari- ance settings common in evaluations of gener- ated texts. We release tools for researchers to conduct their own power analysis and test their assumptions. We also make recommendations for improving statistical power. 1 Introduction Human evaluation remains the gold standard for many natural language generation tasks, including machine translation, data-to-text, summarisation, and dialogue & interactive systems. One common way to elicit text quality ratings from study partic- ipants is to use a rating scale, e.g. a Likert scale which measures agreement with a statement, or other visual or verbal analogue scales, as in Fig- ure 1a. Unfortunately, typically chosen statistical analyses of these scores often rely on the flawed as- sumption that the rating scales are interval, i.e. that the distance between any two adjacent points on the scale is the same across the full range of values, so that, for example, the difference between ‘very disfluent’ & ‘disfluent’ is the same as the distance between ‘slightly disfluent’ & ‘slightly fluent’ on a 6-point scale (see Figure 1). The distributions in Figure 1 illustrate the differ- ent underlying assumptions of interval and ordinal models of rating scale data: The rating scale in (1a) is used to collect human judgements of text quality *Work completed while at Heriot-Watt University. (a) A six point rating scale for fluency 0.0 0.2 0.4 Ordinal Value Probability (b) Example frequency histogram for data collected with (1a). 0.0 0.1 0.2 0.3 0 2 4 6 8 Numeric Representation of Ordinal Response Density (c) Interval model of the data in (1b). 0.0 0.1 0.2 0.3 0.4 −2 0 2 Fluency (Latent Variable Scale) Density (d) Ordered probit model of the data in (1b). Figure 1: The model in (1d) yields more accurate prob- abilities than those in (1c). (e.g. fluency), which results in a distribution of or- dinal data as in (1b). In (1c) we follow the interval assumption, that each point on the six-point rating scale corresponds directly to a real-valued integer and that we can model the relative probability of any pair of ratings based on a Gaussian probability density function. In contrast, (1d) assumes …

Content truncated to 500 words